search

jeff-regier / Celeste.jl

Scalable inference for a generative model of astronomical images
MIT License
183 stars 28 forks source link

assess model fit on real data #146

Closed jeff-regier closed 8 years ago

jeff-regier commented 8 years ago

The scores are starting to look reasonable, despite issues #189 and #191.

│ Row │ field        │ primary   │ celeste   │ diff         │ diff_sd    │ N   │
┝━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━━━━━━┿━━━━━┥
│ 1   │ position     │ 0.280856  │ 0.240141  │ 0.0407152    │ 0.0256301  │ 421 │
│ 2   │ missed_stars │ 0.0213777 │ 0.0760095 │ -0.0546318   │ 0.0115322  │ 421 │
│ 3   │ missed_gals  │ 0.0546318 │ 0.0451306 │ 0.00950119   │ 0.00874892 │ 421 │
│ 4   │ mag_r        │ 0.216789  │ 0.343632  │ -0.126842    │ 0.0198543  │ 421 │
│ 5   │ color_ug     │ 1.2507    │ 0.75151   │ 0.499192     │ 0.0550487  │ 375 │
│ 6   │ color_gr     │ 0.330433  │ 0.296815  │ 0.0336183    │ 0.0280016  │ 418 │
│ 7   │ color_ri     │ 0.203661  │ 0.216151  │ -0.0124892   │ 0.0140117  │ 421 │
│ 8   │ color_iz     │ 0.294741  │ 0.157746  │ 0.136995     │ 0.0222605  │ 421 │
│ 9   │ gal_fracdev  │ 0.272315  │ 0.272472  │ -0.000156956 │ 0.018477   │ 177 │
│ 10  │ gal_ab       │ 0.19273   │ 0.176032  │ 0.0166977    │ 0.0121665  │ 177 │
│ 11  │ gal_scale    │ 1.93786   │ 2.25016   │ -0.312298    │ 0.467199   │ 177 │
│ 12  │ gal_angle    │ 17.4246   │ 44.244    │ -26.8194     │ 2.63651    │ 81  │
rgiordan commented 8 years ago

Could you explain what this table is showing?

On Wed, Mar 30, 2016 at 6:56 PM, Jeffrey Regier notifications@github.com wrote:

The scores are starting to look reasonable, despite issues #189 https://github.com/jeff-regier/Celeste.jl/issues/189 and #191 https://github.com/jeff-regier/Celeste.jl/issues/191.

│ Row │ field │ primary │ celeste │ diff │ diff_sd │ N │ ┝━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━━━━━━┿━━━━━┥ │ 1 │ position │ 0.280856 │ 0.240141 │ 0.0407152 │ 0.0256301 │ 421 │ │ 2 │ missed_stars │ 0.0213777 │ 0.0760095 │ -0.0546318 │ 0.0115322 │ 421 │ │ 3 │ missed_gals │ 0.0546318 │ 0.0451306 │ 0.00950119 │ 0.00874892 │ 421 │ │ 4 │ mag_r │ 0.216789 │ 0.343632 │ -0.126842 │ 0.0198543 │ 421 │ │ 5 │ color_ug │ 1.2507 │ 0.75151 │ 0.499192 │ 0.0550487 │ 375 │ │ 6 │ color_gr │ 0.330433 │ 0.296815 │ 0.0336183 │ 0.0280016 │ 418 │ │ 7 │ color_ri │ 0.203661 │ 0.216151 │ -0.0124892 │ 0.0140117 │ 421 │ │ 8 │ color_iz │ 0.294741 │ 0.157746 │ 0.136995 │ 0.0222605 │ 421 │ │ 9 │ gal_fracdev │ 0.272315 │ 0.272472 │ -0.000156956 │ 0.018477 │ 177 │ │ 10 │ gal_ab │ 0.19273 │ 0.176032 │ 0.0166977 │ 0.0121665 │ 177 │ │ 11 │ gal_scale │ 1.93786 │ 2.25016 │ -0.312298 │ 0.467199 │ 177 │ │ 12 │ gal_angle │ 17.4246 │ 44.244 │ -26.8194 │ 2.63651 │ 81 │

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/jeff-regier/Celeste.jl/issues/146#issuecomment-203714744

jeff-regier commented 8 years ago

It's basically the same table as in the ICML paper. I switched from flux to magnitude. The results are computed on one field in stripe 82 now, rather than stamps.

On Wed, Mar 30, 2016, 11:31 PM Ryan notifications@github.com wrote:

Could you explain what this table is showing?

On Wed, Mar 30, 2016 at 6:56 PM, Jeffrey Regier notifications@github.com

wrote:

The scores are starting to look reasonable, despite issues #189 https://github.com/jeff-regier/Celeste.jl/issues/189 and #191 https://github.com/jeff-regier/Celeste.jl/issues/191.

│ Row │ field │ primary │ celeste │ diff │ diff_sd │ N │

┝━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━━━━━━┿━━━━━┥

│ 1 │ position │ 0.280856 │ 0.240141 │ 0.0407152 │ 0.0256301 │ 421 │ │ 2 │ missed_stars │ 0.0213777 │ 0.0760095 │ -0.0546318 │ 0.0115322 │ 421 │ │ 3 │ missed_gals │ 0.0546318 │ 0.0451306 │ 0.00950119 │ 0.00874892 │ 421 │ │ 4 │ mag_r │ 0.216789 │ 0.343632 │ -0.126842 │ 0.0198543 │ 421 │ │ 5 │ color_ug │ 1.2507 │ 0.75151 │ 0.499192 │ 0.0550487 │ 375 │ │ 6 │ color_gr │ 0.330433 │ 0.296815 │ 0.0336183 │ 0.0280016 │ 418 │ │ 7 │ color_ri │ 0.203661 │ 0.216151 │ -0.0124892 │ 0.0140117 │ 421 │ │ 8 │ color_iz │ 0.294741 │ 0.157746 │ 0.136995 │ 0.0222605 │ 421 │ │ 9 │ gal_fracdev │ 0.272315 │ 0.272472 │ -0.000156956 │ 0.018477 │ 177 │ │ 10 │ gal_ab │ 0.19273 │ 0.176032 │ 0.0166977 │ 0.0121665 │ 177 │ │ 11 │ gal_scale │ 1.93786 │ 2.25016 │ -0.312298 │ 0.467199 │ 177 │ │ 12 │ gal_angle │ 17.4246 │ 44.244 │ -26.8194 │ 2.63651 │ 81 │

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub < https://github.com/jeff-regier/Celeste.jl/issues/146#issuecomment-203714744>

— You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub https://github.com/jeff-regier/Celeste.jl/issues/146#issuecomment-203778592

jeff-regier commented 8 years ago

Scores are looking great now, with the recent PRs. Maybe the correct angle initialization helped especially?

Also inference is running like 50x or 100x faster--wow! (What was going on before?)

│ Row │ field        │ primary   │ celeste   │ diff       │ diff_sd    │ N   │
┝━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━━┿━━━━━━━━━━━━┿━━━━━┥
│ 1   │ position     │ 0.255618  │ 0.225739  │ 0.029879   │ 0.016931   │ 419 │
│ 2   │ missed_stars │ 0.0238663 │ 0.0811456 │ -0.0572792 │ 0.0122142  │ 419 │
│ 3   │ missed_gals  │ 0.0548926 │ 0.026253  │ 0.0286396  │ 0.00991759 │ 419 │
│ 4   │ mag_r        │ 0.201173  │ 0.349337  │ -0.148164  │ 0.0190963  │ 419 │
│ 5   │ color_ug     │ 1.24038   │ 0.700412  │ 0.539969   │ 0.0536563  │ 372 │
│ 6   │ color_gr     │ 0.326111  │ 0.217032  │ 0.109079   │ 0.0163301  │ 416 │
│ 7   │ color_ri     │ 0.203236  │ 0.169912  │ 0.0333236  │ 0.0157859  │ 419 │
│ 8   │ color_iz     │ 0.29284   │ 0.129721  │ 0.163118   │ 0.0198482  │ 419 │
│ 9   │ gal_fracdev  │ 0.271263  │ 0.256713  │ 0.0145497  │ 0.0191935  │ 174 │
│ 10  │ gal_ab       │ 0.188427  │ 0.135029  │ 0.053398   │ 0.0101751  │ 174 │
│ 11  │ gal_scale    │ 1.91571   │ 2.39779   │ -0.482082  │ 0.485313   │ 174 │
│ 12  │ gal_angle    │ 17.4348   │ 12.3188   │ 5.11597    │ 1.62281    │ 80  │
jeff-regier commented 8 years ago

I'm going to filter out saturated sources now, which should improve the scores even more, especially mag_r, but likely across the board.

rgiordan commented 8 years ago

Great!

Also inference is running like 50x or 100x faster--wow! (What was going on before?)

At some point a change was made that passed all the sources to ElboDeriv rather than the relevant subset. Even when active_sources was just one source, SourceBrightness and BvnDerivs are calculated for all sources, though most were not used. I made a copied subset of mp that only contains relevant sources. This also gets around the sequential dependence problem where the results from optimization on a particular thread were used in the optimization of subsequent sources.

│ Row │ field │ primary │ celeste │ diff │ diff_sd │ N │ ┝━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━━━┿━━━━━━━━━━━━┿━━━━━┥ │ 1 │ position │ 0.255618 │ 0.225739 │ 0.029879 │ 0.016931 │ 419 │ │ 2 │ missed_stars │ 0.0238663 │ 0.0811456 │ -0.0572792 │ 0.0122142 │ 419 │ │ 3 │ missed_gals │ 0.0548926 │ 0.026253 │ 0.0286396 │ 0.00991759 │ 419 │ │ 4 │ mag_r │ 0.201173 │ 0.349337 │ -0.148164 │ 0.0190963 │ 419 │ │ 5 │ color_ug │ 1.24038 │ 0.700412 │ 0.539969 │ 0.0536563 │ 372 │ │ 6 │ color_gr │ 0.326111 │ 0.217032 │ 0.109079 │ 0.0163301 │ 416 │ │ 7 │ color_ri │ 0.203236 │ 0.169912 │ 0.0333236 │ 0.0157859 │ 419 │ │ 8 │ color_iz │ 0.29284 │ 0.129721 │ 0.163118 │ 0.0198482 │ 419 │ │ 9 │ gal_fracdev │ 0.271263 │ 0.256713 │ 0.0145497 │ 0.0191935 │ 174 │ │ 10 │ gal_ab │ 0.188427 │ 0.135029 │ 0.053398 │ 0.0101751 │ 174 │ │ 11 │ gal_scale │ 1.91571 │ 2.39779 │ -0.482082 │ 0.485313 │ 174 │ │ 12 │ gal_angle │ 17.4348 │ 12.3188 │ 5.11597 │ 1.62281 │ 80 │

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/jeff-regier/Celeste.jl/issues/146#issuecomment-204785601