Closed kbarbary closed 6 years ago
Are you using multiple threads from running the accuracy benchmarks? It still finished for me in < 10 minutes, with 4 threads. It would be helpful to see the scores for all the sources on this branch. Partly, to make sure the N
column remains about the same. Also hard to compare if master has scores for all sources, and this branch just has half.
Yeah makes sense about the N
column. How do I select number of threads? Environment variable?
Looking at CPU utilization, it seems like I'm using 2 out of 4 cores. When I ran on master, it took multiple hours. I'll try again though.
Yep, environment variable:
export JULIA_NUM_THREADS=4
And also set
export OMP_NUM_THREADS=1
since you want all the parallelism to be from the Julia threads, not from OMP threads.
Would you test the accuracy benchmarks with synthetic imagery too, comparing master to this branch? It's more reliable in some sense than the stripe 82 scores because we have real ground truth for it.
There's not much to it, just two steps, and then it's the same as running the accuracy benchmarks with real data:
To generate the ground truth catalog (synthetic)
$ ./write_ground_truth_catalog_csv.jl prior
To generate the image:
$ generate_synthetic_field.jl <ground truth CSV>
Merging #719 into master will increase coverage by
1.02%
. The diff coverage is100%
.
@@ Coverage Diff @@
## master #719 +/- ##
==========================================
+ Coverage 80.11% 81.13% +1.02%
==========================================
Files 37 37
Lines 4013 3987 -26
==========================================
+ Hits 3215 3235 +20
+ Misses 798 752 -46
Impacted Files | Coverage Δ | |
---|---|---|
src/AccuracyBenchmark.jl | 80.64% <ø> (-0.21%) |
:arrow_down: |
src/GalsimBenchmark.jl | 100% <100%> (ø) |
:arrow_up: |
src/ParallelRun.jl | 94.09% <100%> (+9.74%) |
:arrow_up: |
src/joint_infer.jl | 76.8% <0%> (+3.2%) |
:arrow_up: |
src/mcmc/mcmc_misc.jl | 53.88% <0%> (+5.69%) |
:arrow_up: |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update b4721f9...4950a96. Read the comment docs.
Stripe 82 above updated with full results.
Synthetic results:
│ Row │ N │ first │ field │
├─────┼─────┼───────────┼─────────────────┤
│ 1 │ 131 │ 0.10687 │ missed_stars │
│ 2 │ 340 │ 0.0294118 │ missed_galaxies │
│ 3 │ 471 │ 0.163471 │ position │
│ 4 │ 471 │ 0.18367 │ flux_r_mag │
│ 5 │ 471 │ 1.33026 │ flux_r_nmgy │
│ 6 │ 48 │ 8.39943 │ gal_angle_deg │
│ 7 │ 91 │ 0.191313 │ gal_frac_dev │
│ 8 │ 91 │ 0.141528 │ gal_axis_ratio │
│ 9 │ 91 │ 0.71273 │ gal_radius_px │
│ 10 │ 471 │ 0.432434 │ color_ug │
│ 11 │ 471 │ 0.169713 │ color_gr │
│ 12 │ 471 │ 0.128667 │ color_ri │
│ 13 │ 471 │ 0.198651 │ color_iz │
│ Row │ N │ first │ field │
├─────┼─────┼────────────┼─────────────────┤
│ 1 │ 122 │ 0.0491803 │ missed_stars │
│ 2 │ 301 │ 0.00664452 │ missed_galaxies │
│ 3 │ 423 │ 0.10236 │ position │
│ 4 │ 423 │ 0.118397 │ flux_r_mag │
│ 5 │ 423 │ 0.338704 │ flux_r_nmgy │
│ 6 │ 44 │ 6.27761 │ gal_angle_deg │
│ 7 │ 79 │ 0.200732 │ gal_frac_dev │
│ 8 │ 79 │ 0.117555 │ gal_axis_ratio │
│ 9 │ 79 │ 0.523512 │ gal_radius_px │
│ 10 │ 423 │ 0.336727 │ color_ug │
│ 11 │ 423 │ 0.139262 │ color_gr │
│ 12 │ 423 │ 0.109304 │ color_ri │
│ 13 │ 423 │ 0.162469 │ color_iz │
Without looking at the data, my guess here is that we're missing 10% of the faintest sources (buried in the noise) with this branch, and doing better on most benchmarks as a result of their absence in the comparison.
That makes sense. Is there a setting we can change to lower the 10% "false negative" rate as much as we want, if we're willing to tolerate more false positives?
On the real data, how does SEP compare to Primary, in terms of false positives and false negatives? If you run score_predictions.jl
twice, once to score the primary predictions and once to score the SEP predictions, is the N column about the same (or better) for SEP?
That makes sense. Is there a setting we can change to lower the 10% "false negative" rate as much as we want, if we're willing to tolerate more false positives?
Yes, it is currently hard-coded to something that seems like a reasonable trade-off, but I plan to make it configurable as part of a larger configuration refactor. It is the 1.3
in this line
sep_catalog = SEP.extract(calpixels, 1.3; noise=SEP.global_rms(bkg))
On the real data, how does SEP compare to Primary, in terms of false positives and false negatives? If you run score_predictions.jl twice, once to score the primary predictions and once to score the SEP predictions, is the N column about the same (or better) for SEP?
│ Row │ N │ first │ field │
├─────┼─────┼───────────┼─────────────────┤
│ 1 │ 130 │ 0.0769231 │ missed_stars │
│ 2 │ 357 │ 0.0392157 │ missed_galaxies │
│ 3 │ 487 │ 0.268329 │ position │
│ 4 │ 487 │ 0.181464 │ flux_r_mag │
│ 5 │ 487 │ 1.1297 │ flux_r_nmgy │
│ 6 │ 106 │ 17.1374 │ gal_angle_deg │
│ 7 │ 213 │ 0.263027 │ gal_frac_dev │
│ 8 │ 213 │ 0.201823 │ gal_axis_ratio │
│ 9 │ 213 │ 1.30143 │ gal_radius_px │
│ 10 │ 375 │ 1.02586 │ color_ug │
│ 11 │ 478 │ 0.340056 │ color_gr │
│ 12 │ 486 │ 0.203456 │ color_ri │
│ 13 │ 482 │ 0.389609 │ color_iz │
│ Row │ N │ first │ field │
├─────┼─────┼───────────┼─────────────────┤
│ 1 │ 156 │ 0.121795 │ missed_stars │
│ 2 │ 479 │ 0.0417537 │ missed_galaxies │
│ 3 │ 635 │ 0.292428 │ position │
│ 4 │ 475 │ 0.170777 │ flux_r_mag │
│ 5 │ 475 │ 1.68524 │ flux_r_nmgy │
│ 6 │ 158 │ 17.8022 │ gal_angle_deg │
│ 7 │ 292 │ 0.2123 │ gal_frac_dev │
│ 8 │ 292 │ 0.179954 │ gal_axis_ratio │
│ 9 │ 292 │ 0.832056 │ gal_radius_px │
│ 10 │ 490 │ 0.693602 │ color_ug │
│ 11 │ 627 │ 0.222191 │ color_gr │
│ 12 │ 634 │ 0.163294 │ color_ri │
│ 13 │ 632 │ 0.224448 │ color_iz │
Great!
I'll update the wiki page shortly.
This substitutes in
infer_init_new()
, which uses SEP to find sources for the initialization catalog, forinfer_init()
which used an SDSS catalog.In the accuracy benchmarks,
run_celeste_on_field.jl
now takes no arguments and uses source detection by default. The option to initialize from a catalog is still there, with--initialization-catalog <filename>
.master
(first = primary, second = celeste prediction)
This branch
(first = primary, second = celeste prediction)
(note that I only ran about half the sources due to the long run time of the benchmarks at the moment)
Future work: SEP definitely picks up lots of bright and saturated stars that are already masked in the SDSS catalog. We'll want some heuristic to mask these. They should not affect benchmarks though, as only sources in both the coadd and prediction catalogs are included in benchmarks.
Closes #157.