Staged protoDC2 image generation

[jchiang87]

Since we will not be able to generate the full protoDC2 data set before the Sprint Week, we would like to produce an initial subset that would still be useful for the Working Groups in the near term.

We would like to do the full 25 sq degrees, so downscoping for the initial stage would mean fewer bands and a shorter observation time frame.

Questions:

• How many visits (or sensor-visits) seem feasible to have done by Sprint Week? @TomGlanzman?
• How many and which bands? @cwwalter?
• What depth? 1 year? @cwwalter?

[cwwalter]

It is hard to answer this question before we first address what the WGs would like to do in the sprint week. What studies do people envision that they can't address with DC1 or the catalog? Knowing this will help answer the question what exactly would be useful. Do we have any information on this? For example, What is setting the 25 degrees?

Generally, I would think 6 bands is very important since we haven't tried this yet both from an analysis and CI perspective.

[katrinheitmann]

Weak lensing studies cannot be done with DC1. One weak lensing person (maybe Mike J.?) said two bands are sufficient for some reasonable tests. For the sprint week, one main goal is to make sure the full pipeline is set up correctly. Another aim is to ensure that all the data formats are well under control and we can set up cross-catalog queries. The size 25 sq degrees was set to have something that is small enough so that it can be easily handled by CatSim, that it is big enough so that some statistics can be measured and validated, that we could use the flat sky approximation for weak lensing, .... many more reasons -- but mostly, small enough to be easily handled and big enough to enable validation tests.

[cwwalter]

OK, if we mostly want to make sure the full pipelines work then I would think all six bands are important. It might be useful to think about how many exposures are necessary to test stack functionality? Maybe 10 minimum?

Also, do you want it to be dithered?

[jchiang87]

We have run the L2 pipeline on Twinkles data for all six LSST bands at 10 year depth for a DDF. I don't think the criteria should primarily be stack functionality for these data. I think we should try to provide something that the WGs would find useful for Sprint Week, so that we can understand and exercise catalog access, queries, etc. from their perspectives.

Don't we already know whether we want protoDC2 to be dithered? I would have assumed yes. There are still fundamental items that I don't see specified anywhere that are required before anyone can generate instance catalogs or images. What opsim db are we using? What object classes will be included in the instance catalogs? Just non-variable objects? Is a dithering implementation available now? If not, then I say we pick an opsim db and go with no dithering.

[cwwalter]

We have run the L2 pipeline on Twinkles data for all six LSST bands at 10 year depth for a DDF. I don't think the criteria should primarily be stack functionality for these data. I think we should try to provide something that the WGs would find useful for Sprint Week, so that we can understand and exercise catalog access, queries, etc. from their perspectives.

Oh, OK, I thought if we were testing the pipelines having the six bands would be important, and that it might also be useful from a CI perspective just to think about how to handle all of the different bands since we will need to do this for DC2.

Don't we already know whether we want protoDC2 to be dithered? I would have assumed yes. There are still fundamental items that I don't see specified anywhere that are required before anyone can generate instance catalogs or images. What opsim db are we using? What object classes will be included in the instance catalogs? Just non-variable objects? Is a dithering implementation available now? If not, then I say we pick an opsim db and go with no dithering.

Yes, good questions. I think we need to have a bit more of a conversation about what protoDC2 is and exactly what it is for (including this scaled down version for the hack week). It sounds like (at least for imSim) we can't make it feature complete relative to what we would like to do by the spring. We are also still waiting to hear from the WGs and making a decision for the cadence including how to handle transient sources. So this would argue for us focusing on what we want to test and running with what is ready with an understanding that this output will be replaced in the future. So, I think a clear list of what we want to test/study will help with this. Katrin said above that we needed at least two bands for WL and:

" For the sprint week, one main goal is to make sure the full pipeline is set up correctly. Another aim is to ensure that all the data formats are well under control and we can set up cross-catalog queries."

I think for this 6 bands non-dithered should be fine for now to start to test those things. I think there is a good chance we will find basic issues we will need to fix so having everything perfect on this first go is not critical.

But, I am of course interested in everyone one else's opinion. If it is important that we use whatever output we make now (for NERSC allocation reasons or something) that is of course important too.

[katrinheitmann]

We have now established a sprint at the December meeting jointly with the WLPipe group for running the downscaled DC2 catalogs end-to-end through the weak lensing pipeline (this would even lead to cosmology parameters if it all works). Chihway sent the following message: "Usually weak lensing is done on r band or riz bands, but I think looking at all bands will be good if that’s easy to get."

Then we should discuss the PhoSim options for now since that's what we are going to do for the Sprint week. My understanding is that all features that we want to do for DC2 are complete in PhoSim, right? So the features are according to issue #4: "BF, tree-rings and saturation, and perhaps fringing and cosmic rays." Is that correct?

Why would we do non-dithered?

If the LSS group has time during the Sprint week, they could also run some simple tests on the catalogs.

I think for now we should not worry about the transients, though I can ping Rahul again if people think that that should be included as well.

@rmandelb Rachel: can you help us make the list of what is needed for a useful WL test? I am also happy to ask Chihway already directly but it would be good to have a first plan to show her that we agree can be done. Thanks!

[rmandelb]

Are you asking what's needed on the extragalactic catalog side, what types of images/bands/area for the image sims, what should be run on the DM side, all of the above, something else entirely?

[katrinheitmann]

I am basically repeating Jim's question from the start of the issue, with particular view on WL and running the catalogs through WLPipe as the test case:

• How many visits (or sensor-visits) are needed for a meaningful test and seem feasible to have done by Sprint Week?
• How many and which bands?
• What depth? 1 year?

The extragalactic catalog is done for the Sprint week, so we have what we have. The question of the bands were kind of answered by Chihway ("Usually weak lensing is done on r band or riz bands, but I think looking at all bands will be good if that’s easy to get.") but I wanted to double check with you.

Lastly, dithering and which effects to switch on in PhoSim (BF, tree-rings and saturation, and perhaps fringing and cosmic rays which is the list from Chris in issue #4)

Jim: were there other questions?

Thanks very much! Katrin

On 11/8/17 9:38 AM, Rachel Mandelbaum wrote:

Are you asking what's needed on the extragalactic catalog side, what types of images/bands/area for the image sims, what should be run on the DM side, all of the above, something else entirely?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/DC2_Repo/issues/33#issuecomment-342855706, or mute the thread https://github.com/notifications/unsubscribe-auth/AMQ9jJOdie77LgxJ7gDunYptgVNd3ojqks5s0cr4gaJpZM4QTtzI.

[jchiang87]

Since @danielsf will be generating the instance catalogs, I'm hoping he can weigh in with any items we might have missed that would be needed to make those files.

Regarding dithering: I think we should definitely include it, but my concern is that we don't have a suitable dithering implementation on hand. For DC1, we selected a region of the sky and a baseline opsim db file, then ran an afterburner code to generate the dithered visits. So I'm wondering what opsim db file should we use for protoDC2 and whether can we use the old dithering afterburner or is there something new that we should use?

[salmanhabib]

Is there some "standard" dithering implementation from the project folks? One would think there would at least be a straw man --

[rmandelb]

This is not a complete answer, but it has a few high-level points that I think we have to consider before I can give those details.

It depends what level of test you are trying to do. For example: if you want to do this test with a subset of bands (which I personally think is good: we should have >1 band but not all 6), then they would have to use the true redshifts since we cannot get photo-z. Using true redshifts means we'd need the capability to easily match DM outputs against the truth catalogs. Is that functionality we expect to have in place by the sprint week? If not, then we would need all 6 bands plus photo-z code, which seems challenging in other respects.

The other question we have to ask is whether we need more area than we really want to do for this test using images. The images of real galaxies have shape noise, so realistically for weak lensing to have decent S/N there has to be a decent area. (In contrast, with the extragalactic catalog we could imagine using just the lensing shear without the intrinsic galaxy shape to test the pipeline without shape noise.) So I tried to answer this question of the area required, to figure out whether an image-based WL test even makes sense at this stage. Our estimate is that 1-year depth images should give an effective source number density (with reasonable cuts) of ~12/arcmin^2. In the regime of shape noise-limited WL measurements (which the LSST survey will not be, but a small-area sim will be), the S/N for cosmic shear scales like sqrt(area) * neff.

For HSC we expect to have S/N for cosmic shear of around 25 in our 136 deg^2 with neff=24/arcmin^2. Let's say for this test of DC2, it's interesting even with S/N=10. Given our LSST Y1 neff=12/arcmin^2, we can roughly estimate the area needed for an interesting WL test (this is ignoring redshift scalings which will modify it slightly):

area needed = (HSC area) * (2 * (10/25))^2

where the 2 comes from the number density difference, and the 10/25 comes from the S/N we can accept for this test. So we need about 0.64 times the HSC area, which is around 90 deg^2.

I guess if it's 2 bands = 1/3 of the bands for DC2, and 90/300 of the area for DC2, and 1/10 of the time (Y1 instead of Y10), then that amount of imaging would correspond to 1% of DC2. So maybe that's not bad after all?

Anyway, if it's acceptable to do 2 bands to Y1 depth in a ~90 deg^2 area, and if we'll be able to match to the truth catalogs easily, then we could plan to do this WL test. And I will give you the rest of what you asked for, but first I wanted to check: do the basics sound feasible?

If not, the WLPipe team definitely has something they can do at sprint week (working from the extragalactic catalog would already enable an interesting test).

[rmandelb]

I just realized the initial issue in this thread said 25 deg^2, so maybe this test is not doable after all?

[rmandelb]

If we want to stick with what's doable for 25 deg^2, then I think it's pretty clear we cannot do the project of going all the way to cosmology. I did some thinking about what could be usefully done:

• A bunch of null tests for PSF model fidelity, and certain shear systematics. (The sensitivity of the tests will not be great, so this would mainly be a demonstration that we can get things all the way through the null test pipeline.) This would not require output vs. input catalog matching, and would operate purely on the DM outputs.

• If there is concern about shear conventions, we can use output vs. input catalog comparisons to do confirm that the shear conventions are all under control.

• Basic sanity check of the measured RMS ellipticity dispersion (as a function of magnitude, redshift, etc.) compared to that in HSC or DES.

Edited to add: I think that 2 bands at Y1 depth would be enough. We should have a few pointings (not just 1), but within that 25 deg^2 constraint it should be fine.

[danielsf]

@jchiang87 I am finalizing the InstanceCatalog generation code here

https://github.com/danielsf/gcr-catalogs/tree/sed_fitting

I've been backchanneling Eve any time I run across something I don't understand. So far, I think we can work with protoDC2 as it is, now. I will let you know here if there is an insurmountable problem.

[cwwalter]

If we want to stick with what's doable for 25 deg^2, then I think it's pretty clear we cannot do the project of going all the way to cosmology. I did some thinking about what could be usefully done:

Are there basic performance checks shapes etc that we should/could be doing now with the DC1 dataset with no external shear applied in r-band? I'm a bit worried that people are excited about testing the analysis pipelines but we are missing testing basic things we can do now and will also exercise people learning how to interface with data.

[cwwalter]

Is there some "standard" dithering implementation from the project folks? One would think there would at least be a straw man

Basically no. Dithering is added via an afterburner which adds a new column to the OpSim database. In the past Simon had done one and then for DC1 Humna and Eric designed a optimized one including applying a rotation on each filter change.

I guess if we are using the same OpSim database as before those columns are still there so we can use them for this small test. Then we also designed a way to reduce the unnecessary simulation for sensors that would give us uneven depth in a region that was chosen by the LSS group. You still get more visits than no dithering but the "wasted" sensors are removed. There are a couple ways of doing this.

This is one of the reasons I asked what we really wanted to test at the hack day. If we are making a small test that doesn't rely on the depth smoothing that dithering gives you then running with the standard OpSim non-dithered pointings will give you what you need with no extra work and no extra run factor for dithering and trimming issues.

[cwwalter]

Actually what the scheduler will really do about dithering is an interesting question. I can follow up on this in project meetings. @SimonKrughoff or @danielsf do you know anything about the plan / status of the work now?

[cwwalter]

BTW our 40 square degrees before is basically 4 undithered pointings. So it is a bit discretized and, as I mentioned, you need to think about if the depth uniformity is important when the dithering is included.

[rmandelb]

Are there basic performance checks shapes etc that we should/could be doing now with the DC1 dataset with no external shear applied in r-band?

There are PSF modeling checks that could be done, I am sure.

Were any shear estimation algorithms run in DC1?

[cwwalter]

Were any shear estimation algorithms run in DC1?

Yes, I think so; we ran the entire standard DM measurement algorithm suite. Let me check the actual variable list...

[cwwalter]

Looks like we have at least SDSS shape and HSM. Take a peek here:

https://github.com/LSSTDESC/SSim_DC1/blob/master/Notebooks/Butler%20access%20demo.ipynb

[rmandelb]

Yep. Any of the tests I listed in https://github.com/LSSTDESC/DC2_Repo/issues/33#issuecomment-342885951 could be done with DC1, it would seem.

[rmandelb]

Actually now that I think of it I had already recommended the PSF model tests for DC1 (but I think I didn't realize regaussianization had been run, so I didn't recommend any galaxy shape tests).

[TomGlanzman]

There is now an operational workflow at NERSC running phoSim v3.7.1, quickbackground configuration on the first three DC1 visits. Due to queue latency, full results may not be available until ~Thursday. But some initial results are dribbling out of the system which may allow for a realistic estimate of how much CPU and elapsed time will be required to generate [[staged]proto]DC2.

The first result is that using the DC1 sky, 34 sensor-visits were completed on a single KNL node in ~177 minutes. There were 34 instances of phoSim per node running with 8-threads each. (Recall that DC1 imposed a mag=10 limit on bright stars.) This result is good for a fully-loaded KNL node in a realistic production context. This represents an average raytrace throughput of 1 sensor-visit every 5.2 minutes per node. Unless there are some super-scaling I/O issues, this should be a good planning number until final catalogs are available for test.

By comparison, DC1 sensor-visits averaged 560 minutes (with long tails on the distribution) running 8 instances per node and 8-threads each. The raytrace throughput on Haswell was more like one sensor-visit every 70 minutes per node, or a factor of 13.5 less (per node). This factor accounts for all changes in hardware, node loading, phosim code fixes & improvements, and phosim configuration differences (e.g., quickbackground).

A limitation of the KNL result is that it is still to early to measure operational inefficiencies - which could be significant. Also, I've not yet had the opportunity to run this new workflow at scale...with 100s of nodes, so there may be scaling surprises waiting to be discovered.

---

With regard to preparing data for the Sprint Week, consider what might be done in a single (very ideal) day of a ramped-up workflow running on 500 KNL nodes.

(1 sensor-visit / 5.2 min / node) (500 nodes) (1440 min/day) * (1 focal plane / 189 sensors) = 732 focal planes/day

This early in the game a derating factor of 5-10 is probably prudent.

The trick will be assembling the missing pieces (instance catalogs, dithered map of sensors to simulate, agreeing on the desired phoSim configuration), AND getting jobs to run at NERSC in a timely way. In general, it can take days for jobs to start at NERSC so this becomes a critical factor given the limited time.

• Tom

[salmanhabib]

Tom's runs are done with what would nominally the best number of threads (34 * 8=272, the total number of hardware threads on the KNL). From the previous KNL performance numbers I got from Glenn and John, I had estimated 3.3 minutes per job, so I am a little puzzled by the 5.2 minutes we are seeing. What's the difference between these two cases?

[cwwalter]

@TomGlanzman Are these the visits we are tracking from #19 or something else?

[TomGlanzman]

@salmanhabib Glenn's estimate was from running an (arbitrary) "standard" test configuration (as I did with my tests as well), while today's report uses the DC1 sky and config. Also Glenn's numbers are averages of multiple tests - although I do not know the causes for such a distribution. I used phoSim v3.7.1 and compiled it (along with fftw and cfitsio) with the Intel compiler. Another difference is admittedly small, but Glenn timed the entire phosim run, whereas I am presenting results for only the raytrace steps.

@cwwalter Not necessarily. These are the first three DC1 visits plus a fourth partial visit that was not part of DC1, but represented very long execution times. These four constitute the first shake-down cruise for the new task. Perhaps they could be used for issue #19 but Seth had posted a list of "interesting" visits so had thought to try running a couple of those. If these three suffice, that would be great!

[cwwalter]

Not necessarily. These are the first three DC1 visits plus a fourth partial visit that was not part of DC1, but represented very long execution times. These four constitute the first shake-down cruise for the new task. Perhaps they could be used for issue #19 but Seth had posted a list of "interesting" visits so had thought to try running a couple of those. If these three suffice, that would be great!

OK thanks. I think we want to do the list from Seth because it will sample over all of the different sky levels with the moon etc. I just wanted to clarify what was done here. Great to see first results.

[cwwalter]

Forgot to add, when we do those lets keep the results in that issue for bookkeeping purposes.

[cwwalter]

BTW just to be clear are "the first three DC1 visits" really the 1st three, or the first three from Seth's list?

[TomGlanzman]

The first three: obshistID 40336, 40337, 40338

[cwwalter]

OK thanks. I asked because Seth's first three have the maximum moon brightness.

[salmanhabib]

@TomGlanzman Thanks, Tom, I am happy the numbers are making much more sense now. We'll do some more tests on the Argonne KNLs. I am pretty sure we can eventually improve the performance over what is being seen now. I am willing to bet a large sum of money that there will be no scaling issues. Any takers ;-)?

[cwwalter]

I am willing to bet a large sum of money that there will be no scaling issues. Any takers ;-)?

Hi Salman, I was confused by your comment and I realized we have been probably been using the same language for different things. I have been saying I want to know the scaling factor. But, I just mean the amount of run time relative to what we did for DC1 in a realistic environment so we really know how long things will take and how they perform. There can't be an issue with that. I just want to know what the number actually is.

But, I think when you say scaling you are probably talking about issues with threads and CPUs right? No, I will not bet against you all :) I'm quite confident you will be able to make things run as efficiently as theoretically possible. That would be bad bet on my part!

[johnrpeterson]

Chris, I think we have that number already. Basically 3.3 jobs/min on a fully loaded node seems robust on KNL and 1.8 on Haswell. And the test above was worst case with moonlight and was 5.2 jobs/min, and i previously said on another thread it might be ~30% slower when the moon is out, so that seems all consistent within errors. It could even be faster on Argonne due to the more memory per core allowing more concurrent jobs and reducing setup times further.

If you want to compare to DC1 numbers, we weren't estimating fully loaded node production rates during the data challenge (which was the right metric), but we have already estimated the speed up between 25 and 40 times faster.

[salmanhabib]

@cwwalter @johnrpeterson Tom Uram is running the PhoSim tests today at Argonne -- more when Tom sends out the results.

[sethdigel]

Here's a quick comparison of two images from obshistID 40336 from Tom's original DC1 run (task DC1-phoSim-2) and the newly generated version of the same visit (DC1-phoSim-3). The files are phoSim e-images, lsst_e_40336_f2_R01_S02_E000.fits.gz. The images are almost the same (median ratio new/old = 1.0045). The most obvious difference is that the DC1-phoSim-3 image has a fairly bright star that is not in the original. (I tried to get ds9 to save a blink comparison animation, but the resulting file was blank.). Also, a difference map shows that the images are not quite in registration.

DC1-phoSim-2 DC1-phoSim-3

DC1-phoSim-3 minus DC1-phoSim-2 (detail)

[cwwalter]

Hi Seth,

Great thanks. Can you tell us what the run conditions for 40336, 40337, 40338 were so we know what the factor of 13 corresponds to? I mean was the moon up etc.

[cwwalter]

The most obvious difference is that the DC1-phoSim-3 image has a fairly bright star that is not in the original. (I tried to get ds9 to save a blink comparison animation, but the resulting file was blank.).

Is this because the magnitude cap was applied in one and not the other?

Also, a difference map shows that the images are not quite in registration.

Hmm... I wouldn't have expected that. Anyone have any ideas?

[jchiang87]

Also, a difference map shows that the images are not quite in registration.

Hmm... I wouldn't have expected that. Anyone have any ideas?

I think phosim puts in random pointing errors. Maybe using the same seed would fix that?

[sethdigel]

Tom just pointed out that I had misunderstood the task names/locations for the respective runs. The comparison that I posted above was between two generations of DC1-era phoSim runs (DC1-phoSim-2 was a prototype for the main DC1 runs, which were DC1-phoSim-3). Sorry for the noise. I'll make the same comparison using the new visit (DC2-phoSim-1).

[TomGlanzman]

A spreadsheet describing the current prototype DC2 workflow has been created here: https://docs.google.com/spreadsheets/d/17-EUgaDMtfgQLt84WquxeAjycqJaqHJZS_jZqrQXOVw/edit?usp=sharing

Also, @jchiang87 the obshistID has typically been used as the random number seed so for DC1-phosim and DC2-phosim workflows, so they should be the same for various runs of the same visit. But I do not know if the random number generators will always produce exactly the same sequence on different architectures or even different compilers.

[sethdigel]

Here is a comparison between the DC1 image (from the DC1-phoSim-3 task) and the corresponding image from DC2 (the DC2-phoSim-1 task). Both are e-images lsst_e_40336_f2_R01_S02_E000.fits.gz. The DC2 image has a brighter background than the DC1. That they are different in this regard is probably not a surprise.

The stars and galaxies appear to be in registration. For the most part they seem to be brighter in the DC1 image, in the sense that they appear as deficits in the DC2-minus-DC1 image below. The middle of the color range in the difference image is set to be the difference of the medians of the images (~250 electrons).

A few exceptions are evident - the brighter stars appear as positive deficits. And a few sources (galaxies?) look more diffuse and brighter in these images. The image below shows the lower-right quadrant of the difference image.

Some metadata for the visit: moonAlt = -19.3 deg moonBright = 0.0 vSkyBright = 21.52 airmass = 1.081 The magnitude of the brightest star is 10.0 (meaning that it probably was brighter in the CatSim catalog)

[TomGlanzman]

Just a reminder that both the DC1 (phoSim v3.6.1) and this prototype DC2 (phoSim v3.7.1 + quickbackgroun) workflows used the following phoSim override:

zenith_v 21.8

From recent discussion, it sounds like this should be removed for production. Don't know if this might have contributed to the higher background @sethdigel noted.

[sethdigel]

The bright diffuse source in the lower right of the images I posted yesterday is this one (in the phoSim trimcat file for this visit)

object 98201600202779 87.4486737 -30.7574175 19.1857491 galaxySED/Inst.32E07.02Z.spec.gz 0.216197893 0 0 0 0 0 sersic2d 10.8018637 3.36279058 251.691818 1 CCM 0.600000024 3.0999999 CCM 0.120991951 3. The centroid file says that this source has 95165 photons, making it relatively bright.

It seems to be just a big galaxy. The first argument after 'sersic2d' is the 'size of axis 1 in arcseconds', and this galaxy has the fourth largest value of this parameter among the 68000 in the trimcat file. I have not located the corresponding trimcat file for the DC1 run, but I don't have any reason to expect the parameters for the galaxy to have been different.

Its speckled appearance seems to just relate to its large size and perhaps also how the photons are bunched in the simulation. Each of the individual bright pixels scattered around the galaxy has about 600 more electrons than the adjacent background pixels (which have ~400 electrons).

(this is from the DC2-phoSim-1 e-image; linear scaling from 268 to 1000 electrons)

[TomGlanzman]

@sethdigel the catalog used for DC1 is exactly the same as that used for this DC2 test. I've not seen any changes to the 'trim' code, so would expect those files to be identical.

(You will not find the DC1 trim files - they get created in $SCRATCH and are automatically purged by NERSC. Normally, phosim.py removes all traces of its 'work/' directory immediately after the job completes. In the case of DC2-phoSim-1, phosim.py was deconstructed and I intentionally preserve work/ -- modulo the NERSC purging cycle.)

[rmandelb]

Hmmm. I'm not sure what the DM stack would make of individual bright pixels. If it's smaller than a PSF, and it doesn't look quite like a cosmic ray either, then what happens to that detection?

We don't actually care if a few rare bright galaxies look wonky, but the bright pixels are scattered quite far around this one, so I wonder what it's doing to everything around it.

[cwwalter]

@johnrpeterson should comment but I thought the bunching/optimization should only apply to the background photons not the astrophysical sources. So it seems strange that blocks of 200 photons are showing up in individual pixels.

[SimonKrughoff]

Hmmm. I'm not sure what the DM stack would make of individual bright pixels. If it's smaller than a PSF, and it doesn't look quite like a cosmic ray either, then what happens to that detection?

@rmandelb I'm pretty sure all those individual bright pixels will be called cosmic rays. The algorithm essentially looks for edges that are sharper than the PSF.

This will cause problems because there is logic in the CR algorithm that throws an exception if there are too many CRs found. The assumption is that if there are more than X CRs something has gone terribly wrong.

We have run into this in the past when we have assumed an initial PSF that is too broad for the data. In that case, it's OK that an exception was thrown. In the above case, I can imagine it causing the exception even on otherwise passable data.

[cwwalter]

@johnrpeterson Is it possible that an algorithm designed for the bright star optimization is also being applied to this galaxy since it has near 100K electrons?