search

rmjarvis / Piff

PSFs In the Full FOV. A software package for modeling the point-spread function (PSF) across the full field of view (FOV). Documentation:
http://rmjarvis.github.io/Piff/
Other
58 stars 13 forks source link

Unit Tests #54

Closed areshernandez closed 7 years ago

areshernandez commented 7 years ago

I wanted to clarify if I have a couple of points correct with regard to my understanding of the "test_optatmo.py" unit test.

  1. It seems the main incomplete testing function is "def test_optical_fit()." It looks like the reason it is incomplete is because the misalignment errors from the minuit fitting are not trustworthy and a way should be devised to check the minuit errors in a more direct way to see if they match up with the errors we currently get when running this unit test.

2.The only other incomplete testing function is "def create_synthetic_optatmo()." It seems this is incomplete because stars are drawn with it only with the optical PSF and not the atmospheric PSF. At this point, I'm assuming nothing needs to be added to this function until after some new features are added to the atmospheric PSF to make it ready for testing.

rmjarvis commented 7 years ago

I guess these questions are for @cpadavis. You seem to be talking about tests on the combined_wavefront_psf branch, which seems to be his work...

areshernandez commented 7 years ago

When I ran test_optatmo.py with the new changes to the "create_synthetic_optatmo()" function I got a new error:

File "test_optatmo.py", line 395, in
test_optical_fit()
File "/afs/slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2
result = f(*args, **kwargs)
File "test_optatmo.py", line 365, in test_optical_fit
optpsf.fit(stars_clean, wcs, pointing, logger=logger)
File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 430, in fit
logger=logger)
File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 455, in _minuit_fit
self._minuit.migrad(ncall=self.kwargs['max_iterations'])
File "_libiminuit.pyx", line 331, in iminuit._libiminuit.Minuit.migrad (iminuit/_libiminuit.cpp:5099)
File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 579, in _fit_func
shapes = self._measure_shapes(self.drawStarList(self._fit_stars),)# logger=logger)
File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 634, in drawStarList
stars = self.decaminfo.pixel_to_focalList(stars)
File "/u/ec/aresh/Piff-wavefront/piff/des/decaminfo.py", line 321, in pixel_to_focalList
return [Star(self.pixel_to_focal_stardata(star.data), star.fit) for star in stars]
File "/u/ec/aresh/Piff-wavefront/piff/des/decaminfo.py", line 290, in pixel_to_focal_stardata
stardata['chipnum'], stardata['x'], stardata['y'])
File "/u/ec/aresh/Piff-wavefront/piff/star.py", line 682, in getitem
return self.properties[key]
KeyError: 'chipnum'

Looking in the decaminfo.py file it looks like there is trouble with the pixel to focal x and focal y conversion and that the "chipnum" property is missing. Is this due to the WCS coordinate conversion issue or is it because we are using Simple PSF rather than SingleChip?

cpadavis commented 7 years ago

Neither, actually. If it were that, then the code wouldn't have run even before your changes. It looks like when the atmospheric piece of the combined PSF draws the stars, it doesn't preserve any of the properties in the star data except for x/y and u/v. I have fixed that and pushed it to the combined_wavefront_psf branch. Give it a pull and see if that fixes your problem :)

---chris

On Mon, Jul 10, 2017 at 5:35 PM Ares Hernandez notifications@github.com wrote:

When I ran test_optatmo.py with the new changes to the "create_synthetic_optatmo()" function I got a new error:

File "test_optatmo.py", line 395, in test_optical_fit() File "/afs/ slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2 result = f(*args, **kwargs) File "test_optatmo.py", line 365, in test_optical_fit optpsf.fit(stars_clean, wcs, pointing, logger=logger) File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 430, in fit logger=logger) File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 455, in _minuit_fit self._minuit.migrad(ncall=self.kwargs['max_iterations']) File "_libiminuit.pyx", line 331, in iminuit._libiminuit.Minuit.migrad (iminuit/_libiminuit.cpp:5099) File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 579, in _fit_func shapes = self._measure_shapes(self.drawStarList(self._fit_stars),)# logger=logger) File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 634, in drawStarList stars = self.decaminfo.pixel_to_focalList(stars) File "/u/ec/aresh/Piff-wavefront/piff/des/decaminfo.py", line 321, in pixel_to_focalList return [Star(self.pixel_to_focal_stardata(star.data), star.fit) for star in stars] File "/u/ec/aresh/Piff-wavefront/piff/des/decaminfo.py", line 290, in pixel_to_focal_stardata stardata['chipnum'], stardata['x'], stardata['y']) File "/u/ec/aresh/Piff-wavefront/piff/star.py", line 682, in getitem return self.properties[key] KeyError: 'chipnum'

Looking in the decaminfo.py file it looks like there is trouble with the pixel to focal x and focal y conversion and that the "chipnum" property is missing. Is this due to the WCS coordinate conversion issue or is it because we are using Simple PSF rather than SingleChip?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-314288813, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5MvJdiY5nxrdDfkyFvOJ45ol2AQrks5sMsNigaJpZM4ORngF .

areshernandez commented 7 years ago

With the new updates, the error is no longer there. However, I now receive a different error that comes from the test_optical_fit() function:

  File "test_optatmo.py", line 395, in <module>
    test_optical_fit()
  File "/afs/slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2
    result = f(*args, **kwargs)
  File "test_optatmo.py", line 378, in test_optical_fit
    assert abs(misalignment[key] - misalignment_fit[key]) < 2 * misalignment_fit_errors[key], '{0} Failed Optical Fit. Parameter: {1}, Fit: {2} +- {3}'.format(key, misalignment[key], misalignment_fit[key], misalignment_fit_errors[key])
AssertionError: z04d Failed Optical Fit. Parameter: -0.5, Fit: -1.99999999998 +- 2.40979307846e-05

This error seems to appear where the misalignment fitting is done. Is this likely due to the problem with the number of parameters used? If so, should I try to remove the z04d parameter that seems to be causing problems? I'm going to attempt that next.

aaronroodman commented 7 years ago

Ares,

it looks like the fit is just returning a bad value - should find -0.5 for z04d (thats focus delta) but finds -2.

Does the fit seem to run? I believe you should see it looking for its minimum and should see the value of each parameter cycle through reasonable values.

Aaron

On Jul 11, 2017, at 2:29 PM, Ares Hernandez notifications@github.com<mailto:notifications@github.com> wrote:

With the new updates, the error is no longer there. However, I now receive a different error that comes from the test_optical_fit() function:

File "test_optatmo.py", line 395, in test_optical_fit() File "/afs/slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.pyhttp://slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2 result = f(*args, *kwargs) File "test_optatmo.py", line 378, in test_optical_fit assert abs(misalignment[key] - misalignment_fit[key]) < 2 misalignment_fit_errors[key], '{0} Failed Optical Fit. Parameter: {1}, Fit: {2} +- {3}'.format(key, misalignment[key], misalignment_fit[key], misalignment_fit_errors[key]) AssertionError: z04d Failed Optical Fit. Parameter: -0.5, Fit: -1.99999999998 +- 2.40979307846e-05

This error seems to appear where the misalignment fitting is done. Is this likely due to the problem with the number of parameters used? If so, should I try to remove the z04d parameter that seems to be causing problems? I'm going to attempt that next.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/rmjarvis/Piff/issues/54#issuecomment-314532203, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEEa8z5XyNaiHby6sGFeKNXII67ZHjdbks5sM78EgaJpZM4ORngF.

Prof. Aaron Roodman Chair Dept. of Particle Physics & Astrophysics SLAC National Accelerator Laboratory Stanford University

SLAC National Accelerator Laboratory E-mail: roodman@slac.stanford.edumailto:roodman@slac.stanford.edu 2575 Sand Hill Rd. Phone: 650-926-2705 MS 29 Menlo Park, CA 94025 URL: http://www.slac.stanford.edu/~roodman

cpadavis commented 7 years ago

I think you were adding in the atmospheric PSF, but that test doesn't expect there to be an additional atmospheric piece...

On Tue, Jul 11, 2017 at 11:36 AM aaronroodman notifications@github.com wrote:

Ares,

it looks like the fit is just returning a bad value - should find -0.5 for z04d (thats focus delta) but finds -2.

Does the fit seem to run? I believe you should see it looking for its minimum and should see the value of each parameter cycle through reasonable values.

Aaron

On Jul 11, 2017, at 2:29 PM, Ares Hernandez <notifications@github.com mailto:notifications@github.com> wrote:

With the new updates, the error is no longer there. However, I now receive a different error that comes from the test_optical_fit() function:

File "test_optatmo.py", line 395, in test_optical_fit() File "/afs/ slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py< http://slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py>", line 78, in f2 result = f(*args, *kwargs) File "test_optatmo.py", line 378, in test_optical_fit assert abs(misalignment[key] - misalignment_fit[key]) < 2 misalignment_fit_errors[key], '{0} Failed Optical Fit. Parameter: {1}, Fit: {2} +- {3}'.format(key, misalignment[key], misalignment_fit[key], misalignment_fit_errors[key]) AssertionError: z04d Failed Optical Fit. Parameter: -0.5, Fit: -1.99999999998 +- 2.40979307846e-05

This error seems to appear where the misalignment fitting is done. Is this likely due to the problem with the number of parameters used? If so, should I try to remove the z04d parameter that seems to be causing problems? I'm going to attempt that next.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub< https://github.com/rmjarvis/Piff/issues/54#issuecomment-314532203>, or mute the thread< https://github.com/notifications/unsubscribe-auth/AEEa8z5XyNaiHby6sGFeKNXII67ZHjdbks5sM78EgaJpZM4ORngF>.

Prof. Aaron Roodman Chair Dept. of Particle Physics & Astrophysics SLAC National Accelerator Laboratory Stanford University

SLAC National Accelerator Laboratory E-mail: roodman@slac.stanford.edu mailto:roodman@slac.stanford.edu 2575 Sand Hill Rd. Phone: 650-926-2705 <(650)%20926-2705> MS 29 Menlo Park, CA 94025 URL: http://www.slac.stanford.edu/~roodman

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-314534206, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5KM9rJ4Eud7Ac_eQfPiHqKNAxH9oks5sM8DIgaJpZM4ORngF .

areshernandez commented 7 years ago

After test_optatmo.py finished running, it went through test_optical_fit() and did not cause any problems like usual. However, when it got to the optical test in test_fit() it failed in the same way that test_optical_fit() used to fail (which was due to feeding optatmo_stars_clean from the full PSF into the fit() function inside the OpticalWavefrontPSF(PSF) class instead of just opt_stars_clean):

  File "test_optatmo.py", line 483, in <module>

  File "/afs/slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2
    result = f(*args, **kwargs)
  File "test_optatmo.py", line 373, in test_fit
    # construct results
AssertionError: z04d Failed Optical Fit inside full test_fit() function. Parameter: -0.5, Fit: -1.99999999996 +- 1.92922281265e-05

In test_fit(), however, I fed optatmo_stars_clean into the fit() function inside the OptAtmoPSF(PSF) class, rather than the fit() function inside the OpticalWavefrontPSF(PSF) class. Is there likely a bug interfering with the fit() function inside the OptAtmoPSF(PSF) class that is causing it to behave like the fit() function from the OpticalWavefrontPSF(PSF) class?

cpadavis commented 7 years ago

So it might be what we talked about earlier today -- there might not be enough stars sampling the focal plane distribution, and so a parameter degeneracy develops. But: another thing might be that we aren't allowing the model the ability to compensate for the atmosphere during the fit.

The thinking is this: when you make the opt + atmo stars, you are including the atmospheric piece in making the stars. Here, this means adding a constant atmosphere and ellipticity, or possibly a 1st order polynomial across the focal plane. But, looking at your code, the optical piece has no way of fitting that (since we said it must have a fixed r0 == atmospheric seeing == affects the size of the stars), so it leans heavy on the defocus to get the larger stars. The solution is to allow r0, g1, and g2 to be fitted in the optical part, but then "turn them off" and let the atmosphere fit that piece later. I thought this was what the model was doing, but I might not have done a good job coding that up. I'll take a look at it this weekend, and we can chat on Monday about this idea.

---chris

On Fri, Jul 14, 2017 at 5:47 PM Ares Hernandez notifications@github.com wrote:

After test_optatmo.py finished running, it went through test_optical_fit() and did not cause any problems like usual. However, when it got to the optical test in test_fit() it failed in the same way that test_optical_fit() used to fail (which was due to feeding optatmo_stars_clean from the full PSF into the fit() function inside the OpticalWavefrontPSF(PSF) class instead of just opt_stars_clean):

File "test_optatmo.py", line 483, in

File "/afs/slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2 result = f(*args, **kwargs) File "test_optatmo.py", line 373, in test_fit

construct results

AssertionError: z04d Failed Optical Fit inside full test_fit() function. Parameter: -0.5, Fit: -1.99999999996 +- 1.92922281265e-05

In test_fit(), however, I fed optatmo_stars_clean into the fit() function inside the OptAtmoPSF(PSF) class, rather than the fit() function inside the OpticalWavefrontPSF(PSF) class. Is there likely a bug interfering with the fit() function inside the OptAtmoPSF(PSF) class that is causing it to behave like the fit() function from the OpticalWavefrontPSF(PSF) class?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-315497557, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5JKkStxjZThNRz9LzeYGbhBhPwvcks5sOAw_gaJpZM4ORngF .

areshernandez commented 7 years ago

I have narrowed down that the reason why the graphs for fitted_opt_stars are empty is because something goes wrong in the second line below:

opt_stars_drawn = optpsf.drawStarList(opt_stars_clean)
fitted_opt_shapes = optpsf._measure_shapes(stars=opt_stars_drawn)

I have successfully printed optpsf._measure_shapes(stars=opt_stars_clean), so I know there is nothing wrong with opt_stars_clean. I have printed the focal_x and focal_y coordinates of opt_stars_drawn, so I don't think there is anything wrong with opt_stars_drawn either. However, fitted_opt_shapes is a large list filled with "nan" when I print it.

It thus looks like there may be a bug inside of _measure_shapes() that only happens when I feed it opt_stars_drawn. Do you think this is the case? If so, is there an alternative method of retrieving the shapes besides _measure_shapes() that I can use?

cpadavis commented 7 years ago

Have you looked at the drawn stars? You can get the image for the i-th star via

opt_stars_drawn[i].image.array

and then you can plot the image with matplotlib imshow. On Wed, Jul 19, 2017 at 16:36 Ares Hernandez notifications@github.com wrote:

I have narrowed down that the reason why the graphs for fitted_opt_stars are empty is because something goes wrong in the second line below:

opt_stars_drawn = optpsf.drawStarList(opt_stars_clean) fitted_opt_shapes = optpsf._measure_shapes(stars=opt_stars_drawn)

I have successfully printed optpsf._measure_shapes(stars=opt_stars_clean), so I know there is nothing wrong with opt_stars_clean. I have printed the focal_x and focal_y coordinates of opt_stars_drawn, so I don't think there is anything wrong with opt_stars_drawn either. However, fitted_opt_shapes is a large list filled with "nan" when I print it.

It thus looks like there may be a bug inside of _measure_shapes() that only happens when I feed it opt_stars_drawn. Do you think this is the case? If so, is there an alternative method of retrieving the shapes besides _measure_shapes() that I can use?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-316551547, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5AeCsCmZMuwjVdaxS38N9myRyZe5ks5sPpLkgaJpZM4ORngF .

areshernandez commented 7 years ago

I see a small yellow square inside a purple background. Does this mean the images from opt_stars_drawn are corrupted somehow? In other words, are the star image normally large yellow ellipses in a purple background? If so, could this indicate that there is a bug inside of drawStarList() causing this problem?

cpadavis commented 7 years ago

hmmm I would say something is going wrong :)

This is in test_fit, right? where optpsf is the optical piece of the combined psf fit, right? Remember how, after the fit, we turn off the optical psf's constant size and ellipticity portions (since that will be covered by the atmospheric psf)? It might be when you are making those stars using only the optical psf that that piece is turned off. So since r0 is turned off, then the psf never gets scattered by the atmosphere, and you get effectively space quality images == basically only in one pixel. This means you should do two things:

On Wed, Jul 19, 2017 at 5:23 PM Ares Hernandez notifications@github.com wrote:

I see a small yellow square inside a purple background. Does this mean the images from opt_stars_drawn are corrupted somehow?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-316560461, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5AxXWKBu3Iot7KFzU2jxfspykjnkks5sPp3xgaJpZM4ORngF .

areshernandez commented 7 years ago

I tried putting r0, g1, and g2 back in before drawing the stars:

    optpsf.fitter_kwargs['fix_r0'] = False
    optpsf.fitter_kwargs['fix_g1'] = False
    optpsf.fitter_kwargs['fix_g2'] = False
    optpsf.model.kolmogorov_kwargs = {'r0':optpsf.backup_shapes[0]}
    optpsf.model.g1 = optpsf.backup_shapes[1]
    optpsf.model.g2 = optpsf.backup_shapes[2]
    optpsf._update_psf_params(r0=optpsf.backup_shapes[0], g1=optpsf.backup_shapes[1], g2=optpsf.backup_shapes[2])
    opt_stars_drawn = optpsf.drawStarList(opt_stars_clean)
    optpsf._update_psf_params(r0=None, g1=None, g2=None)
    optpsf.fitter_kwargs['fix_r0'] = True
    optpsf.fitter_kwargs['fix_g1'] = True
    optpsf.fitter_kwargs['fix_g2'] = True
    optpsf.model.kolmogorov_kwargs = {}
    optpsf.model.g1 = None
    optpsf.model.g2 = None
    fitted_opt_shapes = optpsf._measure_shapes(stars=opt_stars_drawn)

However, after doing this I got a new error:

  File "test_optatmo.py", line 889, in <module>
    test_fit()
  File "/afs/slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2
    result = f(*args, **kwargs)
  File "test_optatmo.py", line 593, in test_fit
    opt_stars_drawn = optpsf.drawStarList(opt_stars_clean)
  File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 644, in drawStarList
    stars = [self.model.draw(star) for star in stars]
  File "/u/ec/aresh/Piff-wavefront/piff/optical_model.py", line 258, in draw
    prof = self.getProfile(star.fit.params)
  File "/u/ec/aresh/Piff-wavefront/piff/optical_model.py", line 227, in getProfile
    atm = galsim.Kolmogorov(gsparams=self.gsparams, **self.kolmogorov_kwargs)
  File "/nfs/slac/g/ki/ki18/cpd/anaconda2/lib/python2.7/site-packages/galsim/base.py", line 593, in __init__
    "r0_500) must be specified for Kolmogorov")
TypeError: One of lam_over_r0, fwhm, half_light_radius, or lam (with r0 or r0_500) must be specified for Kolmogorov

It seems that I'm missing something called "r0_500," which I need in addition to r0, g1, and g2 in order to use optpsf.drawStarList(). If this is the case, is there a specific location where this "r0_500" value is stored?

I should add that I've checked optpsf.backup_shapes[0], optpsf.backup_shapes[1], and optpsf.backup_shapes[2] and compared them to optpsf.model.kolmogorov_kwargs['r0'], optpsf.model.g1, and optpsf.model.g2 and can confirm that they are the same, as they should be.

cpadavis commented 7 years ago

looking at it a little closer, you'll want to do this:

kolmogorov_keys = ('lam', 'r0', 'lam_over_r0', 'scale_unit', 'fwhm', 'half_light_radius', 'r0_500') optpsf.model.kolmogorov_kwargs = { key : optpsf.model.kwargs[key] for key in optpsf.model.kwargs if key in kolmogorov_keys } optpsf.model.kolmogorov_kwargs['r0'] = optpsf.backup_shapes[0]

if we constantly have to turn on and off the kolmogorov piece, this could be made into a function.

On Thu, Jul 20, 2017 at 12:25 PM Ares Hernandez notifications@github.com wrote:

I tried putting r0, g1, and g2 back in before drawing the stars:

optpsf.fitter_kwargs['fix_r0'] = False
optpsf.fitter_kwargs['fix_g1'] = False
optpsf.fitter_kwargs['fix_g2'] = False
optpsf.model.kolmogorov_kwargs = {'r0':optpsf.backup_shapes[0]}
optpsf.model.g1 = optpsf.backup_shapes[1]
optpsf.model.g2 = optpsf.backup_shapes[2]
optpsf._update_psf_params(r0=optpsf.backup_shapes[0], g1=optpsf.backup_shapes[1], g2=optpsf.backup_shapes[2])
opt_stars_drawn = optpsf.drawStarList(opt_stars_clean)
optpsf._update_psf_params(r0=None, g1=None, g2=None)
optpsf.fitter_kwargs['fix_r0'] = True
optpsf.fitter_kwargs['fix_g1'] = True
optpsf.fitter_kwargs['fix_g2'] = True
optpsf.model.kolmogorov_kwargs = {}
optpsf.model.g1 = None
optpsf.model.g2 = None
fitted_opt_shapes = optpsf._measure_shapes(stars=opt_stars_drawn)

However, after doing this I got a new error:

File "test_optatmo.py", line 889, in test_fit() File "/afs/slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2 result = f(*args, kwargs) File "test_optatmo.py", line 593, in test_fit opt_stars_drawn = optpsf.drawStarList(opt_stars_clean) File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 644, in drawStarList stars = [self.model.draw(star) for star in stars] File "/u/ec/aresh/Piff-wavefront/piff/optical_model.py", line 258, in draw prof = self.getProfile(star.fit.params) File "/u/ec/aresh/Piff-wavefront/piff/optical_model.py", line 227, in getProfile atm = galsim.Kolmogorov(gsparams=self.gsparams, self.kolmogorov_kwargs) File "/nfs/slac/g/ki/ki18/cpd/anaconda2/lib/python2.7/site-packages/galsim/base.py", line 593, in init "r0_500) must be specified for Kolmogorov") TypeError: One of lam_over_r0, fwhm, half_light_radius, or lam (with r0 or r0_500) must be specified for Kolmogorov

It seems that I'm missing something called "r0_500," which I need in addition to r0, g1, and g2 in order to use optpsf.drawStarList(). If this is the case, is there a specific location where this "r0_500" value is stored?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-316805009, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5M1Z7dVPU7nLMgjVvalmvXAy5Yenks5sP6mOgaJpZM4ORngF .

areshernandez commented 7 years ago

I have tried the following input misalignment terms to be used for test_fit():

  1. z04d=-0.5, z05d=0.1
  2. z04d=-0.5, z05d=0.1, z06d=0.06
  3. z04d=-0.5, z07d=0.01, z08d=0.03
  4. z04d=-0.5, z08d=0.03
  5. z04d=-0.5, z09d=0.1, z10d=0.1

So far, non have produced a good fit. Are there any possible sets of inputs that I haven't tried that you think would be more likely to succeed than others?

cpadavis commented 7 years ago

Hmm. I am honestly not sure what the best inputs are. How bad are the fits? How many stars are you using? Do the fits converge (as in, do they finish before they run out of available steps)? What do the residuals look like?

It's not the end of the world if there is a degeneracy between our input optical model and the final atmosphere -- it just means that that is something we shouldn't test.

Can you make plots of the following:

scans of the parameters vs chi2 in the neighborhood of the fitted value and the input value. Taking (1) as an example, say you fit for z04d and z05d -1 and 0.5. Then there would be one plot where you show the chi2 as a function of dz about the two fits of z04d, say dz from -0.2 to 0.2. And another one around z05d. Does that make sense? They should look like parabolas, probably with the incorrect value at a higher y intercept.

---chris

On Thu, Jul 27, 2017 at 12:09 PM Ares Hernandez notifications@github.com wrote:

I have tried the following input misalignment terms to be used for test_fit():

  1. z04d=-0.5, z05d=0.1
  2. z04d=-0.5, z05d=0.1, z06d=0.06
  3. z04d=-0.5, z07d=0.01, z08d=0.03
  4. z04d=-0.5, z08d=0.03
  5. z04d=-0.5, z09d=0.1, z10d=0.1

So far, non have produced a good fit. Are there any possible sets of inputs that I haven't tried that you think would be more likely to succeed than others?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-318458150, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5ESGceWKbD2v2IkFrM6hlTWa5_Psks5sSOBNgaJpZM4ORngF .

areshernandez commented 7 years ago

I'm not sure but I think you want me to make a graph of:

the fitted values I get for z04d vs the misalignment_fit_errors['z04d'] values that I get for the fits

Then I should expect this graph to be a parabola centered around z04d=-0.5 (the input value I always used for z04d). The graph should have a misalignment_fit_errors['z04d'] value of 0 there that increases parabolically as you move away from z04d=-0.5. Finally, I do the exact same thing but for z05d for all of the instances where I tested z05d.

Is this correct?

areshernandez commented 7 years ago

On a separate note. I have also tried the following:

  1. z04d=-0.5, z05d=0.1, z08d=0.03
  2. z04d=-0.5, z06d=0.06, z07d=0.01
  3. z04d=-0.5, z08d=0.03, z09d=0.1
  4. z04d=-0.5, z07d=0.01, z10d=0.1
  5. z04d=-0.5, z10d=0.1
  6. z04d=-0.5, z09d=0.1
  7. z04d=-0.5, z07d=0.01
  8. z04d=-0.5, z06d=0.06
  9. z04d=-0.5, z05d=0.1, z10d=0.1
  10. z04d=-0.5, z06d=0.06, z09d=0.1
  11. z04d=-0.5, z06d=0.06, z08d=0.03
  12. z04d=-0.5, z05d=0.1, z07d=0.01
  13. z04d=-0.5, z07d=0.01, z09d=0.1
  14. z04d=-0.5, z08d=0.03, z10d=0.1
  15. z04d=-0.5, z05d=0.1, z09d=0.1
  16. z04d=-0.5, z06d=0.06, z10d=0.1

None of these have produced good fits either. They all seem to be pretty bad fits just like (1). For example, for (21) the fit gives z04d=-0.01826, whereas for (1) the fit gives z04d=-0.1579, even though both are supposed to give -0.5

cpadavis commented 7 years ago

Sorry, let me try to be a bit clearer:

In OpticalWavefrontPSF there is a method called _fit_func. It takes in all the misalignment and shape parameters (r0, z04d, etc) and returns a chi squared. I want you to vary the value of your fit parameters near their input values, so if z04d = -0.5, I want to know the value returned by optpsf._fit_func for like 30 values of z04d from -0.7 to -0.2 -- call it -0.5 + dz, with dz = np.linspace(-0.2, 0.2, 30) If you also have a suboptimal fit result, I want a similar +- 0.2 range of values. Then I want you to plot the chi2 vs dz. For parameters you don't want to modify, you can pass in np.nan and they will remain unchanged.

Does that make sense?

On Thu, Jul 27, 2017 at 3:53 PM Ares Hernandez notifications@github.com wrote:

On a separate note. I have also tried the following:

  1. z04d=-0.5, z05d=0.1, z08d=0.03
  2. z04d=-0.5, z06d=0.06, z07d=0.01
  3. z04d=-0.5, z08d=0.03, z09d=0.1
  4. z04d=-0.5, z07d=0.01, z10d=0.1
  5. z04d=-0.5, z10d=0.1
  6. z04d=-0.5, z09d=0.1
  7. z04d=-0.5, z07d=0.01
  8. z04d=-0.5, z06d=0.06
  9. z04d=-0.5, z05d=0.1, z10d=0.1
  10. z04d=-0.5, z06d=0.06, z09d=0.1
  11. z04d=-0.5, z06d=0.06, z08d=0.03
  12. z04d=-0.5, z05d=0.1, z07d=0.01
  13. z04d=-0.5, z07d=0.01, z09d=0.1
  14. z04d=-0.5, z08d=0.03, z10d=0.1
  15. z04d=-0.5, z05d=0.1, z09d=0.1
  16. z04d=-0.5, z06d=0.06, z10d=0.1

None of these have produced good fits either. They all seem to be pretty bad fits just like (1). For example, for (21) the fit gives z04d=-0.01826, whereas for (1) the fit gives z04d=-0.1579, even though both are supposed to give -0.5

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-318508827, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5HWvPiC85Lin4UrOQOZ6Pi-9IitVks5sSRTXgaJpZM4ORngF .

areshernandez commented 7 years ago

The chi-squared graph for astigmatism-y also does not have a minimum at either the input value or the fitted value. The minimum is to the left of both, just like what was the case with defocus. Unfortunately, the chi-squared graphs for the temporary atmosphere parameters (size, first ellipticity, second ellipticity) failed because of a runtime error:

  File "test_optatmo.py", line 1289, in <module>
    test_fit()
  File "/afs/slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2
    result = f(*args, **kwargs)
  File "test_optatmo.py", line 1109, in test_fit
    chi_squareds_list.append(optpsf._fit_func(r0=0.1+dz_size_due_to_temporary_atmosphere[i], g1=misalignment['g1'], g2=misalignment['g2'], z04d=misalignment['z04d'], z04x=np.nan, z04y=np.nan, z05d=misalignment['z05d'], z05x=np.nan, z05y=np.nan, z06d=np.nan, z06x=np.nan, z06y=np.nan, z07d=np.nan, z07x=np.nan, z07y=np.nan, z08d=np.nan, z08x=np.nan, z08y=np.nan, z09d=np.nan, z09x=np.nan, z09y=np.nan, z10d=np.nan, z10x=np.nan, z10y=np.nan, z11d=np.nan, z11x=np.nan, z11y=np.nan))
  File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 593, in _fit_func
    shapes = self._measure_shapes(self.drawStarList(self._fit_stars),)# logger=logger)
  File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 654, in drawStarList
    stars = [self.model.draw(star) for star in stars]
  File "/u/ec/aresh/Piff-wavefront/piff/optical_model.py", line 261, in draw
    image = prof.drawImage(star.data.image.copy(), method='no_pixel', offset=offset)
  File "/nfs/slac/g/ki/ki18/cpd/anaconda2/lib/python2.7/site-packages/galsim/gsobject.py", line 1356, in drawImage
    added_photons = prof.drawFFT(imview, wmult)
  File "/nfs/slac/g/ki/ki18/cpd/anaconda2/lib/python2.7/site-packages/galsim/gsobject.py", line 1467, in drawFFT
    "If you can handle the large FFT, you may update gsparams.maximum_fft_size.")
RuntimeError: drawFFT requires an FFT that is too large: 7516.If you can handle the large FFT, you may update gsparams.maximum_fft_size.

Do you know what could be causing this runtime error? More importantly, do you know what the minimum in the chi-squared graphs being in the wrong place could imply?

cpadavis commented 7 years ago

So for the runtime error -- what parameters are you putting in? Be aware that r0 ranges from 0.1 to like 0.3 and if you put in a negative value or a value outside that range then it might run out of memory doing the Fourier transform.

The chi square is worrisome, so I want to check something. When you vary a parameter about its fitted value, are you putting in for the other parameters their fitted values as well? And when you vary a parameter about its input value, are you setting the other parameters to their input value? The thing we're trying to figure out is whether the fitted location or the input locations are coming out to be minima in our chi square. Let me know if you have more questions. On Tue, Aug 1, 2017 at 11:32 Ares Hernandez notifications@github.com wrote:

The chi-squared graph for astigmatism-y also does not have a minimum at either the input value or the fitted value. The minimum is to the left of both, just like what was the case with defocus. Unfortunately, the chi-squared graphs for the temporary atmosphere parameters (size, first ellipticity, second ellipticity) failed because of a runtime error:

File "test_optatmo.py", line 1289, in

test_fit() File "/afs/ slac.stanford.edu/u/ec/aresh/Piff-wavefront/tests/piff_test_helper.py", line 78, in f2 result = f(*args, **kwargs)

File "test_optatmo.py", line 1109, in test_fit chi_squareds_list.append(optpsf._fit_func(r0=0.1+dz_size_due_to_temporary_atmosphere[i], g1=misalignment['g1'], g2=misalignment['g2'], z04d=misalignment['z04d'], z04x=np.nan, z04y=np.nan, z05d=misalignment['z05d'], z05x=np.nan, z05y=np.nan, z06d=np.nan, z06x=np.nan, z06y=np.nan, z07d=np.nan, z07x=np.nan, z07y=np.nan, z08d=np.nan, z08x=np.nan, z08y=np.nan, z09d=np.nan, z09x=np.nan, z09y=np.nan, z10d=np.nan, z10x=np.nan, z10y=np.nan, z11d=np.nan, z11x=np.nan, z11y=np.nan)) File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 593, in _fit_func shapes = self._measure_shapes(self.drawStarList(self._fit_stars),)# logger=logger) File "/u/ec/aresh/Piff-wavefront/piff/optatmo.py", line 654, in drawStarList

stars = [self.model.draw(star) for star in stars]

File "/u/ec/aresh/Piff-wavefront/piff/optical_model.py", line 261, in draw image = prof.drawImage(star.data.image.copy(), method='no_pixel', offset=offset) File "/nfs/slac/g/ki/ki18/cpd/anaconda2/lib/python2.7/site-packages/galsim/gsobject.py", line 1356, in drawImage added_photons = prof.drawFFT(imview, wmult) File "/nfs/slac/g/ki/ki18/cpd/anaconda2/lib/python2.7/site-packages/galsim/gsobject.py", line 1467, in drawFFT "If you can handle the large FFT, you may update gsparams.maximum_fft_size.") RuntimeError: drawFFT requires an FFT that is too large: 7516.If you can handle the large FFT, you may update gsparams.maximum_fft_size.

Do you know what could be causing this runtime error? More importantly, do you know what the minimum in the chi-squared graphs being in the wrong place could imply?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-319457387, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5Ju-dHcettKwaRxewbt3j6q6Z64Iks5sT280gaJpZM4ORngF .

areshernandez commented 7 years ago

I changed the chi-squared graphs to make sure the input graphs used input parameters and the fitted graphs used fitted parameters. Most of the chi-squared graphs look parabolic now.

However, some appear off-center (the minimum looks like it's not at correct point). Also, the astigmatism-y graphs look like they might be quartic and the defocus graphs look like they might be a high-order polynomial.

Do you think this is likely due to the range of dz's tested being too large for the defocus and astigmatism-y graphs (I expanded the range to much more than -0.2 to 0.2 for those graphs)? Also, is there a reason why some of the chi-squared graphs might be off-center?

areshernandez commented 7 years ago

I tried Aaron's technique of keeping the temporary atmosphere without disabling it while keeping it at a fixed value; however, it did not fit. I am wondering if the problem is that the stars used to do all the fitting are the optatmo stars, so when the optical fit is attempting it fits to stars that were created with stuff not generated by the input optical terms (i.e. the atmospheric inputs), it doesn't fit correctly. Do you think this is likely the problem? If so, do you think I should change the OptAtmoPSF.fit() function, so that it does the optical fit with the opt stars first before fitting everything else with the optatmo stars?

areshernandez commented 7 years ago

I re-ran the fit using create_data() instead of create_synthetic_optatmo. It still hasn't finished fitting but it looks like it is converging on the wrong values for defocus and astigmatism-y. I'm not sure, but looking at create_data(), it looks like both psf and psf_clean have the temporary atmosphere disabled, even though it is supposed to be enabled for psf_clean (which is what I am using in the fit). Is there something inside optpsf.disable_atmosphere() that allows it to recognize that psf_clean should not have its temporary atmosphere wiped or should I comment out the line that seems to wipe it?:

psf_clean.optpsf.disable_atmosphere(logger=logger)

cpadavis commented 7 years ago

You should remove that line for psf_clean :) On Fri, Aug 11, 2017 at 18:06 Ares Hernandez notifications@github.com wrote:

I re-ran the fit using create_data() instead of create_synthetic_optatmo. It still hasn't finished fitting but it looks like it is converging on the wrong values for defocus and astigmatism-y. I'm not sure, but looking at create_data(), it looks like both psf and psf_clean have the temporary atmosphere disabled, even though it is supposed to be enabled for psf_clean (which is what I am using in the fit). Is there something inside optpsf.disable_atmosphere() that allows it to recognize that psf_clean should not have its temporary atmosphere wiped or should I comment out the line that seems to wipe it?:

psf_clean.optpsf.disable_atmosphere(logger=logger)

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/rmjarvis/Piff/issues/54#issuecomment-321947675, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsw5OoiLLYWCvVDKZXnSTSHr1VX6SPsks5sXPqwgaJpZM4ORngF .