search

Keck-DataReductionPipelines / MosfireDRP

http://keck-datareductionpipelines.github.io/MosfireDRP
10 stars 13 forks source link

many bizarre errors running a fresh install of the DRP- here is one: #150

Open csteidel opened 2 years ago

csteidel commented 2 years ago

This one may have to do with Macosx (I am running 10.14.6) - is there any easy way to turn off the use of multiple processors for this step? I cannot get past this point for some reason: 2021-09-28 10:42:53,917 - Wavelength. fit_lambda_helper - INFO: S18] TOOK: 162 s 2021-09-28 10:43:06,694 - Wavelength. fit_lambda_helper - INFO: S20] TOOK: 151 s 2021-09-28 10:43:25,730 - Wavelength. fit_lambda_helper - INFO: S21] TOOK: 135 s 2021-09-28 10:43:26,290 - Wavelength. fit_lambda_helper - INFO: S23] TOOK: 109 s 2021-09-28 10:43:37,241 - Wavelength. fit_lambda_helper - INFO: S24] TOOK: 113 s multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, *kwds)) File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar return list(map(args)) File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/site-packages/MOSFIRE-1.0.dev0-py3.6.egg/MOSFIRE/Wavelength.py", line 392, in fit_lambda_helper sol_1d = center_solutions[slitidx]["sol_1d"] TypeError: 'int' object is not subscriptable """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "Driver.py", line 36, in Wavelength.fit_lambda(maskname, band, obsfiles, obsfiles,waveops) File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/site-packages/MOSFIRE-1.0.dev0-py3.6.egg/MOSFIRE/Wavelength.py", line 365, in fit_lambda solutions = p.map(fit_lambda_helper, list(range(len(bs.ssl)))) File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/multiprocessing/pool.py", line 266, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/multiprocessing/pool.py", line 644, in get raise self._value TypeError: 'int' object is not subscriptable

joshwalawender commented 2 years ago

@csteidel My guess is that this has to do with a failure of the wavelength fit rather than a multiprocessing problem. Are you reducing long slit data or mask? If you point me to the data, I can take a look.

If you want to turn off multicore processing, you can edit one line in the Wavelength.py file. Set multicore = False here:

    if multicore:
        p = Pool()
        solutions = p.map(fit_lambda_helper, list(range(len(bs.ssl))))
        p.close()
    else:
        solutions = map(fit_lambda_helper, list(range(len(bs.ssl))))

I believe this is line 361 or thereabouts.

csteidel commented 2 years ago

Yes, I got some very strange errors on this mask, called q2343nb1 in J band, whose data can be found in /s/sdata1300/mosfire6/2021sep28/ - it is about 4 hours of integration, so the fact that this mask will not run and all of the other from the same night do, is puzzling...

BTW, in order to install the DRP I had to comment out a line in the setup.py :

setup(name=PACKAGENAME, version=VERSION, description=DESCRIPTION, scripts=scripts, requires=['astropy'], install_requires=['astropy'], provides=[PACKAGENAME], author=AUTHOR, author_email=AUTHOR_EMAIL, license=LICENSE, url=URL, long_description=LONG_DESCRIPTION, cmdclass=cmdclassd, zip_safe=False,

use_2to3=True,

  entry_points=entry_points,
  **package_info

) because that particular package raised a fatal error. It seems that it might be incompatible with something else in the environment?

Thanks, Chuck

On Tue, Sep 28, 2021 at 1:03 PM Josh Walawender @.***> wrote:

@csteidel https://github.com/csteidel My guess is that this has to do with a failure of the wavelength fit rather than a multiprocessing problem. Are you reducing long slit data or mask? If you point me to the data, I can take a look.

If you want to turn off multicore processing, you can edit one line in the Wavelength.py file. Set multicore = False here:

if multicore:
    p = Pool()
    solutions = p.map(fit_lambda_helper, list(range(len(bs.ssl))))
    p.close()
else:
    solutions = map(fit_lambda_helper, list(range(len(bs.ssl))))

I believe this is line 361 or thereabouts.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/150#issuecomment-929582234, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKJFJUARKCPEDOG5P3RAS3UEINRJANCNFSM5E57JFWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

csteidel commented 2 years ago

I think you are right, Josh. It fails every time in trying to do the initial wavelength filt for slits 12 and 17 -- and it has to do with the conversion from slit number to pixel it seems, something involving NaN's ; this is the output accompanying the failure to fit slit 12; the slit edge tracing looks OK, so I am at a loss.

2021-09-28 14:55:11,890 - Wavelength. call - INFO: n 12608.163776 0.0676179450109 2021-09-28 14:55:11,890 - Wavelength. nextobject - INFO: Saving to: lambda_center_coeffs_wave_stack_J_m210928_0124-0245.npy 2021-09-28 14:55:11,900 - Wavelength. setup - INFO: CSU slits [] acting as slit number [] /Users/ccs/anaconda3/envs/mospy/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2909: RuntimeWarning: Mean of empty slice. out=out, *kwargs) Traceback (most recent call last): File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/site-packages/matplotlib/cbook/init.py", line 216, in process func(args, *kwargs) File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/site-packages/MOSFIRE-1.0.dev0-py3.6.egg/MOSFIRE/Wavelength.py", line 2031, in call actions_mouseless[kp](x, y) File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/site-packages/MOSFIRE-1.0.dev0-py3.6.egg/MOSFIRE/Wavelength.py", line 1902, in nextobject self.setup() File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/site-packages/MOSFIRE-1.0.dev0-py3.6.egg/MOSFIRE/Wavelength.py", line 1692, in setup self.extract_pos = self.bs.science_slit_to_pixel(self.slitno) File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/site-packages/MOSFIRE-1.0.dev0-py3.6.egg/MOSFIRE/CSU.py", line 309, in science_slit_to_pixel return self.csu_slit_to_pixel(np.median(slits)) File "/Users/ccs/anaconda3/envs/mospy/lib/python3.6/site-packages/MOSFIRE-1.0.dev0-py3.6.egg/MOSFIRE/CSU.py", line 295, in csu_slit_to_pixel pixel = np.int(y0 - (slit -1) 44.22) ValueError: cannot convert float NaN to integer

On Tue, Sep 28, 2021 at 1:03 PM Josh Walawender @.***> wrote:

@csteidel https://github.com/csteidel My guess is that this has to do with a failure of the wavelength fit rather than a multiprocessing problem. Are you reducing long slit data or mask? If you point me to the data, I can take a look.

If you want to turn off multicore processing, you can edit one line in the Wavelength.py file. Set multicore = False here:

if multicore:
    p = Pool()
    solutions = p.map(fit_lambda_helper, list(range(len(bs.ssl))))
    p.close()
else:
    solutions = map(fit_lambda_helper, list(range(len(bs.ssl))))

I believe this is line 361 or thereabouts.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/150#issuecomment-929582234, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKJFJUARKCPEDOG5P3RAS3UEINRJANCNFSM5E57JFWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

csteidel commented 2 years ago

Turns out to be an even stranger problem than I thought- for a while I thought it was an edge tracing thing, so I fooled it into using an H band flat instead of J band find the edges, but when it gets to slits 12 and 17 it sill fails - almost as if there is a corrupted mask file - it somehow does not know about the the slits, or is confused about them.

I would really appreciate it if someone can give this a try and see if you have the same problem. Have never seen anything like this... Thanks, Chuck

On Tue, Sep 28, 2021 at 1:03 PM Josh Walawender @.***> wrote:

@csteidel https://github.com/csteidel My guess is that this has to do with a failure of the wavelength fit rather than a multiprocessing problem. Are you reducing long slit data or mask? If you point me to the data, I can take a look.

If you want to turn off multicore processing, you can edit one line in the Wavelength.py file. Set multicore = False here:

if multicore:
    p = Pool()
    solutions = p.map(fit_lambda_helper, list(range(len(bs.ssl))))
    p.close()
else:
    solutions = map(fit_lambda_helper, list(range(len(bs.ssl))))

I believe this is line 361 or thereabouts.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/150#issuecomment-929582234, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKJFJUARKCPEDOG5P3RAS3UEINRJANCNFSM5E57JFWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

nickkonidaris commented 2 years ago

It looks like there's a bookkeeping error. Could you take a look at the FITS header with the mask design and see if there's something weird in the header? I reckon the problem would be obvious (like, a chunk of the header was not properly written).

csteidel commented 2 years ago

Nothing that I have been able to find; weirdly, the pixelflat has the correct range of Y for each slit in the header. The failure is happening when it tries to extract the middle of the slit for initial wavelength calibration.

On Tue, Sep 28, 2021 at 10:37 PM Nick Konidaris @.***> wrote:

It looks like there's a bookkeeping error. Could you take a look at the FITS header with the mask design and see if there's something weird in the header? I reckon the problem would be obvious (like, a chunk of the header was not properly written).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/150#issuecomment-929851176, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKJFJXAIGMQHHBM46M44JLUEKQYXANCNFSM5E57JFWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

CallumWitten commented 2 years ago

I have the same error for a different set of observations. Is there any update on how this was solved?