search

PyWiFeS / pipeline

The Python data reduction pipeline for WiFeS
6 stars 26 forks source link

Fix ascii save when lam_array has NaNs #30

Closed timothyfrankdavies closed 7 months ago

timothyfrankdavies commented 7 months ago

On some test runs I saw the following error: ValueError: could not broadcast input array from shape (1276,) into shape (1277,)

The stack trace pointed to line 272 of wifes_calib.py: save_data[:,0] = filtered_lam_array

We recently changed filtered_lam_array to exclude NaN values, but that results in a smaller array. save_data was created with the original size, so if save mode was 'ascii' lam_array had NaNs, we'd hit the error.

The error occurred on some nod and shuffle data. @CMartinezLombilla wasn't able to reproduce the error, but I think the error makes sense and the changes should be safe.

I've tested the new version and the error doesn't occur.

CMartinezLombilla commented 7 months ago

Just to let you know, Felipe got this same error in his tests today, so probably what happened to me was that my data did not have NaN values in the flux so this issue didn’t raise up. Good to have different datasets!

timothyfrankdavies commented 7 months ago

Just to let you know, Felipe got this same error in his tests today, so probably what happened to me was that my data did not have NaN values in the flux so this issue didn’t raise up. Good to have different datasets!

Thanks for the confirmation, good to know it's not just me!

If Felipe needs to test without the error, we could try merging https://github.com/PyWiFeS/pipeline/pull/31, it's updated with the latest changes.

CMartinezLombilla commented 7 months ago

No worries, we replicated your changes in src/wifes_calib.py (we didn't commit anything) and it worked, so that's a good sign for the merging.