search

Keck-DataReductionPipelines / MosfireDRP

http://keck-datareductionpipelines.github.io/MosfireDRP
10 stars 13 forks source link

MOSFIRE DRP handle step not examining .fits files #140

Closed askoch785 closed 5 years ago

askoch785 commented 5 years ago

I am going through the MOSFIRE DRP tutorial on macOS 10.14.5 Mojave, and the pipeline is failing during the handle step. I am using the example data and have unzipped all .fits files as directed. However, when I run the "mospy handle" step, the pipeline returns "Examining 0 files."

Input:

mospy handle /Users/alex/Documents/UMaine/Research/Data/DRP_Test_Case_Hband/2012sep10/*fits

Output:

2019-07-17 13:39:39,528 - MosfireDrpLog. - INFO: 3.6.3 |Anaconda, Inc.| (default, Dec 5 2017, 17:30:25) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] 2019-07-17 13:39:39,529 - MosfireDrpLog. - INFO: python version = 3.6.3 2019-07-17 13:39:39,529 - MosfireDrpLog. - INFO: numpy version = 1.13.3 2019-07-17 13:39:39,642 - MosfireDrpLog. - INFO: matplotlib version = 2.1.1 2019-07-17 13:39:39,642 - MosfireDrpLog. - INFO: astropy version = 2.0.3 2019-07-17 13:39:39,642 - MosfireDrpLog. - INFO: ccdproc version = 1.3.0 2019-07-17 13:39:39,646 - mospy_handle. - INFO: Examining 0 files

These are the versions of Anaconda and Python I am using per the tutorial instructions Anaconda Version: conda 4.6.11 Python Version: Python 3.6.3

joshwalawender commented 5 years ago

@askoch785 Can you verify the contents of that directory? For example, if the files are compressed (as they come downloaded from KOA),handle won't find them.

askoch785 commented 5 years ago

@joshwalawender I have unzipped all files in the directory. Some files contained in the directory are:

m120910_0163.fits m120910_0164.fits m120910_0165.fits

Please let me know if there is any other info I can provide.

joshwalawender commented 5 years ago

@askoch785 This is strange behavior. I can't see anything wrong with the command, but the software is not finding the files. Do you have the latest (2018) release of the MOSFIRE DRP software? My only idea is that there is something wrong in the path that was typed. Can you double check the contents of the directory by running the following two commands back to back and showing us the output?

ls /Users/alex/Documents/UMaine/Research/Data/DRP_Test_Case_Hband/2012sep10/*fits

and

mospy handle /Users/alex/Documents/UMaine/Research/Data/DRP_Test_Case_Hband/2012sep10/*fits

Make sure that the correct conda environment has been activated before running mospy of course.

thanks, Josh

askoch785 commented 5 years ago

@joshwalawender I entered the ls command, and received a list of all .fits files in the directory. Note that the "U Maine" folder has a space in its name.

Input: ls /Users/alex/Documents/U\ Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/*fits

Output: /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0163.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0164.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0165.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0166.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0167.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0168.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0169.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0170.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0171.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0172.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0173.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0174.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0175.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0176.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0410.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0411.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0412.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0413.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0414.fits /Users/alex/Documents/U Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/m120910_0415.fits

Next, I entered in the mospy handle command (after activating the mospy_2018_macos environment), and received the same output from the first post of this issue.

Input: mospy handle /Users/alex/Documents/U\ Maine/Research/Data/DRP_Test_Case_Hband/2012sep10/*fits

Output: 2019-07-22 11:09:10,600 - MosfireDrpLog. - INFO: 3.6.3 |Anaconda, Inc.| (default, Dec 5 2017, 17:30:25) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] 2019-07-22 11:09:10,600 - MosfireDrpLog. - INFO: python version = 3.6.3 2019-07-22 11:09:10,600 - MosfireDrpLog. - INFO: numpy version = 1.13.3 2019-07-22 11:09:10,735 - MosfireDrpLog. - INFO: matplotlib version = 2.1.1 2019-07-22 11:09:10,735 - MosfireDrpLog. - INFO: astropy version = 2.0.3 2019-07-22 11:09:10,736 - MosfireDrpLog. - INFO: ccdproc version = 1.3.0 2019-07-22 11:09:10,740 - mospy_handle. - INFO: Examining 0 files

For simplicity, I removed the space in the UMaine folder name, and tried everything again. I received the same list of .fits files, but when I tried the "mospy handle" command, it seems to have worked correctly!

Input: mospy handle /Users/alex/Documents/UMaine/Research/Data/DRP_Test_Case_Hband/2012sep10/*fits

Output: 2019-07-22 11:13:13,734 - MosfireDrpLog. - INFO: 3.6.3 |Anaconda, Inc.| (default, Dec 5 2017, 17:30:25) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] 2019-07-22 11:13:13,734 - MosfireDrpLog. - INFO: python version = 3.6.3 2019-07-22 11:13:13,734 - MosfireDrpLog. - INFO: numpy version = 1.13.3 2019-07-22 11:13:13,760 - MosfireDrpLog. - INFO: matplotlib version = 2.1.1 2019-07-22 11:13:13,760 - MosfireDrpLog. - INFO: astropy version = 2.0.3 2019-07-22 11:13:13,760 - MosfireDrpLog. - INFO: ccdproc version = 1.3.0 2019-07-22 11:13:13,761 - mospy_handle. - INFO: Examining 20 files 2019-07-22 11:13:14,490 - mospy_handle. handle_file_list - INFO: Writing 6 files to MOSFIRE_DRP_MASK/2012sep10/H/Flat.txt 2019-07-22 11:13:14,490 - mospy_handle. handle_file_list - INFO: Writing 7 files to MOSFIRE_DRP_MASK/2012sep10/H/Offset_2.5.txt 2019-07-22 11:13:14,491 - mospy_handle. handle_file_list - INFO: Writing 7 files to MOSFIRE_DRPMASK/2012sep10/H/Offset-2.5.txt

Could it be that the space in the folder name caused the error? As far as I can tell that's the only thing that changed. I renamed the folder to "U Maine" one more time to replicate the error, and it occurred again. Renaming the folder to "UMaine" worked again, so the error can be replicated.

Thank you for the help!

Alex

joshwalawender commented 5 years ago

@askoch785 Yeah, that's it. There's a difference in how the command line handles spaces in strings (needs an escape character \) versus python (allows spaces in strings). I'll try to look in to better handling of this. I got trained to avoid spaces in paths by IRAF, so I've not encountered this before.