WFPC2 File Completeness Checking Script

[acviana]

Create a script that checks the completeness of the WFPC2 data products.

Specs

• The script should be located in scripts/checks/check_filesystem_completeness.py
• The script should import the make_output_file_dict function we tested in #81 to construct the file dictionaries.
• The outputs should go to log folder
• The log folder and the WFPC2 data locations should be imported from the settings.yaml file.

  Steps

I suggest you run git status between these steps to see what git is and isn't tracking. Be sure to git add all your changes.

Updating the file tree

• Create scripts/production/
• Move scripts/run_imaging_pipeline.py to scripts/production/run_imaging_pipeline.py
• Create scripts/checks/
• Create scripts/checks/check_filesystem_completeness.py
• Add and commit your changes. Cite issue #80 in your commit message. Push your changes to the same feature branch as in #81. I'll comment on anything that needs changing.

  Importing settings from the yaml file

• In mtpipeline/get_settings change connection.yaml to settings.yaml
• In .gitignore change connection.yaml to settings.yaml
• Create a settings.yaml file at the root of the repository. This file will contain all the machine-specific settings the software will use. We don't want to hardcode these into the repo because they will be different for each user (or at least they could be). This will be ignored by git because it's listed in the .gitignore file.
• Create a file at the root of the repository called template_settings.yaml. Like the name suggest this is a template file that users can use to create a valid settings.yaml file with all the necessary keys. Use the following:

# Database Settings
# db_connection: mysql+pymysql://root:user@localhost/mtpipeline
# db_echo: False

# Email Settings
# email_switch: True
# contact_email: user@institute.edu

# File System Settings
# acs_input_path: /absolute/path/to/files
# acs_output_path: /absolute/path/to/files
# wfc3_input_path: /absolute/path/to/files
# wfc3_output_path: /absolute/path/to/files
# wfpc2_input_path: /absolute/path/to/files
# wfpc2_output_path: /absolute/path/to/files
# logging_path: /absolute/path/to/files

• Add from mtpipeline.get_settings import get_settings to check_filesystem_completeness.py
• Create a function called check_filesystem_completeness_main have it print the dictionary returned by get_settings to confirm that your settings are being correctly imported from yaml file.
• Add, commit, and push.

[acviana]

From your email:

Do I need to implement the function that is going to create the c0m.fits files?

No, the files have already been created by imaging_pipeline.py, we're just checking to make sure that all the output files were actually creates and nothing was skipped due to a crash or something.

Are these files supposed to be/to be stored at /astro/3/mutchler/mt/drizzled/? If so, where is this directory located? Do I need to create it?

All the subdirectories of /astro/3/mutchler/mt/drizzled/ should contain the c0m.fits files you are looking for. They should all have the format <target_name>_<proposal_id>.

In the settings.yaml file, where should I put '/astro/3/mutchler/mt/drizzled/'?

wfpc2_output_path

To import this directory from the settings.yaml file I will need to import yaml so that I can get the directory, right?

Yes, this function already exists. Add this to your script: from mtpipeline.get_settings import SETTINGS. SETTINGS is capitalized by convention because it is a global variable.

Which files are going to be in this directory? Only the c0m.fits or all of them including the corrected files, png files and so on.

/astro/3/mutchler/mt/drizzled/*_*/ should contain all the FITS images. All the PNG images should be in an /png/ subdirectory of each folder. Looking at the code now this /png/ subdirectory might not be accurate encoded in make_output_file_dict, but they still end up in the correct directory. I'll have to take a look at why that is before we can change anything with that.

From the moment I know where the files are stored, I need to implement a function that is going to check if all the required files were created, is that correct?

Yes, but we should talk a little bit about how you plan to do that.

[walyssonBarbosa]

I tried to run python checks/check_filesystem_completeness.py just to see how it works and it says permission denied. Will I need access permission to this directory?

Traceback (most recent call last):
  File "checks/check_filesystem_completeness.py", line 73, in <module>
    setup_logging()
  File "checks/check_filesystem_completeness.py", line 20, in setup_logging
    level = logging.INFO)
  File "/Users/wbarbosa/Downloads/Ureka/python/lib/python2.7/logging/__init__.py", line 1529, in basicConfig
    hdlr = FileHandler(filename, mode)
  File "/Users/wbarbosa/Downloads/Ureka/python/lib/python2.7/logging/__init__.py", line 902, in __init__
    StreamHandler.__init__(self, self._open())
  File "/Users/wbarbosa/Downloads/Ureka/python/lib/python2.7/logging/__init__.py", line 925, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/astro/3/mutchler/mt/logs/check_file_completeness/check_file_completeness_2014-06-03-11-21.log'

About how I plan to do the script, here is what I am thinking (I don't know if it is clear):

• For each initial c0m.fits file (e.g. u2eu0101t_c0m.fits), the script would create a dictionary with all other required files;
• A list with all of the directory files would be created;
• Then each file of the dictionary would be checked in that list to see if any of them is missing.

[acviana]

So checks/check_filesystem_completeness.py is the script I was hoping to replace. I didn't like like it because it reimplements the file name generation logic from make_output_dict. I also thought it would be a good first project for you because it would expose you to the repository and the output file system.

The reason you are getting an error is because it is trying to write to the log files area, which I haven't given you permission to access yet. I just fixed that.

Can you get the following to run?

> mkdir ~/notebooks
> cd ~/notebooks
> ipython notebook

An interactive IPython notebook window should appear.

[walyssonBarbosa]

Yes, I could see that it was creating the dict all over again.

Done!

[walyssonBarbosa]

As we are going to replace checks/check_filesystem_completeness.py and still we will need to log and check the missing files, can I use the setup_logging() and check_file() functions?

[acviana]

Yes, we are going to replace checks/check_filesystem_completeness.py. We're going to move setup_logging to it's own module in mtpipeline so other scripts can use it. check_file is one option, I'm looking at others as well.

Ok, I'm assuming that "done" means you got the IPython notebook to work. I like to use the notebook as an environment to do some rapid prototyping. You can do everything you can do in python in the notebook including importing modules. Try some different ways of checking to see if the files are present.

[walyssonBarbosa]

I moved setup_logging to a module named setup_logging.py in mtpipeline.

The files you want me to check are the ones located in '/astro/3/mutchler/mt/drizzled/`? And the code for the checking should be written in the IPython notebook, right?

[acviana]

Yes, that's correct.

[walyssonBarbosa]

Before I added the png files the total of files was 313257, now it's 541687.

I changed this line:

c0m_file_list = [filename for filename in all_fits_file_list if filename.split('/')[-1].split('_')[-1] == 'c0m.fits']

I removed len and added .split('_')[-1] because there was a file that ended with drz.fits. So I wanted to make sure the files ended with c0m.fits.

[walyssonBarbosa]

How should the png section in make_output_file_dict be changed?