[Documentation]: nircam jupyter notebook tutorial not able to download data

cleevesli commented 2 months ago

Check Existing Issues

[X] Yes, I have checked existing issues to ensure this problem hasn't already been reported.

Describe the issue or suggest an improvement

This is probably an issue with either jwst_download.py or maybe it's just me, but in the tutorial tutorial_NIRCam_reductions.ipynb fails to grab the data via the command "download_cmd = "jwst_download.py --propID 1386 -i nircam -l 700 --obsnums 1 2 3 "+ "--outsubdir data_nircam_hd65426/uncal --skip_propID2outsubdir -f uncal"" I also tried adding a MAST Token and it didn't work either. The error I get after adding a MAST Token is: Loading config file jwst_query.cfg WARNING! environment variable JWSTDOWNLOAD_OUTDIR in outrootdir not defined! Thus setting outrootdir to None... INSTRUMENT: nircam obsmode: [None] propID: 01386 obsnums: [1, 2, 3] INFO: MAST API token accepted, welcome Ilse Cleeves [astroquery.mast.auth] MJD range: 59848.59438081115 60548.69438082715 No obsmode given. Querying for all files for nircam. allowed filetype list: ['_uncal.fits'] select obsnum [1, 2, 3]: keeping 0 from 36

######################

Selected Products:

###################### ['proposal_id obsnum obsID parent_obsid obs_id sca visit dataproduct_type filetype calib_level size outfilename dl_code dl_str\n']

#################

Observations:

################# ['proposal_id obsnum obsid obs_id t_min t_exptime date_min instrument_name _uncal.fits\n']

##########################

Summary propID/obsnum:

########################## ['proposal_id obsnum _uncal.fits date_start\n'] ############## Nothing selected!!!! exiting...

URLs of the documentation pages (if applicable)

No response

Additional context or information

Sorry if this is the wrong spot. Wasn't sure if this was a 'bug' or just a me issue.

AarynnCarter commented 2 months ago

@cleevesli Don't download that file

AarynnCarter commented 2 months ago

@itbear-shu Please provide more information or I will delete that comment, I do not think it is an appropriate solution to this issue.

AarynnCarter commented 2 months ago

@cleevesli This is an issue with the current tutorial, we are developing newer ones at the moment. As a workaround for now you can set the lookback time "-l 700" to something like "-l 2000" as the observations were taken more than 700 days ago. Let me know if that works or not.

mperrin commented 2 months ago

I concur with @AarynnCarter about changing the look-back time in that command. (I don't know why the jwst_download script has a required argument for "how many days ago to look back and search for this data"; that doesn't seem like in general a helpful parameter to have to specify... But that is an upstream issue with jwst_download, not something we can directly do much about here, other than changing the number in the example code which I agree we should do.)

cleevesli commented 1 month ago

Thanks everyone, and sorry for being slow to respond (I hadn't actually seen the responses until recently!).

That fixed one problem, but a second issue related to jwst_download.py came up, where it asks confirmation before downloading: Do you want to continue and download these products [y/n]? which then crashes with:

File "/Users/lic3f/miniconda3/envs/spaceklip/lib/python3.11/site-packages/jwst_mast_query/jwst_download.py", line 177, in download_products
    do_it = input('Do you want to continue and download these products [y/n]?  ')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: EOF when reading a line

I wasn't able to find a general parser option to pass directly to jwst_download.py and just passing something like "ask_confirm_download: False" in the cfg file didn't work, but I obviously can just do this outside of the jupyter notebook at a terminal. Thought I'd raise the issue if you are working on making these notebooks more general purpose..

cleevesli commented 1 month ago

Just while commenting, another part of the notebook that had trouble was the create_database, but is fixed when you set the obsnum directory like here:

# Initialize spaceKLIP database.
database = spaceKLIP.database.create_database(input_dir = os.path.join(data_root, 'uncal/obsnum01/'),
                                              output_dir=data_root, 
                                              filt=filt,
                                              pid=1386)

AarynnCarter commented 1 month ago

Thanks for the update @cleevesli. We are working on new tutorial versions that are almost complete and will keep this in mind! Until then I think the best approach will be to download the data outside of the notebook.

@kglidic Could you check / confirm whether these issues are present in your latest tutorial notebooks?

kglidic commented 1 month ago

Hi @cleevesli these problems with the downloading of data should now be fixed in the most up-to-date NIRCam tutorial and I would recommend giving that version a try!

kammerje / spaceKLIP