search

NREL / nsrdb

NSRDB data processing pipeline. Includes satellite data assimilation, cloud property prediction and gap-filling, radiative transport modeling, and data collection.
https://nrel.github.io/nsrdb/
BSD 3-Clause "New" or "Revised" License
7 stars 0 forks source link

NSRDB pipeline single job debug #46

Closed grantbuster closed 2 years ago

grantbuster commented 2 years ago

Why this feature is necessary: It would be super nice to be able to submit an NSRDB pipeline job but only actually submit 1 out of the 365 jobs to slurm so you can debug on a single job instead of a full production run. The feature would still have to enter the other jobs into the status file but only a single job would actually get submitted on the hpc.

Current alternative is probably to submit manually from python.

bnb32 commented 2 years ago

Could I pass a debug_day argument through ctx from pipeline to config in the cli to accomplish this?

grantbuster commented 2 years ago

probably not.... i dont think the pipeline is meant to pass kwargs to the lower level jobs. It's just meant to kick off the nsrdb cli and point to the nsrdb config file.