get data as a cli script

OliviaLynn commented 1 year ago

Change Description

Translated from draft PR #43 (rail).

Now implemented as cli script.

Required for rail_attic's https://github.com/LSSTDESC/rail_attic/pull/372 (fixing goldenspike pipeline); addresses rail_attic #356 (moving data) and rail #26 (moving demo files).

Solution Description

A script to check for existence of data files and curl what is missing.

Chose not to add testing, as this could result in overuse of NERSC data storage (and testing presence of data seems to require first wiping data, which...)

Future steps could involve additional options for the command that allow the user to specify which data; perhaps utilizing the prompting feature in click to interactively walk the user through each file and pick skip or download.

Code Quality

[x] I have read the Contribution Guide
[x] My code follows the code style of this project
[x] My code builds (or compiles) cleanly without any errors or warnings
[x] My code contains relevant comments and necessary documentation

Project-Specific Pull Request Checklists

New Feature Checklist

[x] I have added or updated the docstrings associated with my feature using the NumPy docstring format
[x] I have updated the tutorial to highlight my new feature (if appropriate)
[No - see note above] I have added unit/End-to-End (E2E) test cases to cover my new feature
[ ] My change includes a breaking change
- [ ] My change includes backwards compatibility and deprecation warnings (if possible)

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (3429445) 97.53% compared to head (d39962d) 97.53%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #20 +/- ## ======================================= Coverage 97.53% 97.53% ======================================= Files 28 28 Lines 1584 1585 +1 ======================================= + Hits 1545 1546 +1 Misses 39 39 ``` | [Impacted Files](https://app.codecov.io/gh/LSSTDESC/rail_base/pull/20?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LSSTDESC) | Coverage Δ | | |---|---|---| | [src/rail/cli/scripts.py](https://app.codecov.io/gh/LSSTDESC/rail_base/pull/20?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LSSTDESC#diff-c3JjL3JhaWwvY2xpL3NjcmlwdHMucHk=) | `70.45% <ø> (ø)` | | | [src/rail/cli/options.py](https://app.codecov.io/gh/LSSTDESC/rail_base/pull/20?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LSSTDESC#diff-c3JjL3JhaWwvY2xpL29wdGlvbnMucHk=) | `90.24% <100.00%> (+0.24%)` | :arrow_up: |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

OliviaLynn commented 1 year ago

Maybe change the get_data function to use kwargs to specify which data, and then set the default in the script to get the nersc data… that will be easier to extend…

Yes, absolutely!

I touch on that a little bit at the end of my solution description, but I think there are enough design decisions to be made around data specification that I just opened issue #22 to take the next step on that.

LSSTDESC / rail_base