Ds 5469 additional file formats

iamchrisearle commented 1 year ago

Description

Whitelist additional file extension in granule handler. Allows for netcdf and h5 files to be downloaded in addition to hdf and xml. File extensions are validated on get_url_from_ganule method. We may want to do away with extension validation all together, but this expands product use without opening the system too wide without knowing potential consequences. This will deprecate the ParamType as a tuple is now used to check file extensions.

Closes https://github.com/fraymio/modis-tools/issues/30 and https://github.com/fraymio/modis-tools/issues/8

Type of change

Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?

[x] Attempt to download any of these files from main:
```
VNP46A1 - V:1
VNP03DNB - V:1,2
VNP02DNB - V:1,2
VNP09A1 - V:001
```
following the repo example: collections = collection_client.query(short_name="VNP03DNB", version="2")

These will fail on main with No matching link error.

Attempt again in this branch with the same changes above, .nc and .h5 files will download.

Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[x] I have checked my code and corrected any misspellings

Next Steps

[ ] Assign a reviewer based on the code owner document.
[ ] Once your review is approved, merge and delete the feature branch

On behalf of the Modis Tools Dev Team, thank you for your hard work! ✨

MohammadJavadSoltani commented 1 year ago

Thank you so much for your kindness and precious time. It worked out perfectly.

iamchrisearle commented 1 year ago

Agreed. Because we return on the first match in the loop in this function, the tuple here would essentially set a ranking of file preferences that the user still couldn't change in download_from_granules

@lmcindewar would it be preferable to download all files with any of the passed file extensions, instead of moving forward from the first match?

lmcindewar commented 1 year ago

Agreed. Because we return on the first match in the loop in this function, the tuple here would essentially set a ranking of file preferences that the user still couldn't change in download_from_granules

@lmcindewar would it be preferable to download all files with any of the passed file extensions, instead of moving forward from the first match?

I think that might change the behavior of the function enough to cause issues, or at least download a lot of extra files. I do think the logic for choosing which link to download could use some improvement

fraymio / modis-tools