Adds in an unpacking loop to replace spaces in URLs. This happens in two places so this solution could be converted to a function. I'm looking for some input on cleaning up this solution as well, as this works but is not very clean. The dictionary unpacking lookup wasn't updating the source dictionary after looping through, so a copy is made and the indices that need to be changes are saved as a list of tuples to iterate through. The solution uses string replace which could be prone to error. I tried using something like quote() but found it was breaking other parts of the URL.
To recreate and isolate the broken URL, you can add a break point to the resources.py file where CollectionFeed is called and use step into until you reach the requests' models.py where it returns content. Content will be a json that contains the broken link as an href.
I have also created a draft pull request to start investigating how we could eventually update to pydantic V2. It contains jsons from this product and for another MODIS product that works as expected without this change for comparison.
Closes #38
Type of change
Please delete options that are not relevant.
[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
How Has This Been Tested?
[x] In this new branch, use the test.py file to download MODISA_L3b_CHL files, they should now download without issue.
[x] In main, use the same test.py file to download MODISA_L3b_CHL which will fail with the error seen in #38
[ ] Additional tests still need to be built for this
Checklist:
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[x] I have checked my code and corrected any misspellings
Description
Adds in an unpacking loop to replace spaces in URLs. This happens in two places so this solution could be converted to a function. I'm looking for some input on cleaning up this solution as well, as this works but is not very clean. The dictionary unpacking lookup wasn't updating the source dictionary after looping through, so a copy is made and the indices that need to be changes are saved as a list of tuples to iterate through. The solution uses string replace which could be prone to error. I tried using something like
quote()but found it was breaking other parts of the URL.To recreate and isolate the broken URL, you can add a break point to the resources.py file where CollectionFeed is called and use step into until you reach the requests' models.py where it returns content. Content will be a json that contains the broken link as an href.
I have also created a draft pull request to start investigating how we could eventually update to pydantic V2. It contains jsons from this product and for another MODIS product that works as expected without this change for comparison.
Closes #38
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
MODISA_L3b_CHLfiles, they should now download without issue.MODISA_L3b_CHLwhich will fail with the error seen in #38Checklist:
Next Steps
[x] Assign a reviewer based on the code owner document.
[ ] Once your review is approved, merge and delete the feature branch
On behalf of the Modis Tools Dev Team, thank you for your hard work! ✨