search

podaac / data-subscriber

Subscribe and bulk download collections of data at PO.DAAC
Apache License 2.0
83 stars 29 forks source link

Issue/164: Upgrade harmony-py to avoid 413 request entity too large issue #165

Closed skorper closed 6 months ago

skorper commented 6 months ago

See #164

The Harmony team released a new version which uses a POST request instead of a GET request when submitting Harmony jobs, which fixes our "413 request entity too large" error.

Testing:

poetry run python3 subscriber/podaac_data_subscriber.py -c SWOT_L2_LR_SSH_BASIC_2.0 -d ./data --start-date 2023-05-01T00:00:00Z --end-date 2024-12-21T00:00:00Z --subset -b="120,-30,160,20"
[2024-03-14 10:41:58,354] {podaac_data_subscriber.py:183} WARNING - No .update__SWOT_L2_LR_SSH_BASIC_2.0 in the data directory. (Is this the first run?)
[2024-03-14 10:42:07,065] {subsetting.py:102} INFO - Waiting for Harmony subsetting job to complete...
./data/SWOT_L2_LR_SSH_Basic_006_547_20231122T012915_20231122T022044_PIC0_01_subsetted.nc4
...
./data/SWOT_L2_LR_SSH_Basic_012_019_20240307T091643_20240307T100711_PIC0_01_subsetted.nc4
[2024-03-14 10:57:00,828] {podaac_data_subscriber.py:298} INFO - END

(removed lines from output for brevity)