search

podaac / data-subscriber

Subscribe and bulk download collections of data at PO.DAAC
Apache License 2.0
88 stars 31 forks source link

Improve/fix functionality of download by extension in podaac-data-downloader #104

Closed celiaou-podaac closed 1 year ago

celiaou-podaac commented 2 years ago

The downloader does not seem to identify granules correctly during a request with '-e', for specific extensions. In contrast the subscriber identifies and downloads granules using all the same parameters. Examples below:

Requesting .nc files using downloader podaac-data-downloader -c JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F -d S6_L2_HR_STD_NRT -e .nc -sd 2021-06-01T00:46:02Z -ed 2021-06-01T03:00:00Z [2022-09-02 16:02:12,020] {podaac_data_downloader.py:242} INFO - Found 0 total files to download [2022-09-02 16:02:12,021] {podaac_data_downloader.py:284} INFO - Downloaded Files: 0 [2022-09-02 16:02:12,025] {podaac_data_downloader.py:285} INFO - Failed Files: 0 [2022-09-02 16:02:12,029] {podaac_data_downloader.py:286} INFO - Skipped Files: 0 [2022-09-02 16:02:12,297] {podaac_access.py:118} INFO - CMR token successfully deleted [2022-09-02 16:02:12,297] {podaac_data_downloader.py:288} INFO - END

Requesting .nc files using subscriber podaac-data-subscriber -c JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F -d S6_L2_HR_STD_NRT -e .nc -sd 2021-06-01T00:46:02Z -ed 2021-06-01T03:00:00Z [2022-09-02 16:01:06,815] {podaac_data_subscriber.py:179} WARNING - No .updateJASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F in the data directory. (Is this the first run?) [2022-09-02 16:01:07,953] {podaac_data_subscriber.py:270} INFO - Found 10 total files to download [2022-09-02 16:01:12,063] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:12.063932 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STDNR_020_181_20210601T025238_20210601T025438_F02.nc [2022-09-02 16:01:13,701] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:13.701817 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STDNR_020_181_20210601T024238_20210601T025238_F02.nc [2022-09-02 16:01:15,301] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:15.301483 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STDNR_020_181_20210601T023308_20210601T024238_F02.nc [2022-09-02 16:01:16,771] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:16.769997 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STDNR_020_181_20210601T022320_20210601T022637_F02.nc [2022-09-02 16:01:18,283] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:18.283086 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STDNR_020_180_20210601T020158_20210601T020254_F02.nc [2022-09-02 16:01:20,296] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:20.296783 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STDNR_020_180_20210601T014632_20210601T015135_F02.nc [2022-09-02 16:01:21,884] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:21.884365 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STDNR_020_180_20210601T013541_20210601T013645_F02.nc [2022-09-02 16:01:23,508] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:23.508012 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STDNR_020_180_20210601T012546_20210601T013428_F02.nc [2022-09-02 16:01:25,340] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:25.340850 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STDNR_020_179_20210601T005400_20210601T010128_F02.nc [2022-09-02 16:01:27,031] {podaac_data_subscriber.py:299} INFO - 2022-09-02 16:01:27.031012 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2HR_STD__NR_020_179_20210601T004400_20210601T005400_F02.nc [2022-09-02 16:01:27,032] {podaac_data_subscriber.py:314} INFO - Downloaded Files: 10 [2022-09-02 16:01:27,035] {podaac_data_subscriber.py:315} INFO - Failed Files: 0 [2022-09-02 16:01:27,036] {podaac_data_subscriber.py:316} INFO - Skipped Files: 0 [2022-09-02 16:01:27,358] {podaac_access.py:118} INFO - CMR token successfully deleted [2022-09-02 16:01:27,364] {podaac_data_subscriber.py:318} INFO - END

mike-gangl commented 2 years ago

@celiaou-podaac do you know what version you're running? i've attempted this with the latest version of the subscriber/downloader and do not see your exisitng issue:

Run with non-explicit, default extensions

podaac-data-downloader -c JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F -d S6_L2_HR_STD_NRT -sd 2021-06-01T00:46:02Z -ed 2021-06-01T03:00:00Z
[2022-09-12 08:37:29,277] {podaac_data_downloader.py:155} INFO - NOTE: Making new data directory at S6_L2_HR_STD_NRT(This is the first run.)
[2022-09-12 08:37:30,195] {podaac_data_downloader.py:243} INFO - Found 10 total files to download
[2022-09-12 08:37:37,905] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:37.905698 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_181_20210601T025238_20210601T025438_F02.nc
[2022-09-12 08:37:40,821] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:40.821302 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_181_20210601T024238_20210601T025238_F02.nc
[2022-09-12 08:37:42,585] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:42.585931 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_181_20210601T023308_20210601T024238_F02.nc
[2022-09-12 08:37:44,385] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:44.385417 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_181_20210601T022320_20210601T022637_F02.nc
[2022-09-12 08:37:46,205] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:46.204995 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_180_20210601T020158_20210601T020254_F02.nc
[2022-09-12 08:37:48,001] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:48.001912 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_180_20210601T014632_20210601T015135_F02.nc
[2022-09-12 08:37:49,394] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:49.394428 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_180_20210601T013541_20210601T013645_F02.nc
[2022-09-12 08:37:51,162] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:51.162424 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_180_20210601T012546_20210601T013428_F02.nc
[2022-09-12 08:37:52,796] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:52.796697 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_179_20210601T005400_20210601T010128_F02.nc
[2022-09-12 08:37:54,426] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:37:54.426536 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_179_20210601T004400_20210601T005400_F02.nc
[2022-09-12 08:37:54,426] {podaac_data_downloader.py:287} INFO - Downloaded Files: 10
[2022-09-12 08:37:54,426] {podaac_data_downloader.py:288} INFO - Failed Files:     0
[2022-09-12 08:37:54,426] {podaac_data_downloader.py:289} INFO - Skipped Files:    0
[2022-09-12 08:37:55,182] {podaac_access.py:122} INFO - CMR token successfully deleted
[2022-09-12 08:37:55,182] {podaac_data_downloader.py:299} INFO - END

Run using explicit extension

podaac-data-downloader -c JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F -d S6_L2_HR_STD_NRT -sd 2021-06-01T00:46:02Z -ed 2021-06-01T03:00:00Z -e .nc
[2022-09-12 08:38:51,377] {podaac_data_downloader.py:155} INFO - NOTE: Making new data directory at S6_L2_HR_STD_NRT(This is the first run.)
[2022-09-12 08:38:52,422] {podaac_data_downloader.py:243} INFO - Found 10 total files to download
[2022-09-12 08:38:57,746] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:38:57.746031 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_181_20210601T025238_20210601T025438_F02.nc
[2022-09-12 08:38:59,234] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:38:59.234188 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_181_20210601T024238_20210601T025238_F02.nc
[2022-09-12 08:39:00,757] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:39:00.757890 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_181_20210601T023308_20210601T024238_F02.nc
[2022-09-12 08:39:02,443] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:39:02.443154 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_181_20210601T022320_20210601T022637_F02.nc
[2022-09-12 08:39:04,336] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:39:04.336127 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_180_20210601T020158_20210601T020254_F02.nc
[2022-09-12 08:39:05,954] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:39:05.954476 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_180_20210601T014632_20210601T015135_F02.nc
[2022-09-12 08:39:07,749] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:39:07.749380 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_180_20210601T013541_20210601T013645_F02.nc
[2022-09-12 08:39:09,375] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:39:09.375768 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_180_20210601T012546_20210601T013428_F02.nc
[2022-09-12 08:39:11,388] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:39:11.388350 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_179_20210601T005400_20210601T010128_F02.nc
[2022-09-12 08:39:13,433] {podaac_data_downloader.py:276} INFO - 2022-09-12 08:39:13.433126 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F/S6A_P4_2__HR_STD__NR_020_179_20210601T004400_20210601T005400_F02.nc
[2022-09-12 08:39:13,433] {podaac_data_downloader.py:287} INFO - Downloaded Files: 10
[2022-09-12 08:39:13,433] {podaac_data_downloader.py:288} INFO - Failed Files:     0
[2022-09-12 08:39:13,433] {podaac_data_downloader.py:289} INFO - Skipped Files:    0
[2022-09-12 08:39:14,034] {podaac_access.py:122} INFO - CMR token successfully deleted
[2022-09-12 08:39:14,034] {podaac_data_downloader.py:299} INFO - END
celiaou-podaac commented 2 years ago

Thanks Mike, I was running 1.10.2. Let me update and try again.

mike-gangl commented 1 year ago

@celiaou-podaac were you able to reproduce the issue with the newest version of subscriber?

celiaou-podaac commented 1 year ago

@mike-gangl I do not see the issue anymore. Thanks!