search

podaac / data-subscriber

Subscribe and bulk download collections of data at PO.DAAC
Apache License 2.0
81 stars 28 forks source link

Subscriber and Dowload warns not all granules will be downloaded, but all granules get downloaded #99

Closed frankinspace closed 2 years ago

frankinspace commented 2 years ago

Example log

A warning is printed to the log when downloading many granules that says only the first 2000 will be downloaded. But all granules get downloaded.

Limits were removed in https://github.com/podaac/data-subscriber/pull/65 so this warning should likely be removed.

 1 ============== Wed Aug  3 22:24:01 UTC 2022 ===============
      2 [2022-08-03 22:24:01,555] {podaac_data_subscriber.py:165} INFO - NOTE: Making new data directory at /cloud/ghrsst/open/data/GDS2/L3U/AVHRRMTC/STAR/v2.80(This is the first run.)
      3 [2022-08-03 22:24:01,558] {podaac_data_subscriber.py:206} INFO - Temporal Range: 2022-07-01T00:00:00Z,2022-08-03T22:24:01Z
      4 [2022-08-03 22:24:01,558] {podaac_data_subscriber.py:212} INFO - Provider: POCLOUD
      5 [2022-08-03 22:24:01,558] {podaac_data_subscriber.py:213} INFO - Updated Since: 2022-07-01T00:00:00Z
      6 [2022-08-03 22:24:01,558] {podaac_access.py:301} INFO - https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=2000&sort_key=-start_date&provider=POC        LOUD&updated_since=2022-07-01T00%3A00%3A00Z&ShortName=AVHRRF_MC-STAR-L3U-v2.80&temporal=2022-07-01T00%3A00%3A00Z%2C2022-08-03T22%3A24%3A01Z&token=D5A7A608-AFCD-719D-7998-B46207622CB1
      7 [2022-08-03 22:24:06,112] {podaac_data_subscriber.py:228} INFO - 4850 new granules found for AVHRRF_MC-STAR-L3U-v2.80 since 2022-07-01T00:00:00Z
>>      8 [2022-08-03 22:24:06,277] {podaac_data_subscriber.py:254} WARNING - Only the most recent 2000 granules will be downloaded; try adjusting your search criteria (suggestion: reduce time period or spatial region of search) to ensure you retrieve all granules.
      9 [2022-08-03 22:24:06,283] {podaac_data_subscriber.py:270} INFO - Found 4850 total files to download
     10 [2022-08-03 22:24:06,284] {podaac_data_subscriber.py:272} INFO - Downloading files with extensions: ['.nc']
     11 [2022-08-03 22:24:10,666] {podaac_data_subscriber.py:299} INFO - 2022-08-03 22:24:10.666259 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/AVHRRF_MC-STAR-L3U-v2.80/2022/215/20220803195000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MC-ACSPO_V2.80-v02.0-fv01.0.nc
...
   4928 [2022-08-04 00:13:32,702] {podaac_data_subscriber.py:299} INFO - 2022-08-04 00:13:32.702847 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/AVHRRF_MC-STAR-L3U-v2.80/2022/182/20220701000000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MC-ACSPO_V2.80-v02.0-fv01.0.nc
>>   4929 [2022-08-04 00:13:32,703] {podaac_data_subscriber.py:314} INFO - Downloaded Files: 4848
   4930 [2022-08-04 00:13:32,703] {podaac_data_subscriber.py:315} INFO - Failed Files:     2
   4931 [2022-08-04 00:13:32,703] {podaac_data_subscriber.py:316} INFO - Skipped Files:    0
   4932 [2022-08-04 00:13:33,051] {podaac_access.py:118} INFO - CMR token successfully deleted
   4933 [2022-08-04 00:13:33,052] {podaac_data_subscriber.py:318} INFO - END
frankinspace commented 2 years ago

https://github.com/podaac/data-subscriber/blob/1a5f534435040575184a73633d0f5d1c28e478f7/subscriber/podaac_data_downloader.py#L225 https://github.com/podaac/data-subscriber/blob/1a5f534435040575184a73633d0f5d1c28e478f7/subscriber/podaac_data_subscriber.py#L253