search

podaac / data-subscriber

Subscribe and bulk download collections of data at PO.DAAC
Apache License 2.0
83 stars 29 forks source link

LARC download showed "success" when granule never downloaded (failure). #118

Closed mike-gangl closed 1 year ago

mike-gangl commented 1 year ago 
[2023-01-31 09:10:11,144] {podaac_data_subscriber.py:165} INFO - NOTE: Making new data directory at temp_larc(This is the first run.)
[2023-01-31 09:10:11,145] {podaac_data_subscriber.py:206} INFO - Temporal Range: 2012-04-30T22:00:00Z,2012-05-01T00:00:00Z
[2023-01-31 09:10:11,145] {podaac_data_subscriber.py:212} INFO - Provider: LARC_ASDC
[2023-01-31 09:10:11,145] {podaac_data_subscriber.py:213} INFO - Updated Since: 2012-04-30T22:00:00Z
[2023-01-31 09:10:11,145] {podaac_access.py:368} INFO - https://cmr.earthdata.nasa.gov/search/granules.umm_json?page_size=2000&sort_key=-start_date&provider=LARC_ASDC&updated_since=2012-04-30T22%3A00%3A00Z&ShortName=CAL_LID_L2_05kmAPro-Standard-V4-20&temporal=2012-04-30T22%3A00%3A00Z%2C2012-05-01T00%3A00%3A00Z&token=eyJ0eXAiOiJKV1QiLCJvcmlnaW4iOiJFYXJ0aGRhdGEgTG9naW4iLCJzaWciOiJlZGxqd3RwdWJrZXlfb3BzIiwiYWxnIjoiUlMyNTYifQ.eyJ0eXBlIjoiVXNlciIsInVpZCI6Im1nYW5nbCIsImV4cCI6MTY3NjEzNTUyOSwiaWF0IjoxNjcwOTUxNTI5LCJpc3MiOiJFYXJ0aGRhdGEgTG9naW4ifQ.stFjoIPymGul4zLyLlWIXqzB3jvN4WLrEjozw9eSM4n9YzeDe4RtrUfBgx5gg8b32_oyR3k3Lx7eYxcgtLijU-l_ouRfjCDZmXxk8a-z1jpvZmWu1YFBizPyrAAJQBYqnozkAashzuPPe0gwjFwK53ggaZs7CFQFvIgBXsQ1irLkYmWO6jeahy-7U5RV81kxUkUYwuC9EBFxmMlGv_rX0kYAyyrqE1ASiYt2BxRtkapg_-FYEaQH7xbjkBr6-4GGae_bCd3iQBswhPg9NPyLzr6wp-qiVegPTrPre__TuhqnpWX258zjchG6v1h8LlCVj8HyfLpzHRSBxeSiSriHmw
[2023-01-31 09:10:12,141] {podaac_data_subscriber.py:232} INFO - 4 new granules found for CAL_LID_L2_05kmAPro-Standard-V4-20 since 2012-04-30T22:00:00Z
[2023-01-31 09:10:12,141] {podaac_data_subscriber.py:271} INFO - Found 4 total files to download
[2023-01-31 09:10:12,142] {podaac_data_subscriber.py:273} INFO - Downloading files with extensions: ['.hdf']
[2023-01-31 09:10:12,624] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T23-24-12ZD.hdf. Retrying download.
[2023-01-31 09:10:13,027] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T23-24-12ZD.hdf. Retrying download.
[2023-01-31 09:10:14,460] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T23-24-12ZD.hdf. Retrying download.
[2023-01-31 09:10:16,465] {podaac_data_subscriber.py:302} INFO - 2023-01-31 09:10:16.465882 SUCCESS: https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T23-24-12ZD.hdf
[2023-01-31 09:10:16,880] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T22-37-42ZN.hdf. Retrying download.
[2023-01-31 09:10:17,288] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T22-37-42ZN.hdf. Retrying download.
[2023-01-31 09:10:18,744] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T22-37-42ZN.hdf. Retrying download.
[2023-01-31 09:10:20,748] {podaac_data_subscriber.py:302} INFO - 2023-01-31 09:10:20.748928 SUCCESS: https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T22-37-42ZN.hdf
[2023-01-31 09:10:21,158] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T21-45-17ZD.hdf. Retrying download.
[2023-01-31 09:10:21,564] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T21-45-17ZD.hdf. Retrying download.
[2023-01-31 09:10:22,963] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T21-45-17ZD.hdf. Retrying download.
[2023-01-31 09:10:24,967] {podaac_data_subscriber.py:302} INFO - 2023-01-31 09:10:24.967408 SUCCESS: https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T21-45-17ZD.hdf
[2023-01-31 09:10:25,370] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T20-58-46ZN.hdf. Retrying download.
[2023-01-31 09:10:25,782] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T20-58-46ZN.hdf. Retrying download.
[2023-01-31 09:10:27,222] {podaac_access.py:343} WARNING - Error downloading https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T20-58-46ZN.hdf. Retrying download.
[2023-01-31 09:10:29,230] {podaac_data_subscriber.py:302} INFO - 2023-01-31 09:10:29.229988 SUCCESS: https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Standard-V4-20/2012/04/CAL_LID_L2_05kmAPro-Standard-V4-20.2012-04-30T20-58-46ZN.hdf
[2023-01-31 09:10:29,230] {podaac_data_subscriber.py:317} INFO - Downloaded Files: 4
[2023-01-31 09:10:29,230] {podaac_data_subscriber.py:318} INFO - Failed Files:     0
[2023-01-31 09:10:29,230] {podaac_data_subscriber.py:319} INFO - Skipped Files:    0
mike-gangl commented 1 year ago

Simple bug to find and diagnose given an actual failing granule... the issue is: given this section of podaac_access:

if r >= retries:
       failed = True

range is exclusive, so range(3): 0,1,2 so retries will never be >= 3; need to subtract 1 (doh)