search

podaac / data-subscriber

Subscribe and bulk download collections of data at PO.DAAC
Apache License 2.0
81 stars 28 forks source link

Downloaded files are broken (11kb). Downloading from LAADS provider #163

Open rmant opened 5 months ago

rmant commented 5 months ago

Hello,

I have successfully utilized the PODAAC Data Subscriber tool to download Sea Surface Temperature (SST) data from the MUR collection using the default data provider without any issues.

However, I've encountered a problem when attempting to download VIIRS data from the LAADS provider. The download process appears to operate correctly; the tool identifies data matching my criteria and proceeds with the download. Unfortunately, the downloaded files are not usable. They are each 11kb in size, suggesting that the files are incomplete or corrupted. When I manually download files directly from the same URLs, I receive the expected, much larger .nc files. For example, manually downloading from this URL results in a complete file.

Command Used: podaac-data-downloader -c VJ102IMG -d test_data -b="-41.916838,-73.112378,-41.471221,-72.637712" -sd 2024-02-02T00:00:00Z -ed 2024-02-03T00:00:00Z -p LAADS

Output: [2024-03-11 17:54:29,481] {podaac_data_downloader.py:270} INFO - Found 8 total files to download [2024-03-11 17:54:32,238] {podaac_data_downloader.py:313} INFO - 2024-03-11 17:54:32.238262 SUCCESS: https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5201/VJ102IMG/2024/033/VJ102IMG.A2024033.2230.021.2024034045318.nc [2024-03-11 17:54:35,058] {podaac_data_downloader.py:313} INFO - 2024-03-11 17:54:35.057829 SUCCESS: https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5201/VJ102IMG/2024/033/VJ102IMG.A2024033.2054.021.2024034031115.nc [2024-03-11 17:54:37,992] {podaac_data_downloader.py:313} INFO - 2024-03-11 17:54:37.991908 SUCCESS: https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5201/VJ102IMG/2024/033/VJ102IMG.A2024033.1912.021.2024034002540.nc [2024-03-11 17:54:40,932] {podaac_data_downloader.py:313} INFO - 2024-03-11 17:54:40.931873 SUCCESS: https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5201/VJ102IMG/2024/033/VJ102IMG.A2024033.1730.021.2024034000157.nc [2024-03-11 17:54:43,899] {podaac_data_downloader.py:313} INFO - 2024-03-11 17:54:43.899482 SUCCESS: https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5201/VJ102IMG/2024/033/VJ102IMG.A2024033.1554.021.2024033235604.nc [2024-03-11 17:54:46,663] {podaac_data_downloader.py:313} INFO - 2024-03-11 17:54:46.663224 SUCCESS: https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5201/VJ102IMG/2024/033/VJ102IMG.A2024033.0354.021.2024033112805.nc [2024-03-11 17:54:49,539] {podaac_data_downloader.py:313} INFO - 2024-03-11 17:54:49.539312 SUCCESS: https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5201/VJ102IMG/2024/033/VJ102IMG.A2024033.0212.021.2024033091939.nc [2024-03-11 17:54:52,297] {podaac_data_downloader.py:313} INFO - 2024-03-11 17:54:52.297681 SUCCESS: https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5201/VJ102IMG/2024/033/VJ102IMG.A2024033.0030.021.2024033071132.nc [2024-03-11 17:54:52,298] {podaac_data_downloader.py:324} INFO - Downloaded Files: 8 [2024-03-11 17:54:52,298] {podaac_data_downloader.py:325} INFO - Failed Files: 0 [2024-03-11 17:54:52,298] {podaac_data_downloader.py:326} INFO - Skipped Files: 0 [2024-03-11 17:54:52,772] {podaac_data_downloader.py:334} INFO - END

rmant commented 5 months ago

An update on this; I checked the content of the file, and even if it's a .nc file, it contains html content saying an authentication error. However, it is able to retrieve the list of available files for that day, and it also is able to donwload data from other providers. My best guess is that it's an issue with how the bearer token is passed to the laads provider.