level-1-landsat-download: Some links seem to be broken

[MandLisa]

Hi all, I just tried to download Landsat data via force-level1-landsat-download (since no Landsat data from 1997 are available via force-level1-csd). All attempts ended in the following error message: Some links seem to be broken, please check: [List with ALL Landsat links). However, the manual download of the data via the links works. Therefore, I do not think that there is a problem with the URLs. I have also heard from others that it does not work anymore. Strangely enough, this error didn't exist a fortnight ago.

Setup Always add information on your system, e.g.

• FORCE version 3.7.10
• OS and version: NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)"
• bare metal Linux server or virtual machine? VM
• did you deviate from the installation instruction? No
• CPUs: 2x Intel Xeon Gold 5220R CPU (24 cores each), RAM: 768 GB RAM

Thanks, Lisa

[ernstste]

Hi Lisa, I can't reproduce the issue. Could you share the full command you are using and post some of the links?

[MandLisa]

Hi Stefan, sure. This is the full command: docker run \ -v $basepath/DataCube:/path \ --user "$(id -u):*******" \ --env FORCE_CREDENTIALS=/app/credentials \ -v $HOME:/app/credentials davidfrantz/force \ force-level1-landsat download /path/projects/foreco/alps/level1/urls_landsat_TM_20230524T073439.txt /path/projects/foreco/alps/level1/LS_1997

And here are some of the links: https://landsatlook.usgs.gov/gen-bundle?landsat_product_id=LT05_L1TP_193029_19970915_20200909_02_T1&requestSignature=eyJjb250YWN0SWQiOjI2NjI4MTIyLCJkb3dubG9hZElkIjozOTU1OTg5MjgsImRhdGVHZW5lcmF0ZWQiOiIyMDIzLTA1LTI0VDAyOjM0OjI4LTA1OjAwIiwiaWQiOiJMVDA1X0wxVFBfMTkzMDI5XzE5OTcwOTE1XzIwMjAwOTA5XzAyX1QxIiwic2lnbmF0dXJlIjoiJDUkJE9ieHlJenpWbHlCcWlrMnEuY0NHWTVBYW80aXc1QjFNNXJwbldyM2toTDYifQ==

https://landsatlook.usgs.gov/gen-bundle?landsat_product_id=LT05_L1TP_193029_19971001_20200909_02_T1&requestSignature=eyJjb250YWN0SWQiOjI2NjI4MTIyLCJkb3dubG9hZElkIjozOTU1OTg5MjksImRhdGVHZW5lcmF0ZWQiOiIyMDIzLTA1LTI0VDAyOjM0OjI4LTA1OjAwIiwiaWQiOiJMVDA1X0wxVFBfMTkzMDI5XzE5OTcxMDAxXzIwMjAwOTA5XzAyX1QxIiwic2lnbmF0dXJlIjoiJDUkJGppemt3MzBEYnJaM2pxcnlDYXBlcFp2bnZ5MFhkR3RsN3ouUzhwZ2RuaDAifQ==

https://landsatlook.usgs.gov/gen-bundle?landsat_product_id=LT05_L1TP_193029_19971118_20200909_02_T1&requestSignature=eyJjb250YWN0SWQiOjI2NjI4MTIyLCJkb3dubG9hZElkIjozOTU1OTg5MzAsImRhdGVHZW5lcmF0ZWQiOiIyMDIzLTA1LTI0VDAyOjM0OjI4LTA1OjAwIiwiaWQiOiJMVDA1X0wxVFBfMTkzMDI5XzE5OTcxMTE4XzIwMjAwOTA5XzAyX1QxIiwic2lnbmF0dXJlIjoiJDUkJG9obFRBRGl2d0tJOUFnZmNnMThyQ0wuVHhlcmhCaXVJYzcyTmFOalhZeEMifQ==

https://landsatlook.usgs.gov/gen-bundle?landsat_product_id=LT05_L1TP_193030_19970203_20200910_02_T1&requestSignature=eyJjb250YWN0SWQiOjI2NjI4MTIyLCJkb3dubG9hZElkIjozOTU1OTg5MzEsImRhdGVHZW5lcmF0ZWQiOiIyMDIzLTA1LTI0VDAyOjM0OjI4LTA1OjAwIiwiaWQiOiJMVDA1X0wxVFBfMTkzMDMwXzE5OTcwMjAzXzIwMjAwOTEwXzAyX1QxIiwic2lnbmF0dXJlIjoiJDUkJGVJQkU2czVuSWNcL1ZpN0hubURsM053emJEcGw3c0tPWmxucDNrVk1YWUcuIn0=

I checked a few out of the 211 links and the manual download worked perfectly fine.

[ernstste]

Thanks Lisa,

the links look fine and they don't cause any issues when I use them with force-level1-landsat. I'm not really sure what to tell you. Feel free to post your urls_landsat_TM_20230524T073439.txt and I can try to see if anything pops up.

Thanks, Stefan

[MandLisa]

Hi Stefan, this is quite strange. I've just run level1-landsat-search again to create a new URL list, but I got the same error. This is the list I used:

urls_landsat_TM_20230524T073439.txt

Thanks for your help!

[ernstste]

Very strange indeed - this file works just fine here. Are you able to work around the issue by using force-level1-landsat search --download to directly download the search results instead of creating the download link list first and then downloading by running force-level1-landsat download?

[MandLisa]

We now tried to run the download command on three different servers. Strangely enough, on one server force-level1-landsat download is working now. So I guess the issue is related to our servers (would explain why it is working for you). We will figure out what's going on here.

Thanks Stefan!

[davidfrantz]

Could this be related to some USGS-quotas?

[ernstste]

Could this be related to some USGS-quotas?

Every link in the text file is checked against a regex, the error that Lisa describes is thrown if that check fails. So no - USGS has nothing to do with it, since this is just a local check to make sure that all files can be downloaded and we're not downloading arbitrary links.