Landsat8 download problem from USGS-EROS for L1C product not available on AWS S3 or Google Storage

[gybely]

I am not having success trying to load a set of specific Landsat-8 scenes from USGS EROS site -- for scenes not available on AWS or Google-- trying this from an OSX machine.

I get the scene names using '> landsat search' just fine... With '> landsat download', I get "product not available on AWS S3 or Google Storage"... one of these scenes, as an example, is LC81290582016007LGN02.

When I supply my USGS EROS credentials --username --password (I do have m2m / special inventory access enabled on this account), I get the error message below. Any help greatly appreciated...

----

Time spent : 0.75 seconds Traceback (most recent call last): File "/usr/local/bin/landsat", line 11, in sys.exit(main()) File "/usr/local/lib/python2.7/site-packages/landsat/landsat.py", line 491, in main exit(*main(args)) File "/usr/local/lib/python2.7/site-packages/landsat/landsat.py", line 408, in main files = d.download(args.scenes, bands) File "/usr/local/lib/python2.7/site-packages/landsat/downloader.py", line 80, in download files.append(self.usgs_eros(scene, self.download_dir)) File "/usr/local/lib/python2.7/site-packages/landsat/downloader.py", line 95, in usgs_eros error_tree = ElementTree.fromstring(str(e.message)) File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1311, in XML parser.feed(text) File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed self._raiseerror(v) File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror raise err xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 171, column 22

[j08lue]

I had this issue, too, and solved it by updating usgs to v0.2.0.

pip install -U usgs

But there might be more issues because USGS is not using SOAP anymore?

[qianliuGit]

I have the problem, is there a solution??

[j08lue]

yes

[gybely]

After pip -U install usgs I still have the problem with some scenes from USGS... My guess is this is still related to the scene reprocessing for Collection-1 L1.

Error below: usgs.USGSError: DATASET_INVALID: Dataset is not configured for Inventory Service

[j08lue]

@gybely @qianliuGit

You also need this change to landsat-util: #235

You can install it from my fork, if you like:

pip install https://github.com/DHI-GRAS/landsat-util/archive/usgs_c1.zip

[qianliuGit]

@j08lue I uninstalled my landsat-util, then i have ran the "pip install -U usgs" , and then ran the "pip install https://github.com/DHI-GRAS/landsat-util/archive/usgs_c1.zip". When i ran "landsat search --start "january 1 2017" --end "january 30 2017" -p 122,036", i got this: ===> 2 items were found { "limit": 10, "results": [ { "cloud": 4.01, "date": "2017-01-24", "path": "122", "row": "036", "sat_type": "L8", "sceneID": "LC81220362017024LGN01", "thumbnail": "http://earthexplorer.usgs.gov/browse/landsat_8/2017/122/036/LC08_L1TP_122036_20170124_20170311_01_T1.jpg" }, { "cloud": 75.02, "date": "2017-01-08", "path": "122", "row": "036", "sat_type": "L8", "sceneID": "LC81220362017008LGN01", "thumbnail": "http://earthexplorer.usgs.gov/browse/landsat_8/2017/122/036/LC08_L1GT_122036_20170108_20170311_01_T2.jpg" } ], "status": "SUCCESS", "total": 2, "total_returned": 2 } ['Search completed!'] ,

then "landsat download LC81220362017024LGN01" i have got this "LC81220362017024LGN01 is not available on AWS S3 or Google Storage",

Any help is greatly appreciated, thanks a lot!

[j08lue]

You did not specify your USGS account

                --username          USGS Eros account Username (only works if the account has special
                                    inventory access). Username and password as a fallback if the image
                                    is not found on AWS S3 or Google Storage
                --password          USGS Eros account Password

which means that USGS is not available as an option when Google and Amazon fail.

I take it you did not look that much into this ;), so you probably do not have applied for machine-to-machine access for your USGS EROS account. For that you need to write to their support and wait a couple of weeks.

[qianliuGit]

@j08lue Thanks, i will have a try !!

[jmellian]

I think the problem is that as of 1 may 2017, there has been a change in dataformat. Because of that the AWS en Google changed the location of the landsatfiles. I have fix that in pull-request #241, but there is a problem with the unittest for Geocoder (no changes there).

See: Download Collection 1 datastructure with AWS en Google-storage #241 Added download Collection 1 landsat 8 data from AWS after 1 may 2017:

Try the change from: https://github.com/jmellian/landsat-util :only geocoder is (probably) not working, because of a api-key problem

[minorsecond]

@jmellian I've tried both the version here and yours and still get the "not available" error. I've applied for machine-to-machine access and am hoping it works.

[j08lue]

I've applied for machine-to-machine access and am hoping it works.

USGS machine-to-machine access, I can confirm, works.

[matthewhanson]

I've updated the usgs library in requirements to 0.2.1 on the develop branch. If any of the issues raised in here persist please open as a new issue.