Closed bucricket closed 6 years ago
matthewhanson
commented
7 years ago Hello @bucricket , thanks for pointing this out. Is this for Landsat data that is not available on AWS? Downloading data for any Landsat data available on AWS should still work using the scene ID. For data available only through USGS we will look into it, thanks again!
bucricket
commented
7 years ago Hi Matt,
This is for the non-AWS data. I actually use the search feature from Landsat-util to find the scenes and then use the espa-api to order and download the data. I have a work around right now using the https://landsat.usgs.gov/landsat/metadata_service/bulk_metadata_files/LANDSAT_8_C1.csv https://landsat.usgs.gov/landsat/metadata_service/bulk_metadata_files/LANDSAT_8_C1.csv file and importing it into pandas but its pretty slow. Don’t know if you are willing to implement collection 1 into Landsat-util since I guess it is a bit out of the scope of it as you are only dealing with AWS and google data. I use ESPA because I need higher level products.
Mitch
On Apr 10, 2017, at 11:52 PM, Matthew Hanson notifications@github.com wrote:
Hello @bucricket https://github.com/bucricket , thanks for pointing this out. Is this for Landsat data that is not available on AWS? Downloading data for any Landsat data available on AWS should still work using the scene ID. For data available only through USGS we will look into it, thanks for bringing it up!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/developmentseed/landsat-util/issues/229#issuecomment-293142421, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdNEHdlLqHleIVw3OjdqzfrAfw2Gq3Cks5ruvjsgaJpZM4M5epB.
noahpvaughn
commented
7 years ago Just wanted to post this here, AWS is also changing their data structure for the new Collection-1. As of April 30th no new imagery will be posted in the old AWS scene ID structure, it is now based on product ID. It appears there scene list is also only current up through 4/11/2017 at this point which is ~3weeks behind so if you need quick turn-around times on imagery AWS might not be the way to get the images at this point. I have an email out to their staff at Open Data now.
This is an email we received last week:
_Hello and thank you for indicating interest in the Landsat on AWS dataset (https://aws.amazon.com/public-datasets/landsat/). We wanted to make you aware of some upcoming changes to the data structure based on USGS’ move to Collections (https://landsat.usgs.gov/april-3-2017-changes-landsat-pre-collection-processing).
We'll proceed with a new Collection 1 prefix: s3://landsat-pds/c1/L8/
Collection 1 data is being brought in as of now. Pre-collection will continue to flow until USGS stops updating the archive, which we believe will be at the end of this month per https://landsat.usgs.gov/april-3-2017-changes-landsat-pre-collection-processing. There is an example Collection 1 image at https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/149/039/LC08_L1TP_149039_20170411_20170415_01_T1/index.html._
bucricket
commented
7 years ago Good to know. Will you be implementing the “ProductID” that collection 1 uses instead of the “sceneID” that pre-collection uses?
Mitch
On May 1, 2017, at 2:10 PM, noahpvaughn notifications@github.com wrote:
Just wanted to post this here, AWS is also changing their data structure for the new Collection-1. As of April 30th no new imagery will be posted in the old AWS scene ID structure, it is now based on product ID. This is an email we received last week:
_Hello and thank you for indicating interest in the Landsat on AWS dataset (https://aws.amazon.com/public-datasets/landsat/ https://aws.amazon.com/public-datasets/landsat/). We wanted to make you aware of some upcoming changes to the data structure based on USGS’ move to Collections (https://landsat.usgs.gov/april-3-2017-changes-landsat-pre-collection-processing https://landsat.usgs.gov/april-3-2017-changes-landsat-pre-collection-processing).
We'll proceed with a new Collection 1 prefix: s3://landsat-pds/c1/L8/// and a fresh scene list located at s3://landsat-pds/c1/L8/scene_list.gz. It has the same form as the previous scene list, with the addition of the product id.
Collection 1 data is being brought in as of now. Pre-collection will continue to flow until USGS stops updating the archive, which we believe will be at the end of this month per https://landsat.usgs.gov/april-3-2017-changes-landsat-pre-collection-processing https://landsat.usgs.gov/april-3-2017-changes-landsat-pre-collection-processing. There is an example Collection 1 image at https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/149/039/LC08_L1TP_149039_20170411_20170415_01_T1/index.html https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/149/039/LC08_L1TP_149039_20170411_20170415_01_T1/index.html._
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/developmentseed/landsat-util/issues/229#issuecomment-298390904, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdNEMzqhFpDSWzPkinBhGM6jqVV-_y7ks5r1iAagaJpZM4M5epB.
bucricket
commented
7 years ago Any updates on this? I'm very interested in getting collection 1 data.
drewbo
commented
7 years ago @bucricket the api used for search now has collection-1 data being added, we're working on updating all of the download links so that it points to the right data across USGS/AWS/Google. That should be coming soon and we can update this library to advantage of that soon
lorenzobernardi
commented
7 years ago @drewbo i'm using your landsat-utils packages for downloading the landsat 8 images over south of Italy. Starting from the half of may the download does not work and i suppose it depends on the new url path for the download. I use google storage modality. I tryed to find the new base link to automatically download the c-1 l8 data. Do you know if the base url to download the collection-1 landsat 8 data has been updated? Thank you, Lorenzo
j08lue
commented
7 years ago The solution for download from USGS is very simple. See PR #235
nick-phillips
commented
7 years ago @drewbo was wondering if there is any news regarding updated download links. I'm trying to download collection 1 data (files that exist on AWS/Google) and I'm getting RemoteFileDoesntExist exceptions.
Any guidance on the best way to download collection 1 data from AWS/Google using the sceneIDs or productIDs generated by landsat-util search? Thanks!
matthewhanson
commented
7 years ago @nick-phillips
Hey Nick, the Landsat on AWS is supposed to be updated to use collection 1, but right now it's still older data, and the links are bad because USGS incremented the revision number but they have not been updated on AWS. I'm not sure of Collection 1 status on Google, it's not working there either?
So, if you want Collection 1 I think you are stuck with USGS for now. If you are fine with using previous revisions of the file, but just want the download to work then there's another option. I made a new lightweight repository called sat-search, works for both sentinel-2 and landsat-8. As a work-around for landsat-8, while it won't update the (incorrect) links, if you use sat-search to download the files and it can't find it, it will try previous revisions until it can.
It's just a beta, and no one has really used it yet, but you are welcome to try if you find it useful. While I put a beta version on PyPi it's already outdated and it's best to install it from source.
matthewhanson
commented
6 years ago This has been fixed in the old version of sat-api: https://api.developmentseed.org/satellites/ and the AWS links work now (although still are the older revisions on AWS).
Note there is a new version of sat-api now that conforms to the STAC spec: https://medium.com/devseed/sat-api-an-api-for-spatiotemporal-asset-catalogs-88c3c78fdb0e
Hi,
I'm wondering if you are planning on implementing the new collection 1 information? The ESPA ordering system no longer stores the pre-collection data and so the sceneID information doesn't work anymore. We need the Landsat-product ID now.
Mitch