replace EarthSearch with more stable STAC API

[rbavery]

possible planetary computer?

I've run into issues where v0 of the api (what we use in the lesson) consistently returns internal server errors when initiating a client, or errors that only come up when you initiate a search.

The v1 endpoint doesn't have all the Sentinel scenes, there's only 8500, very few scenes over the USA, nothing over Amsterdam to match the vector data.

[fnattino]

I see, I haven't experienced issues myself with the Earth Search STAC endpoint, but I agree in that using the planetary computer would provide a more stable solution. And I wasn't aware of the "v1" endpoint - but as I cannot find any documentation/mention of it, maybe it is only experimental?

The drawback in using the planetary computer might be that in order to access data one needs to sign URLs with an access token, which adds some complexity: https://planetarycomputer.microsoft.com/dataset/sentinel-2-l2a#Example-Notebook

Also, anonymous access should be possible for the sentinel-2 L2A dataset, so learners should not need an account in order to get the images. But do you know how much limited is the bandwidth to download data without an access key?

[rbavery]

A quick update on this, it sounds like EarthSearch was experiencing issues with web crawlers specifically the day of the demo. it didn't end up being an issue for me during the demo but might have been for others.

Still, I think we at the least may want to switch to the v1 api endpoint and possibly an api that has auth to remove risk of web crawlers impacting a workshop (I'm not sure how common this is).

And I wasn't aware of the "v1" endpoint - but as I cannot find any documentation/mention of it, maybe it is only experimental?

That might be the case @fnattino , I tested it out and it seems like it is missing many scenes compared to the v0 endpoint. They may be populating it still or testing it. @matthewhanson is the v1 endpoint meant to contain the whole Sentinel-2 collection the v0 endpoint? Or is it experimental?

Also, anonymous access should be possible for the sentinel-2 L2A dataset, so learners should not need an account in order to get the images. But do you know how much limited is the bandwidth to download data without an access key?

I don't know but I'll check this out

[matthewhanson]

@rbavery The v1 endpoint is not yet final, there are load tests to do and a final ingest. It will contain the whole Sentinel-2 archive, as well as a Landsat L2 collection similar to the Planetary Computer Landsat collection (Landsat 4-9, both surface reflectance and surface temp combined).

There may be additional changes as well with rate limiting to prevent outages from users trying to rapidly crawl the whole thing. I don't have a specific timeline on this yet, but expect a complete v1 by end of the year. The v0 endpoint will be deprecated and eventually decommissioned.

[rbavery]

Ok awesome, thanks for these details @matthewhanson. Excited to test the v1 endpoint once it is completed

[rbavery]

It sounds like this is now released @fnattino @rogerkuou https://twitter.com/Element84/status/1666071623412097026?s=20

We should probably switch to this to prepare for v0 decomission.

[rbavery]

The v0 endpoint is currently down, and this change is in progress.

I plan on merging #141 #142 #138 so that @chuckwondo and I can run an internal demo of the lesson and note improvements to make before we teach this at IGARSS cc @fnattino @rogerkuou

mainly the updates are

• switching the api endpoint url
• the new api endpoint uses different band names for the same Sentinel-2 collection. switches asset band names
• there’s no more small visual image called “overview”, switches to using internal overviews in the visual asset
• updates figures
• episode 7 is untouched since it’s a vector episode and doesn’t depend on the endpoint

I'll check the other episodes after episode 8 tomorrow when I'm back online. Thanks for the reviews so far @fnattino !