Closed abarciauskas-bgse closed 1 month ago
The special sauce of titiler-cmr is that it does not require an asset url for tiling Totally naive question, what is the value of this? Is it easier to integrate with, more accurate, faster, something else?
Another naive question, what does this integration provide access to? Is it all of the data in Earthdata search? Or some other set of data? Asking because, I'm curious how the outcome of this work is similar or different from the ArcGIS integration work intended to integrate data from DAACs into VEDA.
Definitely not a naïve question 😄
There are 2 ways in which the set of datasets titiler-cmr can serve is limited:
titiler-cmr, as it is deployed now, will only work with files in a subset of data in earthdata cloud (S3). This access is possible because the lambda running the service is using the veda-data-reader-dev role which has been allowed to access to certain earthdata cloud buckets. As the lambda is not deployed into a VPC, it cannot access the public internet. This means it cannot access files in NASA's archives over HTTPS. I am pretty sure but can't recall at this moment why, that not being in a VPC is required for this S3 access to earthdata cloud buckets to work. Further, not all DAACs have enabled the veda-data-reader-dev role to be able to access their S3 buckets.
Not all data formats will be supported. It should support most NetCDF4 and COG collections, and even many HDF5 collections. But within those formats, there is creative use of the format that means we can't always guarantee they will work. And then there are many other formats employed by NASA that I would not claim titiler-cmr will work with, such as binary.
In terms of communication, in brief I would say something like:
"titiler-cmr can directly create tiles from many collections in NASA's Earthdata cloud so long as they are in NetCDF4 or COG and are using conventional dimensions for latitude, longitude and time. But titiler-cmr is a relatively new technology which has had limited testing so we welcome new use cases to be tested with titiler-cmr so we can expand it's application".
Thanks for writing down the thoughts @abarciauskas-bgse
As we have more and more types of layers than we initially thought,
We now have more types of layers than we initially planned. They have different ways of getting the data they need, but eventually, most of them are displayed as a raster layer in Mapbox. I was wondering if it would be better to separate out the layer for drawing so it can be used throughout the layers and make each different layer pass the information that the raster layer needs (tile URL and parameters).
What you described here basically aligns with my thoughts, and I liked that this is scoped down to the zarr and cmr-related layers now.
As @sandrahoang686 mentioned in pr #436, it will be helpful to know the schema of each type of response. Where can we find these?
helpful to know the schema of each type of response. Where can we find these?
I think you are asking we can know the schema of the response from STAC for each dataset type. We can definitely declare that. But I think we may also want to consider that the workflow is different, not just the type.
You probably know all of this but I am just thinking through how we would need different interfaces for the request + response needed for each dataset type 🤔 .
I think we agree that there could definitely be a way to extract out so there is just one paint layer class. I have done that for the latter 3 layers which all use raster paint layer, but it could probably be improved, made cleaner and benefit from more typed interfaces for the responses.
I could try and refactor raster timeseries to use raster paint layer but am not as comfortable with all of the things that are currently also happening in the raster timeseries module, so I am probably not the best person to do it.
Closing since all the follow-up issues are created.
Context
Titiler-cmr offers a way to dynamically return map tiles via NASA's Common Metadata Repository (CMR). The special sauce of titiler-cmr is that it does not require an asset url for tiling - it handles querying CMR for items and assets and then delivers tiles in response. This is similar to how titiler-pgstac works.
We want to demonstrate the use of titiler-cmr in VEDA in case it could be useful to other NASA applications or extend the set of data available to instances of VEDA.
titiler-cmr integration poses a challenge because it doesn't fit into the same model as any of the existing layer configurations. It is similar to how the
MapLayerRasterTimeseries
layer works with pgSTAC in that both submit a query to the tile API which handles querying the data catalog. But in the case of the existing "raster" type, there is no initial query to the STAC API (for tiling that is). A query is sent to the tile api (titiler-pgstac) which returns a mosaic URL which is used in the tile layer as the source. In this new case, we want to submit the catalog query parameters (to CMR in this case) as part of the tile layer source.Having a STAC request where the STAC response is used as the parameters to the tile API (i.e. not a query to the mosaic tile registration endpoint) is more similar to how the "zarr" and "cmr" types work. However, the existing "zarr" and "cmr" types both query a STAC endpoint for an asset URL and use that asset url in the parameters sent to the tiling request. In the case of "zarr" it uses the asset URL from the collection's assets and in the case of "cmr" it uses the asset url from an item search[^1].
In this new scenario, we don't want to use an asset URL in the tile parameters, but the collection concept id and renders data in the STAC response to the tile endpoint.
Here is an imperfect diagram of the scenarios we support for raster layers, with the scenario in red with dotted lines representing the scenario we want to support:
How things work now
From what I understand about how the code currently works for raster layers:
type
value you can set on the collection which can be "raster", "cmr" or "zarr" (not including vector in this discussion)MapLayerRasterTimeseries
will submit a search payload to the STAC endpoint and then use response's href (mosaicUrl) to create a tile layer.Proposal
mosaicLayer
except that the raster layer has to first fetch the mosaic URL instead of using the asset URL. I propose changing this name to be more generic since we are already using it not just for Zarr (but for netCDFs). I would imagine there is someway to refactorMapLayerRasterTimeseries
to extract the duplicate code to be used in all raster-based layers.useCmr
will becomeuseCmrSTAC
and have the typecmr-stac
,useZarr
can be left as is, and the new type will betitiler-cmr
which will have auseTitilerCmr
hookUltimately, we will have a RasterPaintLayer which accepts the tile parameters returned from the use hook. The use hooks will have type-specific logic for how to query STAC and combine the results with the collection configuration to define a tile layer in the generic
RasterPaintLayer
class.Some thoughts on generalization
We could potentially generalize this further. Since the STAC API and TILER API endpoints are defined in the configuration file, what differs in these scenarios that we need to account for in the layer configuration is:
Interested in your thoughts @anayeaye @sandrahoang686 @hanbyul-here and @j08lue @aboydnw as an FYI.
[^1]: In the case of CMR item search, the selected item is just the first one so if multiple items are returned this is a limitation in the design and implementation - i.e. no way to merge multiple items.