[Raster] Gaps in mosaic response

[charalamm]

Problem description

The mosaic returned by the /mosaic/<search-id>/tiles/WebMercatorQuad/{z}/{x}/{y}@1x.png?assets=<an-asset> has gaps as shown in the screenshots bellow even though all requests have returned. When I zoom in the regions without any data the data appear again normally.

I think that the gaps appear only when the zoom level is low (more geographical space on the screen). I have checked the logs of the API but nothings seems to be out of the ordinary. The database cpu and ram usage is around 80% but I do not see any failed database connections.

Is it something observed in other deployments as well? Is there something I can do about it?

Environment Information

The raster API is deployed as an azure container app and it uses the eoapi.raster-uvicorn docker image. The image is build locally using this repo docker-compose.uvicorn.yml. The database is deployed as a azure postgres flexible server.

[vincentsarago]

This might be due to:

• nodata value in the raster
• wrong footprint in the STAC metadata (using the polygon created from the bbox instead of using the real data geometry)

There are couple options in titiler-pgstac that you can try:

• exitwhenfull=False
• skipcovered=False

link: https://stac-utils.github.io/titiler-pgstac/mosaic_endpoints/#tiles

[charalamm]

@vincentsarago Thanks for your quick response!

I copied the sentinel-2-l2a collection from https://earth-search.aws.element84.com/v1 so I do not think they have wrong metadata there. Also, when I zoom further in the empty regions the data appear normally so I don't think it is because of nodata values.

I am not that experienced so please correct me if those statements could be wrong.

Furthermore, I compared visually the results between a local run of the raster API and the container app and I see that the local run does not have this empty regions. Both deployments are connected to the same database. The container app is scaling up to 20 containers to handle concurrent requests. Could the empty regions be related to concurrent requests not being handled by the same API runtime?

[vincentsarago]

Furthermore, I compared visually the results between a local run of the raster API and the container app and I see that the local run does not have this empty regions.

well if it works locally, it means there is something going on with the container app: maybe it's a timeout issue or a memory issue 🤷

[charalamm]

I think that there are some limits in the azure networking that limits the number of requests done to read the files. I am closing. Thanks!

[charalamm]

Hello again @vincentsarago, I would like to add some more information about this issue and maybe reopen it if you think it is worth it.

I have created a low resolution mosaic raster that covers the whole Africa out of a ~40GB dataset. The final raster has is 150 KB. I have it stored in an azure storage account. The collection that holds this raster has only one item and the search for serving that collection is supposed to work between zoom levels 0 and 3 (although the raster api returns responses for higher zoom levels as well). The eoAPI raster service is running on a 2core CPU - 4GB RAM azure container app that scales 0-100 containers and each container serves up to 4 requests (I have tried also with 20 request and had the same results). Finally my database has 1 core, 2 GB and 640 max iops.

What I am observing is that after scrolling in the frontend to the desired zoom levels, I get many 404 responses for the requests in areas outside of my collection, and pending for the areas inside the collection. Some time passes and I see that some pending requests return an image and some return 404, although I am 100% sure my raster has data for those regions as well.

This is the search URL I am talking about in case you want to have a look: https://eoapi-raster.kindstone-c09394a7.westeurope.azurecontainerapps.io/mosaic/fd86a7513cd5235cb8796016d32529b3/tiles/WebMercatorQuad/{z}/{x}/{y}@1x.png?assets=ESA_WorldCover_10m_2020_v100_Map&rescale=0,100&exitwhenfull=False&skipcovered=False

[charalamm]

I broke down that one file to 12 files according to the tiles of zoom level 3 of the epsg:3857 and it was fixed. So, I guess if too many request try to read the same file, then the tiler responds with 404. Maybe that is an azure blob storage problem