"Holes" In DEM?

[LucRSquared]

I've been querying data in the following bounding box (xmin, ymin, xmax, ymax)

boundingbox = (-91.0, 40.9, -90.0, 41.7)

After writing the data to a GeoTIFF file and loading it into QGIS I noticed some holes in specific parts of the area at 0 elevation that seem to partially follow the outline of the river

However when I looked at the DEP3Elevation layer directly from the WMS in QGIS that area was correctly filled in (Unfortunately at the time I'm writing this it seems the service is down so I cannot show it).

Also I tried querying that same area a couple of times and the same "holes" were there.

My gut feeling is that it isn't a py3dep problem but a server side issue but I'd be curious to know what's happening here!

[cheginit]

Thanks for reporting this issue. Are you passing https://elevation.nationalmap.gov/arcgis/rest/services/3DEPElevation/ImageServer to QGIS?

Regarding the 3DEP service, unfortunately, yes, it's been flaky since a few weeks ago, I will get in touch with the 3DEP team and ask about it.

[cheginit]

3DEP is operational now. The issue is raw elevation data are note available at all resolutions within this bounding box. For example when I requested the data for 1 km resolution, there's no missing data:

But when I requested data at 10 m resolution I reproduced your plot:

I am guessing that when you "display" the WMS layer in QGIS, it first gets the data at coarse resolutions but if you request data for finer you might get the same results that you got here.

Regardless, I don't think the service is supposed to be behaving this way. It's supposed to generate seamless maps. I will get in touch with the 3DEP team about this.

[LucRSquared]

That's right I was requesting it at 10 meters resolution! I don't have my work laptop to double check but I'm pretty sure I'm using the link you provided for the WMS. Although in QGIS it seems to produce a similar resolution in that area. I'll double check again on Monday when I return to work.

Thank you again for your quick answer!

[LucRSquared]

Update:

The WCS layer indeed shows some holes (in the foreground is the data obtained with py3dep) in that area

Upon checking I am not using the REST URL but this one:

https://elevation.nationalmap.gov/arcgis/services/3DEPElevation/ImageServer/WCSServer

Point being: it's definitely a problem on their end :(

One thing though is that if I check the resolution availability on that dark area it says the 10m resolution is available (?):

[cheginit]

I just got a response from the 3DEP team. They said:

What's most likely happened is data was present there at the time of last mosaic generation, and now is missing (possibly due to a new product release that wasn't uploaded to the data store properly).

They also said that they will fix it soon. I will let you know as soon as I hear from them.

You are using the WCS protocol instead of WMS (which py3dep is using). It doesn't make any difference in this case since the issue is with the underlying data.

One thing though is that if I check the resolution availability on that dark area it says the 10m resolution is available (?):

Right, 10 m and 30 m data are available and the 3DEP web service is supposed used those for returning DEMs at any resolution. I think once they fix the mosaicing issue, you'll be able to get a seamless DEM.

[LucRSquared]

Awesome! Thank you for the feedback from the team. I'm leaving this issue open in case you want to wait for them to fix it before closing it.

[cheginit]

The problem is fixed now. Just make sure to remove your cache file (./cache/aiohttp_cache.sqlite) so it gets the new data from the service instead of the old data from the cache:

This is the message that I got from the 3DEP team regarding the issue:

It seems the service was retaining cached MRF images indefinitely, which prevented corrections to the source data from being propagated into the service display. This resulted in an area of older DEMs with edge artifacts showing, even though the source DEMs had been re-created without these artifacts (thus the clean version from Downloader). We're taking steps to ensure these cached files are flushed regularly, so this doesn't happen again.

[LucRSquared]

Thanks for the follow-up! I'm glad it helped optimize the whole process.