NED (from USGS) - USA TILES 1/3" Program No Longer Finds a Valid Download Source

[OldSubSailor]

I have finally gotten GDAL working with Ortho4XP again. However the tiles created with the custom dem using USGS are awful. Major airports at high ZL are on huge plateaus, and the tile seems to be surrounded by cliffs. Example of concern: Step 1 : Building vector data for tile +34-099 :

Dealing with airports

• Recycling OSM data from ./OSM_data/+30-100/+34-099/+34-099_airports.osm.bz2

Loading elevation data and smoothing it over airports. Downloading ./Elevation_data/+30-100/N34W099_NED13.img from USGS.

• Min altitude: 0.0 , Max altitude: 0.0 , Mean: 0.0

Something is evidently going wrong here.

I can not recommend using custom dem and USGS at this time.

[d41k4n]

Attaching the downloaded .img and perhaps also the output of gdalinfo for that file would be helpful.

[OldSubSailor]

I have in the Ortho forums. Headed out of town.

Sent from my iPhone

On Aug 5, 2021, at 00:39, d41k4n @.***> wrote:

 Attaching the downloaded .img and perhaps also the output of gdalinfo for that file would be helpful.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[OldSubSailor]

Screenshot that is, not sure what you mean by .img. I re-created the tile, 30, -100 for Abline, Texas using USGA / Gdal again. I don't know why USGS/GDAL is not finding the elevation data per se when J. de Ferranti does for example, except for the KABL airport (high ZL?) as noted above. The tile should pretty much be around 1700 feet elevation, but it is set at 0.0, flat as a board but KABL is set at actual elevation.![image](https://user-

The surrounding tiles remain at the median 1700 feet but the GDAL tile is zero feet.

Sent from my iPhone

On Aug 5, 2021, at 00:39, d41k4n @.***> wrote:

 Attaching the downloaded .img and perhaps also the output of gdalinfo for that file would be helpful.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[d41k4n]

By .img I mean what is mentioned in your comment:

./Elevation_data/+30-100/N34W099_NED13.img

[OldSubSailor]

I’ll look for it when I get back then.

Sent from my iPhone

On Aug 5, 2021, at 11:31, d41k4n @.***> wrote:

 By .img I mean what is mentioned in your comment:

./Elevation_data/+30-100/N34W099_NED13.img

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[OldSubSailor]

This is the full output of the first step Assemble Vector data

Step 1 : Building vector data for tile +32-100 :

-> Dealing with airports

• Recycling OSM data from ./OSM_data/+30-100/+32-100/+32-100_airports.osm.bz2 KABI Abilene Regional Airport 3 runways, lat= 32.41, lon= -99.68 KDYS Dyess Air Force Base 3 runways, lat= 32.43, lon= -99.84 Abilene Executive Airpark 1 runway , lat= 32.45, lon= -99.62 Albany Municipal Airport 1 runway , lat= 32.72, lon= -99.27 Arledge Field 1 runway , lat= 32.91, lon= -99.74 Elmdale Airpark 1 runway , lat= 32.45, lon= -99.65 Flying B Ranch Airstrip 35tx 2 runways, lat= 32.48, lon= -99.83 Gregory Simmons Memorial Airport 1 runway , lat= 32.37, lon= -99.02 Owen Field 1 runway , lat= 32.25, lon= -99.50 Ross Planes Airport 1 runway , lat= 32.14, lon= -99.16 **** Zimmerle Airport 1 runway , lat= 32.27, lon= -99.60 Loading elevation data and smoothing it over airports. Downloading ./Elevation_data/+30-100/N32W100_NED13.img from USGS.
• Min altitude: 0.0 , Max altitude: 0.0 , Mean: 0.0 list index out of range list index out of range list index out of range Auto-patched 16 runways and 47 pieces of taxiway. Flattened 5 helipads. Number of edges at this point: 6752 -> Dealing with roads
• Recycling OSM data from ./OSM_data/+30-100/+32-100/+32-100_big_roads.osm.bz2
• Checking which large roads need levelling.
• Recycling OSM data from ./OSM_data/+30-100/+32-100/+32-100_small_roads.osm.bz2
• Checking which smaller roads need levelling.
• Buffering banked road network as multipolygon. Encoding it.
• Encoding the remaining primary road network as linestrings. Number of edges at this point: 62727 -> Dealing with coastline
• Recycling OSM data from ./OSM_data/+30-100/+32-100/+32-100_coastline.osm.bz2 Number of edges at this point: 62727 -> Dealing with inland water
• Recycling OSM data from ./OSM_data/+30-100/+32-100/+32-100_water.osm.bz2
• Building water multipolygon. Cleaning it. Encoding it. Number of edges at this point: 119839 -> Inserting edges related to the orthophotos grid -> Inserting additional boundary edges for gluing -> Transcription to the files /Volumes/BigHDD/Custom Scenery/zOrtho4XP_+32-100/Data+32-100.poly and .node

Final number of constrained edges : 234664

Completed in 1m41sec.

And the .poly and .node files

---

On Aug 5, 2021, at 11:31 AM, d41k4n @.***> wrote:

By .img I mean what is mentioned in your comment:

./Elevation_data/+30-100/N34W099_NED13.img

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/oscarpilote/Ortho4XP/issues/164#issuecomment-893597553, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM4R6IXATO7DC3FSYDD3JA3T3K4EVANCNFSM5BSDIQ4Q. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[d41k4n]

You didn't include any of the requested info yet (.img and/or gdalinfo output for that file).

[OldSubSailor]

I thought that Isaid that I can’t find it or the gdalinfo, but I had just gotten back from a long trip. Where would they be? That being said. I tried restarting the process and stopping it right after the file appeared in the text body as above; I rebuilt the locate database and searched again. Below are the results: % locate ./Elevation_data/+30-100/N32W100_NED13.img % % locate +30-100/N32W100_NED13.img % % locate gdalinfo /Users/meeeee/.local/share/fish/generated_completions/gdalinfo.fish /usr/local/Cellar/gdal/3.3.1_3/bin/gdalinfo /usr/local/Cellar/gdal/3.3.1_3/lib/python3.9/site-packages/osgeo_utils/samples/gdalinfo.py /usr/local/Cellar/gdal/3.3.1_3/share/man/man1/gdalinfo.1 /usr/local/bin/gdalinfo /usr/local/lib/python3.9/site-packages/osgeo_utils/samples/gdalinfo.py /usr/local/share/man/man1/gdalinfo.1 %

% which +30-100/N32W100_NED13.img +30-100/N32W100_NED13.img not found

The locate command line never found ./Elevation_data/+30-100/N32W100_NED13.img so I can't include it, and nor, evidently the associated gdalinfo file.

On Aug 10, 2021, at 1:05 AM, d41k4n @.***> wrote:

You didn't include any of the requested info yet (.img and/or gdalinfo output for that file).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/oscarpilote/Ortho4XP/issues/164#issuecomment-895753377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM4R6ISVAJQNK3PH4TGY2ZTT4C6RHANCNFSM5BSDIQ4Q. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[d41k4n]

Then that means Ortho4XP didn't download anything - which hints at an issue with the download location for NED (USGS) - likely something has changed server-side.

You could try increasing verbosity to get more info in the message output pane.

[OldSubSailor]

I set verbosity to 3, and got a bit more: Loading elevation data and smoothing it over airports. Downloading ./Elevation_data/+30-100/N32W100_NED13.img from USGS. Server said 'Not Found' Trying alternative naming scheme. Server said 'Not Found'

• Min altitude: 0.0 , Max altitude: 0.0 , Mean: 0.0

I was messing around in Providers/United_States/USGS.lay and the file reads grid_type=webmercator url_template=https://basemap.nationalmap.gov/arcgis/rest/services/USGSImageryOnly/MapServer/tile/{zoom}/{y}/{x} imagery_dir=grouped in_GUI=False

Which may lead to the server not found message perhaps?
My browser gets as far as MapServer/

The full link returns:

Error: Invalid URL Code: 400

[d41k4n]

This has nothing to do with imagery downloads so looking at the .lay file is pointless.

However, the verbose output confirms that the DEM download failed with no data found (probably a 404 error returned by the server). The download URL for the DEM is probably hardcoded somewhere and would have to be replaced with an alternative - if that's even possible.

[OldSubSailor]

I agree, as pointed about above the complete link from a browser returns an Invalid URL, 400 error. https://basemap.nationalmap.gov/arcgis/rest/services/USGSImageryOnly/MapServer will take you to a USGS web page. I think that it would require a return by Oscar who might possibly know how to find and correct the url; if as you say that is even possible. Perhaps the USGS has blocked it?

[d41k4n]

You keep mixing up imagery and elevation data downloads. The 400 return code (= bad request) is about the USGS imagery provider, which isn't necessarily related to the elevation data in scope here (unless the same server provides both data types). Unfortunately there is no other useful information printed in the message pane except "Not found" which usually translates to the HTTP response code 404.

[OldSubSailor]

I see what you mean. I thought that it was both however, or it is an odd coincidence, or perhaps USGS has locked it all out. I seem to remember that one time several years ago, something odd happened with them and all I saw was a white out. Now, I would have to find the exact step in the program where and to where it calls for the elevation data? Which means now I have to learn Python programming.

[OldSubSailor]

So, how about this, does it take in the elevation error direction? source=='NED1/3': if os.path.exists(FNAMES.elevation_data(source,lat,lon)): UI.vprint(2," Recycling ",FNAMES.elevation_data(source,lat,lon)) return 1 UI.vprint(1," Downloading ",FNAMES.elevation_data(source,lat,lon),"from USGS.") url_base='https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/IMG/' usgs_name='USGS_NED_13_n'+str(lat + 1)+'w'+str(-lon).zfill(3)+'_IMG' r=http_request(url_base+usgs_name+'.zip',source,verbose) if not r: UI.vprint(2," Trying alternative naming scheme.")

[d41k4n]

Well, congratulations, you found the location in the code that needs to be modified using an alternative public download source - if there is one.

[OldSubSailor]

https://aws.amazon.com/s3/ Gets to a real page but of what use? The rest of the url link seems to be full of XML text and no data remaining. Of course, I know nothing of XML. StagedProducts downloads an empty file. I can find no method of contacting Oscar. I think that this horse is dead. The elevation link above generates the following text This XML file does not appear to have any style information associated with it. The document tree is shown below.

NoSuchKey The specified key does not exist. StagedProducts/Elevation/13/IMG/ MB6AV7T018RAYN0M TOSL+n+SajEl3XeTjCm2YZwraGa45KB7PYzgPp75WfSEgY12+uZeDiot60lXCRt6hfq0W4vyMYg=

Which in turn leads to the SERVER NOT FOUND error in the program. I can't but feel that there is no hope unless someone knows how to access that "alternative public download source. . ."

[OldSubSailor]

1" is still working.

[oscarpilote]

Made the necessary changes in my local branch for the next update.