Support GDAL /vsicurl and /vsizip prefixes to DEM paths

gazebosim / gz-common

An audio-visual library supports processing audio and video files, a graphics library can load a variety 3D mesh file formats into a generic in-memory representation, and the core library of Gazebo Common contains functionality that spans Base64 encoding/decoding to thread pools.

Apache License 2.0

11 stars 36 forks source link

Desired behavior

Allow users to supply DEM data to Gazebo that is zipped, or DEM data to Gazebo that is remotely hosted. I would like all of the following able to be supplied to the Load() function:

https://gdal.org/user/virtual_file_systems.html

Alternatives considered

Preventing users from using anything except local raw DEM files and stuff in Fuel.

Implementation suggestion

GDAL can load zipped DEM if you add the /vsizip prefix to the path
GDAL can load remotely hosted DEM if you add /vsicurl/<your_server_address> to the path

Use case:

As a drone user that relies on ArduPilot and grid_map_geo, I want Gazebo to use the same terrain data as my drone's flight controller and path planner.

Example files - these are all zipped DEM hosted on a server

Additional context

You can add the following code block to Dem_TEST.cc and observe gazebo fail to load the data.

Add the code block to the Dem_TEST.cc file:

TEST_F(DemTest, LargeVRT)
{
  // Load a large VRT DEM (used by ArduPilot)
  common::Dem dem;
  EXPECT_EQ(dem.Load("/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt"), 0);
}

Rebuild the repo

Observe the failure


ryan@B650-970:~/Dev/gz_harmonic_ws/src/gz-common$ ./build/gz-common5/bin/UNIT_Dem_TEST 
Running main() from /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest_main.cc
[==========] Running 13 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 13 tests from DemTest
<<<<<<<<<<<<<<<< OMITTED FOR BREVITY >>>>>>>>>>>>>>>>>>>>
[ RUN      ] DemTest.LargeVRT
[Err] [SystemPaths.cc:425] Unable to find file with URI [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [SystemPaths.cc:525] Could not resolve file [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [SystemPaths.cc:425] Unable to find file with URI [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [SystemPaths.cc:525] Could not resolve file [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [Dem.cc:115] Unable to find DEM file[/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt].
/home/ryan/Dev/gz_harmonic_ws/src/gz-common/geospatial/src/Dem_TEST.cc:291: Failure
Expected equality of these values:
dem.Load("/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt")
Which is: -1
0
[  FAILED  ] DemTest.LargeVRT (0 ms)
[----------] 13 tests from DemTest (48 ms total)

[----------] Global test environment tear-down [==========] 13 tests from 1 test suite ran. (48 ms total) [ PASSED ] 12 tests. [ FAILED ] 1 test, listed below: [ FAILED ] DemTest.LargeVRT

1 FAILED TEST


That said, GDAL can load the path `/vsizip/home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/data/ap_srtm1.zip` fine:

$ gdalinfo /vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt <<<<<<<<<<<<<<<< OMITTED FOR BREVITY >>>>>>>>>>>>>>>>>>>> /vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/S84W179.hgt.zip/S84W179.hgt /vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/S84W180.hgt.zip/S84W180.hgt Size is 1296001, 604801 Coordinate System is: GEOGCRS["WGS 84", DATUM["World Geodetic System 1984", ELLIPSOID["WGS 84",6378137,298.257223563, LENGTHUNIT["metre",1]]], PRIMEM["Greenwich",0, ANGLEUNIT["degree",0.0174532925199433]], CS[ellipsoidal,2], AXIS["geodetic latitude (Lat)",north, ORDER[1], ANGLEUNIT["degree",0.0174532925199433]], AXIS["geodetic longitude (Lon)",east, ORDER[2], ANGLEUNIT["degree",0.0174532925199433]], ID["EPSG",4326]] Data axis to CRS axis mapping: 2,1 Origin = (-180.000138888888898,84.000138888888884) Pixel Size = (0.000277777777778,-0.000277777777778) Corner Coordinates: Upper Left (-180.0001389, 84.0001389) (180d 0' 0.50"W, 84d 0' 0.50"N) Lower Left (-180.0001389, -84.0001389) (180d 0' 0.50"W, 84d 0' 0.50"S) Upper Right ( 180.0001389, 84.0001389) (180d 0' 0.50"E, 84d 0' 0.50"N) Lower Right ( 180.0001389, -84.0001389) (180d 0' 0.50"E, 84d 0' 0.50"S) Center ( 0.0000000, -0.0000000) ( 0d 0' 0.00"E, 0d 0' 0.00"S) Band 1 Block=128x128 Type=Int16, ColorInterp=Undefined NoData Value=-32768


Here's more info on the zip file:

$ unzip -l /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/data/ap_srtm1.zip Archive: /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/data/ap_srtm1.zip Length Date Time Name

11758320 2024-01-09 21:34 ap_srtm1.vrt

11758320 1 file



## Long term

I want Gazebo to automatically load terrain wherever my drone flies and expose the necessary services to load more "tiles", move the origin, and not run out of memory. ArduPilot's ground station software of MAVProxy, QGroundControl, and Mission Planner all can do this today. When using Gazebo, the ROS aerial industry currently has to hard code a single DEM tile, and if the drone flies out of bounds of the tile, there is no capability to load new tiles in Gazebo. Being able to just give Gazebo a link to a terrain server and call it good would be awesome.

[==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from DemTest [ RUN ] DemTest.LargeVRT [Err] [SystemPaths.cc:425] Unable to find file with URI [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt] [Err] [SystemPaths.cc:525] Could not resolve file [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt] [Err] [SystemPaths.cc:425] Unable to find file with URI [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt] [Err] [SystemPaths.cc:525] Could not resolve file [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt] [New Thread 0x7fffec2f9640 (LWP 364071)] [Thread 0x7fffec2f9640 (LWP 364071) exited] Thread 1 "UNIT_Dem_TEST" received signal SIGSEGV, Segmentation fault. 0x00007ffff62aefe6 in GDALCopyWords64 () from /lib/libgdal.so.30 (gdb) bt #0 0x00007ffff62aefe6 in GDALCopyWords64 () from /lib/libgdal.so.30 #1 0x00007ffff6160631 in VRTSourcedRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) () from /lib/libgdal.so.30 #2 0x00007ffff627c0b3 in GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) () from /lib/libgdal.so.30 #3 0x00007ffff7faccc7 in gz::common::Dem::LoadData (this=0x7fffffffcdb0) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/geospatial/src/Dem.cc:508 #4 0x00007ffff7faac41 in gz::common::Dem::Load (this=0x7fffffffcdb0, _filename="/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt") at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/geospatial/src/Dem.cc:202 #5 0x000055555557d6a9 in DemTest_LargeVRT_Test::TestBody (this=0x5555556832e0) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/geospatial/src/Dem_TEST.cc:290 #6 0x00005555555c7279 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x5555556832e0, method=&virtual testing::Test::TestBody(), location=0x5555555df94b "the test body") at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2599 #7 0x00005555555bf6e5 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x5555556832e0, method=&virtual testing::Test::TestBody(), location=0x5555555df94b "the test body") at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2635 #8 0x000055555559a7f8 in testing::Test::Run (this=0x5555556832e0) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2674 #9 0x000055555559b2cb in testing::TestInfo::Run (this=0x555555689760) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2853 #10 0x000055555559bc09 in testing::TestSuite::Run (this=0x555555652290) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:3012 #11 0x00005555555abe3b in testing::internal::UnitTestImpl::RunAllTests (this=0x555555688230) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:5870 #12 0x00005555555c8300 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x555555688230, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x5555555aba20 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x5555555e03c0 "auxiliary test code (environments or event listeners)") at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2599 #13 0x00005555555c08fd in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x555555688230, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x5555555aba20 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x5555555e03c0 "auxiliary test code (environments or event listeners)") at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2635 --Type <RET> for more, q to quit, c to continue without paging-- #14 0x00005555555aa45d in testing::UnitTest::Run (this=0x55555560f660 <testing::UnitTest::GetInstance()::instance>) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:5444 #15 0x00005555555dbf1f in RUN_ALL_TESTS () at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/include/gtest/gtest.h:2293 #16 0x00005555555dbe98 in main (argc=1, argv=0x7fffffffd398) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest_main.cc:51

gazebosim / gz-common