search

gazebosim / gz-common

An audio-visual library supports processing audio and video files, a graphics library can load a variety 3D mesh file formats into a generic in-memory representation, and the core library of Gazebo Common contains functionality that spans Base64 encoding/decoding to thread pools.
https://gazebosim.org
Apache License 2.0
15 stars 39 forks source link

DEM: Add support for GDAL vsicurl, vsizip support and avoid segfaults with huge VRT datasets #597

Closed Ryanf55 closed 3 months ago

Ryanf55 commented 7 months ago

šŸŽ‰ New feature

Closes #596

Summary

  1. Adds initial support to load super large DEM datasets by exposing an API to limit the size of the loaded DEM.
  2. Adds an alternative way of loading datafiles that make use of features like vsizip and vsicurl that aren't true filepaths.
  3. Adds a compiler warning for if you use new versions of GDAL, which will lose precision in the current implementation, and potentially misrepresent NoData values.
    • This could be taken out, but could be an issue instead. A printed warning could be ok.
  4. Add a few asserts for safety to catch coding errors earlier
  5. The user-configurable size limits can be used to prevent segmentation faults by loading too big of a raster dataset
  6. FIxed overflow and segfault in RasterIO for massive DEM's and overflow risk in vector resize operations

Test it

I added a unit test which is self explanatory.

./build/bin/UNIT_Dem_TEST

I also hacked a demo with one SRTMHGT file, and confirmed it can load. Gazebo does not perform well with this terrain size (3601 x 3601), but it does load. SDF needs ability to truncate the size of the heightmap in order to be usable, or perhaps decimate the data resolution. image

Open Questions

Future Improvements

Checklist

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

Ryanf55 commented 7 months ago

@ahcorde Would you be able to provide another round of feedback on the functionality of this PR. I'm looking for some direction on how to best contribute my time to Gazebo's support of terrain data. We can use Gazebo for outdoor 3D aerial robotics applications that the aerial working group is currently working on.

codecov[bot] commented 7 months ago

Codecov Report

Attention: Patch coverage is 90.90909% with 4 lines in your changes missing coverage. Please review.

Project coverage is 80.54%. Comparing base (2d06bf7) to head (7e8b197). Report is 26 commits behind head on gz-common5.

Files Patch % Lines
geospatial/src/Dem.cc 90.90% 4 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## gz-common5 #597 +/- ## ============================================== - Coverage 83.65% 80.54% -3.11% ============================================== Files 90 93 +3 Lines 10273 13617 +3344 ============================================== + Hits 8594 10968 +2374 - Misses 1679 2649 +970 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

azeey commented 6 months ago

Also, please merge from gz-common5 to get the latest changes that fix CI failures.

azeey commented 3 months ago

@Ryanf55 we are now in Feature freeze for Gazebo Ionic. Since PR is already open, I've applied the beta label which means it's slotted for the Ionic release if we can get it merged before 08/28/2024. Do you think that's feasible? If not, please let me know and I'll remove the beta label.

Ryanf55 commented 3 months ago

Thanks for the heads up, I'll do my best to make time over the next 2 days.

Ryanf55 commented 3 months ago

gz-common5

Rebase complete.