search

landam / grass-gis-git-migration-test

0 stars 0 forks source link

Read lidar data as vector points using PDAL #172

Open landam opened 5 years ago

landam commented 5 years ago

Reported by wenzeslaus on 4 Sep 2015 15:53 UTC http://www.pdal.io/ PDAL] is a point cloud processing and format transformation library and supersedes [http://www.liblas.org/ libLAS (currently used in r.in.lidar and v.in.lidar) in many ways. Currently the most important difference is support for LAS 1.4. It also seems that we will have to replace libLAS by PDAL at some point.

I'm attaching a patch which is adding a module v.in.pointcloud. It is based on v.in.lidar and the difference is that it is using PDAL C++ API for reading LAS (so it can read LAS 1.4 except for waveform). LAZ is supported in the same way as with libLAS. When PDAL is compiled with LAZ, then v.in.pointcloud supports LAZ.

Serious issues with the patch:

Potential improvements to patch:

Adding PDAL processing into the import or as separate module is clearly a possibility as well but this is out of scope of this particular patch.

General issues with PDAL for GRASS GIS:

I need help with making the compilation check in configure compile C++, otherwise I think it can go trunk.

GRASS GIS version and provenance

svn-trunk

Migrated-From: https://trac.osgeo.org/grass/ticket/2732

landam commented 5 years ago

Attachment from wenzeslaus on 4 Sep 2015 15:55 UTC derived from v.in.lidar and using standard PDAL API with LasReader only https://trac.osgeo.org/grass/attachment/ticket/2732/vinpointcloud.diff

landam commented 5 years ago

Modified by wenzeslaus on 4 Sep 2015 15:56 UTC

landam commented 5 years ago

Comment by wenzeslaus on 5 Sep 2015 22:52 UTC To try the patch you must http://www.pdal.io/compilation/unix.html compile PDAL which in short looks like:

git clone git@github.com:PDAL/PDAL.git pdal
cd pdal
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX:string=/pdal/install/dir

You may need to install some dependencies first (//cmake// will tell what).

Then apply the patch and add to your ./configure call for GRASS:

--with-pdal=yes --with-pdal-config=/pdal/install/dir/bin/pdal-config

It seems that PDAL is not generating //pdal-config// properly, until this gets solved, you need to find the file (/pdal/install/dir/bin/pdal-config) and add the following at the beginning (you may need to chmod u+w /pdal/install/dir/bin/pdal-config before editing the file):

INCLUDES="-I/pdal/install/dir/include/"
LIBS="-L/pdal/install/dir/lib/ -lpdalcpp -lpdal_util -lstdc++"

Unless you install system-wide and configure dynamic libraries, you need to do (for both GRASS compilation and running):

export LD_LIBRARY_PATH="/pdal/install/dir/lib:$LD_LIBRARY_PATH"
landam commented 5 years ago

Comment by wenzeslaus on 10 Sep 2015 14:34 UTC pdal-config is https://github.com/PDAL/PDAL/issues/984 fixed now, so fix of INCLUDES and LIBS in the file /pdal/install/dir/bin/pdal-config is necessary.

To configure GRASS with PDAL, just compile PDAL and then run GRASS ./configure with:

--with-pdal=/home/vasek/dev/grass/pdal/pdal-bin/bin/pdal-config

Please note that the previous instructions said --with-pdal-config= but this is a wrong syntax.

LD_LIBRARY_PATH modification is always need for ./configure, compilation as well as running unless you do ldconfig as part of the installation (I'm not sure but for experimental purposes, the variable seems better, although less convenient).

landam commented 5 years ago

Modified by neteler on 24 Nov 2015 08:02 UTC

landam commented 5 years ago

Comment by wenzeslaus on 30 Dec 2015 19:12 UTC First version now available in trunk. PDAL compilation support added in https://trac.osgeo.org/grass/changeset/67293 and ''v.in.pdal'' in https://trac.osgeo.org/grass/changeset/67436. Configure GRASS with:

./configure ... --with-pdal="/path/to/pdal-config"

PDAL can be compiled with or without LAZ (LAZlib) support and PCL support, so not all the functions may be available. If you compiled PDAL yourself you need to do also:

export LD_LIBRARY_PATH="/path/to/pdal/install/lib:$LD_LIBRARY_PATH"

I decided that the name ''v.in.pdal'' will be best because there are some PDAL-specific functions included. This includes also reprojection during import. However, the name can be changed, v.in.points sounds good as well.

The next steps are:

Things like mask or various decimations have (or can have) GRASS implementation and have PDAL implementation as well. It would make sense to do these operations at the beginning and at the end of the import process. If it would be at the beginning, only PDAL implementations can be used.

Note also that there are still some unresolved issues related to PDAL ability to handle large amounts of points although some solutions were proposed on the PDAL mailing list.

Finally, note also that PDAL is a processing library, so it can be also used in a separate module which would process vector points.

landam commented 5 years ago

Comment by neteler on 30 Dec 2015 22:41 UTC Replying to [comment:5 wenzeslaus]:

An idea could be to put an optional RGB point colorization here. It is a simple PDAL filter, hence v.in.pdal could read from an additionally specified RGB map retrieve the individual colors during import.

landam commented 5 years ago

Comment by wenzeslaus on 31 Dec 2015 00:10 UTC Replying to [comment:6 neteler]:

Replying to [comment:5 wenzeslaus]:

An idea could be to put an optional RGB point colorization here. It is a simple PDAL filter, hence v.in.pdal could read from an additionally specified RGB map retrieve the individual colors during import.

Yes, this is actually on my list already. PDAL colorization could do a GeoTIFF while GRASS raster map would have to be done using native colorization implemented in GRASS. I think that this is exactly where https://lists.osgeo.org/pipermail/grass-dev/2015-September/076552.html using layers and categories for something else than ID and class will be necessary.

landam commented 5 years ago

Comment by sebastic on 20 Mar 2016 21:34 UTC FYI: PDAL has been https://tracker.debian.org/pkg/pdal packaged for Debian and derivatives like Ubuntu.

landam commented 5 years ago

Comment by wenzeslaus on 21 Mar 2016 01:29 UTC Replying to [comment:8 sebastic]:

FYI: PDAL has been https://tracker.debian.org/pkg/pdal packaged for Debian and derivatives like Ubuntu.

Thanks for the info. That's great. How do I install it? I can't find .deb, PPA or instructions at the http://www.pdal.io/download.html official website.

landam commented 5 years ago

Comment by sebastic on 21 Mar 2016 05:23 UTC Replying to [comment:9 wenzeslaus]:

How do I install it? I can't find .deb, PPA or instructions at the http://www.pdal.io/download.html official website.

Wait for the package to reach the mirrors, and then install it on a Debian unstable system with: apt-get install pdal libpdal-dev.

The former gets you the pdal commandline application, the latter the headers, libraries & pdal-config.

There is no PPA for Ubuntu, the package will be synced from Debian unstable into the Ubuntu release after xenial.

The PDAL website doesn't mention the Debian package because it is new and part of the distribution and not a 3rd party repository.

landam commented 5 years ago

Comment by neteler on 5 May 2016 14:08 UTC Milestone renamed

landam commented 5 years ago

Comment by wenzeslaus on 10 Jul 2016 14:00 UTC This is for the next release, although some code is already in place and backports are possible in this case.

landam commented 5 years ago

Comment by @landam on 27 Aug 2016 13:42 UTC Milestone renamed

landam commented 5 years ago

Modified by ptschrum on 19 May 2017 22:38 UTC

landam commented 5 years ago

Comment by hobu on 22 May 2017 03:34 UTC There have been a number of PDAL developments since this ticket was first developed that might be interesting in relation to this task:

1) The PDAL https://github.com/PDAL/PDAL/blob/master/pdal/PipelineExecutor.hpp PipelinExecutor class allows you to insert PDAL Pipeline JSON and get back PDAL PointViews quite easily. This means no more manual or laborious pdal::Stage construction. The executor class can hide all of that for you. Dig in the Python or Java extensions for inspiration on how to use it.

2) PDAL is now available via OSGeo4W64. We won't be committing to maintaining the 32-bit builds, but the 64-bit builds will be generated as part of PDAL's continuous integration and included in all major PDAL releases.

landam commented 5 years ago

Comment by ptschrum on 5 Jun 2017 15:31 UTC I (ptschrum) am working on this as a GSoC project, 2017.

landam commented 5 years ago

Comment by hellik on 14 Dec 2017 19:29 UTC citing a https://trac.osgeo.org/osgeo4w/ticket/550#comment:4 OSGeo4W ticket:

>no chance to fix the broken liblas/laszip stack?

There is a significant amount of software development to catch it up to the latest LASzip release. The LASzip API that libLAS used is no longer supported, and the new LASzip API is very different from the one that libLAS (and previously PDAL) used.

It would be better long term to get GRASS using PDAL. There was some effort on that topic, but I think it has stalled.
landam commented 5 years ago

Comment by hellik on 14 Dec 2017 19:30 UTC wiki of the https://trac.osgeo.org/grass/wiki/GSoC/2017/IntegrationOfPDALintoGRASSGIS GSoC 2017 project to integrate PDAL into GRASS.

landam commented 5 years ago

Comment by neteler on 26 Jan 2018 11:40 UTC Ticket retargeted after milestone closed

landam commented 5 years ago

Comment by wenzeslaus on 16 Feb 2018 03:11 UTC In [changeset:"72246\"] https://trac.osgeo.org/grass/changeset/72246:

v.in.pdal: change API calls to PDAL 1.6.0, see https://trac.osgeo.org/grass/ticket/3496 and #172 (author: felixg)
landam commented 5 years ago

Comment by wenzeslaus on 16 Feb 2018 04:06 UTC There is currently no owner of this issue.

landam commented 5 years ago

Comment by neteler on 18 Feb 2018 11:09 UTC In [changeset:"72253\"] https://trac.osgeo.org/grass/changeset/72253:

v.in.pdal: change API calls to PDAL 1.6.0, see https://trac.osgeo.org/grass/ticket/3496 and #172 (author: felixg, trunk https://trac.osgeo.org/grass/changeset/72246) + GPL header added in https://trac.osgeo.org/grass/changeset/72245
landam commented 5 years ago

Comment by wenzeslaus on 13 Mar 2018 13:55 UTC In [changeset:"72363\"] https://trac.osgeo.org/grass/changeset/72363:

v.in.pdal: change API calls to PDAL 1.6.0, see https://trac.osgeo.org/grass/ticket/3496, https://trac.osgeo.org/grass/ticket/3243, https://trac.osgeo.org/grass/ticket/3101, #172 (author: felixg, backport of trunk https://trac.osgeo.org/grass/changeset/72246) + GPL header added in https://trac.osgeo.org/grass/changeset/72245
landam commented 5 years ago

Modified by neteler on 12 Jun 2018 20:48 UTC

landam commented 5 years ago

Modified by @landam on 25 Aug 2018 16:03 UTC

landam commented 5 years ago

Comment by wenzeslaus on 25 Aug 2018 23:32 UTC To make this useful, we will need PDAL as a dependency in distributions. I don't know the procedure. Should I create ticket(s) for this here?

landam commented 5 years ago

Comment by hellik on 26 Aug 2018 04:12 UTC Replying to [comment:26 wenzeslaus]:

To make this useful, we will need PDAL as a dependency in distributions. I don't know the procedure. Should I create ticket(s) for this here?

recent pdal versions (pdal-1.7.2) are only available in 64bit OSGeo4W.

32bit seems not to be updated for many years:

pdal-0.9.8-4.tar.bz2 05-Aug-2013 13:41 1.2M

landam commented 5 years ago

Comment by sebastic on 26 Aug 2018 08:10 UTC You also need to ensure that LAZ support is optional, as LASzip has a license issue which makes it unredistributable, see:

landam commented 5 years ago

Comment by @landam on 26 Aug 2018 13:21 UTC Replying to [comment:27 hellik]:

recent pdal versions (pdal-1.7.2) are only available in 64bit OSGeo4W.

unfortunately pdal has a lot of dependencies:

eigen   (3.2.1-1)
    The Eigen library.
    Required by: pdal

hexer   (1.4.0-1)
    Hexer: GDAL-based hexagon density and boundary binning
    Required by: pdal

jq  (1.5.0-0)
    jq: command-line JSON processor
    Required by: pdal

jsoncpp (1.6.0-1)
    JsonCpp library
    Required by: pdal

laszip  (3.2.2-1)
    The LASzip compression library
    Required by: pdal

laz-perf    (0.0.1-2)
    Alternative LAZ implementation for C/C++ and JavaScript
    Required by: pdal

nitro   (2.8.7-5)
    NITRO: C/C++ library for NITF manipulation
    Required by: pdal

oci (12.1.0.1.0-1)
    Oracle Instant Client
    Required by: pdal

python3-core    (3.6.0-2)
    Python3 core interpreter and runtime
    Required by: python3-numpy, python3-gdal, python3-matplotlib, python3-pytz, python3-six, python3-cycler, python3-python-dateutil, python3-pyparsing

python3-cycler  (0.10.0-1)
    Composable style cycles
    Required by: python3-matplotlib

python3-gdal    (2.2.4-1)
    The GDAL/OGR Python3 Bindings and Scripts
    Required by: pdal

python3-matplotlib  (2.0.0-1)
    Python plotting package
    Required by: pdal

python3-numpy   (1.12.0+mkl-1)
    NumPy: array processing for numbers, strings, records, and objects.
    Required by: pdal, python3-gdal, python3-matplotlib

python3-pyparsing   (2.1.10-1)
    Python parsing module
    Required by: python3-matplotlib

python3-python-dateutil (2.6.0-1)
    Extensions to the standard Python datetime module
    Required by: python3-matplotlib

python3-pytz    (2018.3-1)
    World timezone definitions, modern and historical
    Required by: python3-matplotlib

python3-six (1.11.0-1)
    Python 2 and 3 compatibility utilities
    Required by: python3-matplotlib, python3-cycler, python3-python-dateutil

xz-devel    (5.2.3-1)
    XZ-format compression library - development files
    Required by: pdal

It will enlarge standalone binaries significantly (especially Python3). Would be nice to split OSGeo4W packages into pdal and python3-pdal.

landam commented 5 years ago

Comment by @landam on 26 Aug 2018 20:22 UTC See https://trac.osgeo.org/osgeo4w/ticket/576

landam commented 5 years ago

Comment by hobu on 27 Aug 2018 21:38 UTC

You also need to ensure that LAZ support is optional, as LASzip has a license issue which makes it unredistributable, see:

Unredistributeable according to Debian's interpretations.

landam commented 5 years ago

Comment by wenzeslaus on 27 Aug 2018 23:28 UTC Replying to [comment:31 hobu]:

You also need to ensure that LAZ support is optional, as LASzip has a license issue which makes it unredistributable, see:

Unredistributeable according to Debian's interpretations.

But there is las-perf, right? Are there some issues in using it and replacing LASzip completely?

landam commented 5 years ago

Comment by @landam on 25 Jan 2019 21:07 UTC Ticket retargeted after milestone closed