unable to install with current documentation

[sohosys]

followed http://applied-geosolutions.github.io/lidar2dems/doc/installation.html to the letter. Ubuntu 14.04 fresh AWS instance, su root, no apparmor. Ran into three issues, unable to resolve third. Not sure if the first two issues contributed to the third.

Anything obvious stand out?

1. LASzip 2.2.0 appears incompatible, installed 2.0.2 instead.

2. lidar2dems install script complains as follows: ERROR: /bin/sh: 1: xslt-config: not found make sure the development packages of libxml2 and libxslt are installed resolved with apt-get install libxslt-dev

3. nosetests test -v fails as follows, unable to resolve:

root@ip-172-31-1-191:/usr/src/lidar2dems# nosetests test -v Test classification ... ERROR Test creating density ... sh: 1: gdalbuildvrt: not found ok Create DTM ... ERROR Create DSM ... ERROR Create CHM ... runtime error ERROR Test getting classification filename ... ok Test finding las files ... ok

====================================================================== ERROR: Test classification

Traceback (most recent call last): File "/usr/src/lidar2dems/test/pdal_test.py", line 66, in test0_classify fouts = find_classified_lasfile(self.testdir, site=f) File "/usr/src/lidar2dems/l2d/utils.py", line 118, in find_classified_lasfile raise Exception("No classified LAS files found") Exception: No classified LAS files found -------------------- >> begin captured stdout << --------------------- Classifying 4 files into test/testdir/features-0_l2d_s1c3.las Created merged file test/las/fa61ceb5-b9d8-4130-acfd-99c201146e4c.las in 0:00:00.496511 pdal ground -i /usr/src/lidar2dems/test/las/fa61ceb5-b9d8-4130-acfd-99c201146e4c.las -o /usr/src/lidar2dems/test/testdir/features-0_l2d_s1c3.las --slope 1 --cellSize 3 --maxWindowSize 10 --maxDistance 1 --classify Created test/testdir/features-0_l2d_s1c3.las in 0:00:01.146856 Classifying 4 files into test/testdir/features-1_l2d_s1c2.las Created merged file test/las/c1bbe0b0-8bb6-4af4-b3ab-b0669011750f.las in 0:00:00.487594 pdal ground -i /usr/src/lidar2dems/test/las/c1bbe0b0-8bb6-4af4-b3ab-b0669011750f.las -o /usr/src/lidar2dems/test/testdir/features-1_l2d_s1c2.las --slope 1 --cellSize 2 --maxWindowSize 10 --maxDistance 1 --classify Created test/testdir/features-1_l2d_s1c2.las in 0:00:00.998046

--------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- shapely.geos: DEBUG: Trying CDLL(libgeos_c.so.1) shapely.geos: DEBUG: Library path: 'libgeos_c.so.1' shapely.geos: DEBUG: DLL: <CDLL 'libgeos_c.so.1', handle 18afd70 at 7f2eab5c65d0> shapely.geos: DEBUG: Trying CDLL(libc.so.6) shapely.geos: DEBUG: Library path: 'libc.so.6' shapely.geos: DEBUG: DLL: <CDLL 'libc.so.6', handle 7f2ebe7cc9b0 at 7f2eab5c6650> --------------------- >> end captured logging << ---------------------

====================================================================== ERROR: Create DTM

Traceback (most recent call last): File "/usr/src/lidar2dems/test/pdal_test.py", line 87, in test2_create_dtm lasfiles = find_classified_lasfile(self.testdir, site=f) File "/usr/src/lidar2dems/l2d/utils.py", line 118, in find_classified_lasfile raise Exception("No classified LAS files found") Exception: No classified LAS files found -------------------- >> begin captured stdout << --------------------- Completed test/testdir/features-0_dtm_r0.56 [den min max idw] in 0:00:00.000939 Completed test/testdir/features-0_dtm_r1.41 [den min max idw] in 0:00:00.000823 Completed test/testdir/features-0_dtm_r2.50 [den min max idw] in 0:00:00.000818

--------------------- >> end captured stdout << ----------------------

====================================================================== ERROR: Create DSM

Traceback (most recent call last): File "/usr/src/lidar2dems/test/pdal_test.py", line 104, in test3_create_dsm lasfiles = find_classified_lasfile(self.testdir, site=f) File "/usr/src/lidar2dems/l2d/utils.py", line 118, in find_classified_lasfile raise Exception("No classified LAS files found") Exception: No classified LAS files found -------------------- >> begin captured stdout << --------------------- Completed test/testdir/features-0_dsm_r0.56 [den max] in 0:00:00.000469

--------------------- >> end captured stdout << ----------------------

====================================================================== ERROR: Create CHM

Traceback (most recent call last): File "/usr/src/lidar2dems/test/pdal_test.py", line 123, in test4_create_chm fout = create_chm(fdtm, fdsm, prefix + 'chm.tif') File "/usr/src/lidar2dems/l2d/utils.py", line 127, in create_chm dtm_img = gippy.GeoImage(dtm) File "/usr/local/lib/python2.7/dist-packages/gippy/gippy.py", line 3866, in init this = _gippy.new_GeoImage(*args) RuntimeError: 4: `/usr/src/lidar2dems/test/testdir/features-1_dtm.idw.tif' does not exist in the file system, and is not recognised as a supported dataset name.

---

Ran 7 tests in 2.573s

FAILED (errors=4) root@ip-172-31-1-191:/usr/src/lidar2dems#

[ircwaves]

Thanks for the heads up. We are overdue for some updates...and will be looking into this tomorrow.

[sohosys]

If you see any clues as to what the issue is in the error messages above let me know. If i can get past them I would be happy to contribute an update installation document.

[ircwaves]

My thought is that with the ubuntugis-unstable PPA added (early in the install script), I don't believe the current version of gippy on pypi will install. Can you check your install log, or just try import gippy in the python interpreter?

[sohosys]

import gippy in python returns no error.

[sohosys]

I should add that I did not use the easy-install script. After it failed, I launched a new instance of ubuntu and went through the steps to install each package. The only errors (and resolutions) were steps 1 and 2 in the original post in this thread.

[ircwaves]

Short version:

• gdal-bin was missing from the dependencies enumerated in easy-install.sh
• the tests may have just been misconfigured.

More info:

I've created a branch 31-install-updates that has the small tweak for the tests, and if you install gdal-bin you should be good to go. Additionally, I will be updating the easy-install.sh to use LASzip version 2.0.2 and gippy version 0.3.9 (which is GDAL2 compatible and has some bug fixes).

The fresh build-and-test of 31-install-updates is running and assuming it passes I will merge those changes and post an update to the easy-install.sh.

[ircwaves]

@sohosys -- the updates to the tests have been pushed to master and released. I have added xslt-dev and gdal-bin to the documentation and the easy-install.sh.

[sohosys]

ran the easy-install script over the last hours, and it completed without errors. 'nostests test -v' still producing errors. have not tested otherwise. should the test complete without error?

[sohosys]

the end of script nosetest produced this output:

---

Successfully installed nose Cleaning up...

---

Ran 0 tests in 0.000s

OK

---

running nosetests test -v from within lidar2dems right afterward and got this:

---

nosetests test -v Test classification ... PDAL: kernels.ground: Unexpected argument 'cellSize'.

ERROR Test creating density ... ERROR Create DTM ... ERROR Create DSM ... ERROR Create CHM ... runtime error ERROR Test getting classification filename ... ok Test finding las files ... ok

====================================================================== ERROR: Test classification

Traceback (most recent call last): File "/usr/src/lidar2dems/test/pdal_test.py", line 68, in test0_classify classify(fnames, fout, site=f, slope=slope, cellsize=cellsize) File "/usr/src/lidar2dems/l2d/pdal.py", line 309, in classify os.remove(ftmp) OSError: [Errno 2] No such file or directory: '/usr/src/lidar2dems/test/las/0610aca5-f9ec-4416-8db6-ce3f4dc008aa.las' -------------------- >> begin captured stdout << --------------------- Classifying 4 files into test/testdir/features-0_l2d_s1c3.las Created merged file test/las/0610aca5-f9ec-4416-8db6-ce3f4dc008aa.las in 0:00:00.047096 pdal ground -i /usr/src/lidar2dems/test/las/0610aca5-f9ec-4416-8db6-ce3f4dc008aa.las -o /usr/src/lidar2dems/test/testdir/features-0_l2d_s1c3.las --slope 1 --cellSize 3 --maxWindowSize 10 --maxDistance 1 --classify

--------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- shapely.geos: DEBUG: Trying CDLL(libgeos_c.so.1) shapely.geos: DEBUG: Library path: 'libgeos_c.so.1' shapely.geos: DEBUG: DLL: <CDLL 'libgeos_c.so.1', handle f9b630 at 7ff71e152ed0> shapely.geos: DEBUG: Trying CDLL(libc.so.6) shapely.geos: DEBUG: Library path: 'libc.so.6' shapely.geos: DEBUG: DLL: <CDLL 'libc.so.6', handle 7ff7310f69b0 at 7ff71e152f50> --------------------- >> end captured logging << ---------------------

====================================================================== ERROR: Test creating density

Traceback (most recent call last): File "/usr/src/lidar2dems/test/pdal_test.py", line 77, in test1_create_density fout = create_dems(lasfiles, 'density', site=f, outdir=self.testdir) File "/usr/src/lidar2dems/l2d/pdal.py", line 322, in create_dems radius=rad, site=site, outdir=outdir, suffix=suffix, overwrite=overwrite, **kwargs)) File "/usr/src/lidar2dems/l2d/pdal.py", line 393, in create_dem raise Exception("Error creating dems: %s" % ' '.join(fouts)) Exception: Error creating dems: den -------------------- >> begin captured stdout << --------------------- Creating test/testdir/features-0_density_r0.56 [den] from 3 files

--------------------- >> end captured stdout << ----------------------

====================================================================== ERROR: Create DTM

Traceback (most recent call last): File "/usr/src/lidar2dems/test/pdal_test.py", line 90, in test2_create_dtm lasfiles = find_classified_lasfile(self.testdir, site=f, params=class_params(f)) File "/usr/src/lidar2dems/l2d/utils.py", line 118, in find_classified_lasfile raise Exception("No classified LAS files found") Exception: No classified LAS files found

====================================================================== ERROR: Create DSM

Traceback (most recent call last): File "/usr/src/lidar2dems/test/pdal_test.py", line 107, in test3_create_dsm lasfiles = find_classified_lasfile(self.testdir, site=f, params=class_params(f)) File "/usr/src/lidar2dems/l2d/utils.py", line 118, in find_classified_lasfile raise Exception("No classified LAS files found") Exception: No classified LAS files found

====================================================================== ERROR: Create CHM

Traceback (most recent call last): File "/usr/src/lidar2dems/test/pdal_test.py", line 126, in test4_create_chm fout = create_chm(fdtm, fdsm, prefix + 'chm.tif') File "/usr/src/lidar2dems/l2d/utils.py", line 127, in create_chm dtm_img = gippy.GeoImage(dtm) File "/usr/local/lib/python2.7/dist-packages/gippy/gippy.py", line 3951, in init this = _gippy.new_GeoImage(*args) RuntimeError: 4: `/usr/src/lidar2dems/test/testdir/features-0_dtm.idw.tif' does not exist in the file system, and is not recognised as a supported dataset name.

---

Ran 7 tests in 0.426s

FAILED (errors=5)

---

[sohosys]

in testing each process, it appears the l2d_classify is passing the parameter 'cellSize' to pdal ground instead of the expected 'cell_size'. Did something change in pdal? installed pdal version is 1.0.1 as per the install script.

root@ip-172-31-31-94:/usr/src/lidar2dems/test# l2d_classify las/ -s vectors/features.shp Classifying 3 files into features-0_l2d_s1c3.las Created merged file las/cb46733a-1a25-4e64-9cfc-a0dcde008bbe.las in 0:00:00.023228 pdal ground -i /usr/src/lidar2dems/test/las/cb46733a-1a25-4e64-9cfc-a0dcde008bbe.las -o /usr/src/lidar2dems/test/features-0_l2d_s1c3.las --slope 1 --cellSize 3 --maxWindowSize 10 --maxDistance 1 --classify PDAL: kernels.ground: Unexpected argument 'cellSize'.

[ircwaves]

Yeah, not having the cd lidar2dems was a commit that didn't make its way into there. Strange that the tests are not passing for you. I triggered another Jenkins successful build & test this morning. Perhaps we can compare some of the artifacts to determine what is going wrong. The build does the following:

CONTAINER=l2d-ei
cp /home/icooke/.l2d_easy-install.sh easy-install.sh
if [ "$(lxc list | grep ${CONTAINER})" ] ;
then
    lxc stop ${CONTAINER}
    lxc delete ${CONTAINER}
fi
lxc launch ubuntu-daily:14.04 ${CONTAINER} # lidar2dems easy-install
sleep 5
lxc file push easy-install.sh ${CONTAINER}/root/
sleep 10
lxc exec ${CONTAINER} -- /root/easy-install.sh ${ghprbSourceBranch}

Here is a complete build log from this morning. As the build leaves the container configured at the end, I then ran the tests w/o capturing standard output and got the following:

root@l2d-ei:~/lidar2dems# nosetests -vs test
Test classification ... Classifying 4 files into test/testdir/features-0_l2d_s1c3.las
Created merged file test/las/ed4e9e87-c8ec-476d-b876-9d5a63f99e62.las in 0:00:00.473154
pdal ground -i /root/lidar2dems/test/las/ed4e9e87-c8ec-476d-b876-9d5a63f99e62.las -o /root/lidar2dems/test/testdir/features-0_l2d_s1c3.las --slope 1 --cellSize 3 --maxWindowSize 10 --maxDistance 1 --classify
Created test/testdir/features-0_l2d_s1c3.las in 0:00:01.117787
Classifying 4 files into test/testdir/features-1_l2d_s1c2.las
Created merged file test/las/82df09f8-a032-46c7-b408-7c618b5624e8.las in 0:00:00.461450
pdal ground -i /root/lidar2dems/test/las/82df09f8-a032-46c7-b408-7c618b5624e8.las -o /root/lidar2dems/test/testdir/features-1_l2d_s1c2.las --slope 1 --cellSize 2 --maxWindowSize 10 --maxDistance 1 --classify
Created test/testdir/features-1_l2d_s1c2.las in 0:00:00.967794
Classifying 4 files into test/testdir/features-2_l2d_s5c2.las
Created merged file test/las/d346b7c6-1abd-4d72-9ebe-914550b2184f.las in 0:00:00.448669
pdal ground -i /root/lidar2dems/test/las/d346b7c6-1abd-4d72-9ebe-914550b2184f.las -o /root/lidar2dems/test/testdir/features-2_l2d_s5c2.las --slope 5 --cellSize 2 --maxWindowSize 10 --maxDistance 1 --classify
Created test/testdir/features-2_l2d_s5c2.las in 0:00:00.857414
ok
Test creating density ... Completed test/testdir/features-0_density_r0.56 [den] in 0:00:00.000643
Completed test/testdir/features-1_density_r0.56 [den] in 0:00:00.000560
Completed test/testdir/features-2_density_r0.56 [den] in 0:00:00.000580
ok
Create DTM ... Completed test/testdir/features-0_dtm_r0.56 [den min max idw] in 0:00:00.001268
Completed test/testdir/features-0_dtm_r1.41 [den min max idw] in 0:00:00.001182
Completed test/testdir/features-0_dtm_r2.50 [den min max idw] in 0:00:00.001217
Completed test/testdir/features-1_dtm_r0.56 [den min max idw] in 0:00:00.002392
Completed test/testdir/features-1_dtm_r1.41 [den min max idw] in 0:00:00.001181
Completed test/testdir/features-1_dtm_r2.50 [den min max idw] in 0:00:00.001170
Completed test/testdir/features-2_dtm_r0.56 [den min max idw] in 0:00:00.001177
Completed test/testdir/features-2_dtm_r1.41 [den min max idw] in 0:00:00.001148
Completed test/testdir/features-2_dtm_r2.50 [den min max idw] in 0:00:00.001137
ok
Create DSM ... Completed test/testdir/features-0_dsm_r0.56 [den max] in 0:00:00.000633
Completed test/testdir/features-1_dsm_r0.56 [den max] in 0:00:00.000623
Completed test/testdir/features-2_dsm_r0.56 [den max] in 0:00:00.000626
ok
Create CHM ... ok
Test getting classification filename ... ok
Test finding las files ... ok

----------------------------------------------------------------------
Ran 7 tests in 3.420s

OK

[sohosys]

I was able to get this to install and run. One more change that needs to be made to the easy-install.sh script is that PDAL needs to be checked out to a lower case pdal directory as specified int he manual install steps.

[ircwaves]

Glad to hear it. Out of curiosity, are you collecting your own LiDAR data, or using publicly available datasets?

I'll look at that script next time I'm in there. We also welcome contributions, so fork & PR away if your interested.

[sohosys]

Public data. Building DSMs to perform virtual line of sight testing for wireless applications. Where no public lidar data exists, we are acquiring photogrammetric DSMs already in our working format of GeoTIFF.

[ircwaves]

Very nice. Best of luck with your app.

[sohosys]

off topic, but what are the best practices for improving throughput? Are the tools multi-threaded? will more CPU cores or RAM help?

[stephenhagen]

I don't have precise answer for you on bottleneck being CPU or RAM. We had the best luck increasing throughput the old fashioned way, by dividing our data into separate "polygons", each with a set of "features". We'd fire up one instance of lidar2dems for each polygon. lidar2dems processes all features in the polygon in series, but we'd have ~4 polygons running at one time. I'd be interested to hear what you find...