cleaning /tmp when site shapefile is not used

[F-Sullivan]

Making preliminary density images when no site or feature shapefile is used throws this error:

Traceback (most recent call last): File "/usr/local/bin/l2d_dems", line 9, in load_entry_point('lidar2dems==0.2.0b1', 'console_scripts', 'l2d_dems')() File "/home/mhanson/work/lidar2dems/l2d/scripts/dems.py", line 88, in main combine(fnames, fouts[product], site=site) File "/home/mhanson/work/lidar2dems/l2d/lidar2dems.py", line 423, in combine bounds = get_vector_bounds(site) File "/home/mhanson/work/lidar2dems/l2d/lidar2dems.py", line 653, in get_vector_bounds extent = vector.Extent() AttributeError: 'list' object has no attribute 'Extent'

After the error is thrown, the script stops running and moves on to the next file, and as a result the tmp directory never gets cleaned out and fills up.

[matthewhanson]

I reproduced this here, will fix this prior to a release (especially since the tutorial says to do this).

[F-Sullivan]

It might clean up the tmp directory automatically after all - there were also several seg faults that might be the actual reason that tmp files aren't removed and /tmp fills.

One workaround that might be useful is building the density image using a shapefile delineating all tiles - not sure if that still might throw a seg fault, but a similar approach seemed to work for a site we were struggling with last week.

[matthewhanson]

Yes, I think the tmp thing is a different issue - since it actually completes and it's a Python error in lidar2dems, not a crash in points2grid. Still, this does need to be completed.

The problem with requiring a shapefile, is that when you first get data you might not actually know where the data is, and using the tile boundaries is inadequate, as it doesn't indicate where there is data within the tiles.

[F-Sullivan]

I'm not sure if the crash isn't from points2grid:

9b0 error 4 in libpts2grd.so[7f0ade42b000+12000] [355257.694281] pdal[61068]: segfault at 0 ip 00007f7d9ae76708 sp 00007ffd46d8f420 error 4 in libpts2grd.so[7f7d9ae69000+12000] [360514.460735] pdal[62218]: segfault at 0 ip 00007f257649c708 sp 00007ffda5680bb0 error 4 in libpts2grd.so[7f257648f000+12000] [361469.252986] pdal[62332]: segfault at 0 ip 00007fe6ee749708 sp 00007ffd33b78210 error 4 in libpts2grd.so[7fe6ee73c000+12000] [362955.066150] pdal[62449]: segfault at 0 ip 00007fbf37a47708 sp 00007ffdbd1e03a0 error 4 in libpts2grd.so[7fbf37a3a000+12000] [364995.040188] pdal[62619]: segfault at 0 ip 00007fbc42ca9708 sp 00007fffe236ca80 error 4 in libpts2grd.so[7fbc42c9c000+12000] [365854.077351] pdal[62843]: segfault at 0 ip (null) sp 00007ffec7be3628 error 14 in pdal[400000+2f000] [366919.570633] pdal[62933]: segfault at 0 ip 00007f71d3a01708 sp 00007ffef6aa0cd0 error 4 in libpts2grd.so[7f71d39f4000+12000] [422231.543629] pdal[68934]: segfault at 0 ip 00007f6265aae708 sp 00007fff732bd9e0 error 4 in libpts2grd.so[7f6265aa1000+12000] [423784.181154] pdal[69196]: segfault at 0 ip 00007fc0873ab708 sp 00007ffdd29b0180 error 4 in libpts2grd.so[7fc08739e000+12000]

I believe one of the problem sites was 243 - the attached image shows the tiles of the site. The output image was 1.5GB, the gray area is all 0s (and they are all 0 underneath the tiles within the gray box). Underneath the tiles above the gray box, there are realistic values for density. It does complete, but there are issues with the output.

Is it possible that the issue that might be causing bad output image is actually the segfault, and is it possibly just due to too large an image/area to process? I'm not sure how that could be fixed other than by adjusting using a shapefile. Granted, the tile boundaries are inadequate, but for a first pass to generate a final image shouldn't they be ok?

[matthewhanson]

so the density image being generated isn't complete? So it could be from points2grid.

Are you suggesting to generate the density with the tile boundaries, as individual polygons...actually that should work fine, because it will include all data areas. I didn't include documentation on the l2d_tiles program, but you're right. Generate the tiles, then run density with that. That will prevent p2g trying to create massive density image at once, which may be what is causing the segfault (ie the reason why we ended up splitting up processing by polygons).

The tiles already exist, so i'd suggest trying this problem one with that.

[F-Sullivan]

I'll give it a go on Polygon_243

[F-Sullivan]

that worked - looks like it was a points2grid issue. We should consider adding documentation for l2d_tiles and include use of the tiles.shp for initial density image runs