Normalizing after clipping.

[bw4sz]

A bit hard to make reproducible, but give me a sense for whether this is avoidable behavior.

I can normalize a point cloud before, but not after clipping. My thought was that it would be much faster to normalize the clipped cloud, since its a dramatically smaller area. The process will need to be 10,000s of times, so performance is somewhat of an issue, but I can work around it if needed.

laz_path
'data/SJER/NEON_D17_SJER_DP1_257000_4110000_classified_point_cloud_colorized.laz'

Original data, non-normalized

pc.data.points.z.min()
58.948
pc.data.points.z.max()
630.018

Normalizes and crops just fine

pc.normalize(5)
pc.filter(min = -5, max = 100, dim = "z")
clipped=pc.clip(poly)
clipped.data.points.z.min()
-0.7969999999999686
clipped.data.points.z.max()
11.634999999999991

But you can't first crop and then normalize

clipped=pc.clip(poly)

clipped.normalize(1)

Traceback (most recent call last):
  Debug Probe, prompt 131, line 3
    '''
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pyfor/cloud.py", line 262, in normalize
    filter.normalize(cell_size)
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pyfor/ground_filter.py", line 243, in normalize
    bem = self.bem(cell_size)
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pyfor/ground_filter.py", line 217, in bem
    ground_cloud = self.ground_points
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pyfor/ground_filter.py", line 206, in ground_points
    ground = self._filter()
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pyfor/ground_filter.py", line 196, in _filter
    return self.cloud.data.points.loc[ix[ground_bins]]
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pandas/core/indexing.py", line 1478, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pandas/core/indexing.py", line 1901, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pandas/core/indexing.py", line 1143, in _getitem_iterable
    self._validate_read_indexer(key, indexer, axis)
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pandas/core/indexing.py", line 1206, in _validate_read_indexer
    key=key, axis=self.obj._get_axis_name(axis)))
builtins.KeyError: 'None of [[35506496. 41295890. 35505383. ... 44116653. 44116654. 44116655.]] are in the [index]'

Maybe need to reset index for pandas frame?

[brycefrank]

Ben,

Do you have an external DEM available? If so I have implemented this in 0.3.2.

Thoughts?

[brycefrank]

Ben, here is my code. It ran on my system fine. I am using the laz_fix branch, but I don't think that would effect clip. (The first two lines just load my local installation of pyfor-laz_fix).

Try using this on the testing data. It should be installed in your local copy of pyfor.

from importlib.machinery import SourceFileLoader
pyfor = SourceFileLoader('pyfor', '/home/bryce/Programming/pyfor/pyfor/__init__.py').load_module()
import geopandas as gpd

pc = pyfor.cloud.Cloud('/home/bryce/Programming/pyfor/pyfortest/data/test.las')
poly = gpd.read_file('/home/bryce/Programming/pyfor/pyfortest/data/clip.shp')
poly = poly.iloc[0]['geometry']
pc.clip(poly)
pc.normalize(1)

Also I would suggest, at the very least, normalizing using a buffered area of your interest area. Best practices are to generate a DEM using the entire acquisition because the generation of the DEM (can be) spatially dependent on neighboring pixels.

[bw4sz]

Its actually alittle subtle. I 100% agree that your code works, but if you replace

>>> clipped=pc.clip(poly)
>>> clipped.normalize(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ben/Documents/pyfor/pyfor/cloud.py", line 262, in normalize
    filter.normalize(cell_size)
  File "/Users/ben/Documents/pyfor/pyfor/ground_filter.py", line 243, in normalize
    bem = self.bem(cell_size)
  File "/Users/ben/Documents/pyfor/pyfor/ground_filter.py", line 217, in bem
    ground_cloud = self.ground_points
  File "/Users/ben/Documents/pyfor/pyfor/ground_filter.py", line 206, in ground_points
    ground = self._filter()
  File "/Users/ben/Documents/pyfor/pyfor/ground_filter.py", line 196, in _filter
    return self.cloud.data.points.loc[ix[ground_bins]]
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pandas/core/indexing.py", line 1478, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pandas/core/indexing.py", line 1901, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pandas/core/indexing.py", line 1143, in _getitem_iterable
    self._validate_read_indexer(key, indexer, axis)
  File "/Users/ben/miniconda3/envs/retinanet/lib/python3.6/site-packages/pandas/core/indexing.py", line 1206, in _validate_read_indexer
    key=key, axis=self.obj._get_axis_name(axis)))
KeyError: 'None of [[179275. 179277. 179885. 179886. 180015. 177832. 177834. 178406. 176127.\n 176278. 176924. 176926. 174715. 174716. 175319. 175321. 175495. 174722.\n 174723. 175310. 175312. 173228. 173229. 173803. 173804. 173926. 168973.\n 168975. 169525. 169694. 166841. 166843. 167430. 167519. 166060. 166062.]] are in the [index]'
>>>

I take it that clip is meant to be an in-memory replacement. That's totally fine with my use-case. Its not obvious when functions return objects, or when that operate in place. I'll close, but I think this might trip others up.

[brycefrank]

You are correct! I did not see the subtlety. I am getting the same error now. I will take a look right now.

[bw4sz]

Thanks.

I think pc.clip definitely calls a new object, if you look at the length of the pandas frame before and after, the clip method has no effect on the initial object.

[brycefrank]

You are right, Ben. I have pushed an update to the laz_fix branch (seems to be a good place to stage all of your bug requests ;) ). Try it now and let me know what you think.

I am wondering if resetting the index may have implications later on, but I think it is fairly safe. When someone wants to clip a cloud, like yourself, the index of the original cloud is likely not of any concern, and the order of the points remains in tact as well. Let me know if you have any thoughts on this.

• Bryce

[brycefrank]

Should be fixed:

https://github.com/brycefrank/pyfor/pull/27