Open danielrode opened 6 months ago
kylemann16
commented
6 months ago Hey @danielrode thanks for posting! I'm taking a look at this, but currently unable to replicate from the main branch of silvimetric and copying your environment process.
Would you mind providing environment versions with
pip list
which pdal
pdal --versionThank you!
danielrode
commented
6 months ago pip list:
Package Version
------------------ -----------
certifi 2024.2.2
click 8.1.7
cloudpickle 3.0.0
dask 2024.4.1
dask-expr 1.0.11
dill 0.3.8
distributed 2024.4.1
fsspec 2024.3.1
GDAL 3.8.4
importlib_metadata 7.1.0
Jinja2 3.1.3
locket 1.0.0
MarkupSafe 2.1.5
msgpack 1.0.8
numpy 1.26.4
packaging 24.0
pandas 2.2.1
partd 1.4.1
pdal 3.3.0
pip 24.0
psutil 5.9.8
pyarrow 15.0.2
pyproj 3.6.1
python-dateutil 2.9.0.post0
pytz 2024.1
PyYAML 6.0.1
scipy 1.13.0
setuptools 65.5.0
silvimetric 1.0.0
six 1.16.0
sortedcontainers 2.4.0
tblib 3.0.0
tiledb 0.27.1
toolz 0.12.1
tornado 6.4
tzdata 2024.1
urllib3 2.2.1
zict 3.0.0
zipp 3.18.1which pdal --> /usr/bin/pdal
pdal --version --> pdal 2.5.6 (git-version: Release)
danielrode
commented
6 months ago Update: I get the same No opening '(' RuntimeError when running shatter as well:
silvimetric --log-level DEBUG --database autzen-smdb.tdb shatter https://s3.amazonaws.com/hobu-lidar/autzen-classified.copc.laz --date 2001-01-01:
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,203 - distributed.worker - WARNING - Compute Failed
Key: ('filter-bag-from-delayed-f8b79650d1280add110b061dd0bb38fa', 0)
Function: execute_task
args: ((<function reify at 0x7f7c2fe35440>, (<function Extents.filter at 0x7f7c13917600>, ([635549.20,639029.20],[848856.21,853566.21]), <silvimetric.resources.data.Data object at 0x7f7c13937050>, 100, 600000, 6)))
kwargs: {}
Exception: 'RuntimeError("No opening \'(\'.")'
Traceback (most recent call last):
File "/home/daniel/bh/.venv/bin/silvimetric", line 8, in <module>
sys.exit(cli())
^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/cli/cli.py", line 161, in shatter_cmd
shatter.shatter(config)
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/commands/shatter.py", line 179, in shatter
leaves = extents.chunk(data, 100)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/resources/extents.py", line 72, in chunk
n = curr.compute()
^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/dask/base.py", line 375, in compute
(result,) = compute(self, traverse=False, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/dask/base.py", line 661, in compute
results = schedule(dsk, keys, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/resources/extents.py", line 106, in filter
pc = data.estimate_count(self.bounds)
^^^^^^^^^^^^^^^^^
File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/resources/data.py", line 153, in estimate_count
qi = pipeline.quickinfo[reader.type]
^^^^^^^^^^^^^^^^^
RuntimeError: No opening '('.The error you're seeing is a PDAL error message referring to the bounds created, and I don't think this exists in the latest version of PDAL.
I can't replicate your error, and I'm fairly certain it's because of the mixing of environments. The latest version of python-pdal that you have installed should be using pdal version 2.7.1 or newer, but this error is coming from the pdal installed in your /usr/bin.
Relevant issue in pdal python: https://github.com/PDAL/python/issues/164
If you have conda available to you, the easiest solution would be to use that to manage your environment with conda env create -f environment.yml && conda activate silvimetric from the base directory of the project. If you're stuck with source installs and pip, then I would update the pdal you have installed in /usr/bin and see if that fixes it.
This error can be replicated through pdal directly:
(a) ~/code/silvimetric % m list pdal
# packages in environment at /Users/kmann/mambaforge/envs/a:
#
# Name Version Build Channel
pdal 2.5.6 hb8312d3_4 conda-forge
(a) ~/code/silvimetric % pdal info tests/data/test_data.copc.laz --readers.copc.bounds '[495579, 4780091, 695301, 4978956]'
PDAL: No opening '('.
(a) ~/code/silvimetric % m upgrade pdal
...
(a) ~/code/silvimetric % m list pdal
# packages in environment at /Users/kmann/mambaforge/envs/a:
#
# Name Version Build Channel
pdal 2.7.1 hff7fe72_4 conda-forge
(a) ~/code/silvimetric % pdal info tests/data/test_data.copc.laz --readers.copc.bounds '[495579, 4780091, 695301, 4978956]'
{
"file_size": 399984,
"filename": "tests/data/test_data.copc.laz",
"now": "2024-04-11T10:19:45-0500",
"pdal_version": "2.7.1 (git-version: Release)",
"reader": "readers.copc",I tried the Conda approach shown in the SilviMetric QuickStart doc. However, that resulted in a libglog.so.2: cannot open shared object file error on my machine. I may open a separate issue about that later.
I got things working using just the native OpenSUSE package manager and venv/pip. Looks like older versions of PDAL expect the ([minx, maxx], [miny, maxy]) bounds format, but SilviMetric uses [minx, miny, maxx, maxy] format internally when passing the bounds object to PDAL. I patched that so the code now works with my older version of PDAL (which is currently the latest version of PDAL available in OpenSUSE repos). Here's the modification I made: https://github.com/hobuinc/silvimetric/commit/1b6e973648a5187308d2515c503c452c1b10b96f
If you are interested in adding this to the main repo, let me know and I can make a PR. Otherwise, feel free to close this issue.
Specifying a minimum expected PDAL version number in the guide/docs is an alternative solution, but I don't know when PDAL switched from expecting that specific bounds format to the several currently supported in the latest version.
I have successfully initialized a new TileDB database and am now trying to move onto the next step: scanning. Python throws a RuntimeError when I try to use the
scanfunction of SilviMetric. This happened with my own data, but I can reproduce the error with test data from the SilviMetric GitHub repository as well.System: OpenSUSE Tumbleweed (Linux) Python version: 3.11
Here is my workflow (in Bash):
When I run that last command, I get the error: