convert fails with numba error

JoaoGuibs commented 4 years ago

Hi, I tried running your conversion script in order to convert from a .las to a .pnts. I ran: py3dtiles convert 20170811bszn.las --out ./tmp/

But then I get an error from the Numba package which hints towards function variable type incongruency and the process stays stalled forever. The output was:

1.52 % in 0 sec [est. time left: 11 sec]OH NO. b'6'
Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (array(float32, 2d, C), array(float64, 1d, C))
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
In definition 2:
    All templates rejected with literals.
In definition 3:
    All templates rejected without literals.
In definition 4:
    All templates rejected with literals.
In definition 5:
    All templates rejected without literals.
In definition 6:
    All templates rejected with literals.
In definition 7:
    All templates rejected without literals.
In definition 8:
    All templates rejected with literals.
In definition 9:
    All templates rejected without literals.
In definition 10:
    All templates rejected with literals.
In definition 11:
    All templates rejected without literals.
In definition 12:
    TypeError: unsupported array index type array(float64, 1d, C) in [array(float64, 1d, C)]
    raised from /home/jgbs/.local/lib/python3.5/site-packages/numba/typing/arraydecl.py:71
In definition 13:
    TypeError: unsupported array index type array(float64, 1d, C) in [array(float64, 1d, C)]
    raised from /home/jgbs/.local/lib/python3.5/site-packages/numba/typing/arraydecl.py:71
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at /home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py (32)

File "../../../../.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py", line 32:
def _insert(cells_xyz, cells_rgb, aabmin, inv_aabb_size, cell_count, xyz, rgb, spacing, shift, force=False):
    <source elided>

        return xyz[notinserted], rgb[notinserted], needs_balance
        ^

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

Traceback (most recent call last):
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/task/node_process.py", line 157, in run
    result, data = _process(node, octree_metadata, name, filenames, queue, begin, log_file)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/task/node_process.py", line 109, in _process
    node.insert(node_catalog, octree_metadata.scale, data['xyz'], data['rgb'], halt_at_depth == 0)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/node.py", line 73, in insert
    self._split(node_catalog, scale)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/node.py", line 155, in _split
    self.insert(node_catalog, scale, xyz, rgb)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/node.py", line 80, in insert
    self.aabb[0], self.inv_aabb_size, xyz, rgb)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py", line 62, in insert
    int(self.cell_count[0] - 1).bit_length(), force)
  File "/home/jgbs/.local/lib/python3.5/site-packages/numba/dispatcher.py", line 401, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/home/jgbs/.local/lib/python3.5/site-packages/numba/dispatcher.py", line 344, in error_rewrite
    reraise(type(e), e, None)
  File "/home/jgbs/.local/lib/python3.5/site-packages/numba/six.py", line 668, in reraise
    raise value.with_traceback(tb)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (array(float32, 2d, C), array(float64, 1d, C))
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
In definition 2:
    All templates rejected with literals.
In definition 3:
    All templates rejected without literals.
In definition 4:
    All templates rejected with literals.
In definition 5:
    All templates rejected without literals.
In definition 6:
    All templates rejected with literals.
In definition 7:
    All templates rejected without literals.
In definition 8:
    All templates rejected with literals.
In definition 9:
    All templates rejected without literals.
In definition 10:
    All templates rejected with literals.
In definition 11:
    All templates rejected without literals.
In definition 12:
    TypeError: unsupported array index type array(float64, 1d, C) in [array(float64, 1d, C)]
    raised from /home/jgbs/.local/lib/python3.5/site-packages/numba/typing/arraydecl.py:71
In definition 13:
    TypeError: unsupported array index type array(float64, 1d, C) in [array(float64, 1d, C)]
    raised from /home/jgbs/.local/lib/python3.5/site-packages/numba/typing/arraydecl.py:71
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at /home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py (32)

File "../../../../.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py", line 32:
def _insert(cells_xyz, cells_rgb, aabmin, inv_aabb_size, cell_count, xyz, rgb, spacing, shift, force=False):
    <source elided>

        return xyz[notinserted], rgb[notinserted], needs_balance
        ^

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

OH NO. b'0'
Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (array(float32, 2d, C), array(float64, 1d, C))
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
In definition 2:
    All templates rejected with literals.
In definition 3:
    All templates rejected without literals.
In definition 4:
    All templates rejected with literals.
In definition 5:
    All templates rejected without literals.
In definition 6:
    All templates rejected with literals.
In definition 7:
    All templates rejected without literals.
In definition 8:
    All templates rejected with literals.
In definition 9:
    All templates rejected without literals.
In definition 10:
    All templates rejected with literals.
In definition 11:
    All templates rejected without literals.
In definition 12:
    TypeError: unsupported array index type array(float64, 1d, C) in [array(float64, 1d, C)]
    raised from /home/jgbs/.local/lib/python3.5/site-packages/numba/typing/arraydecl.py:71
In definition 13:
    TypeError: unsupported array index type array(float64, 1d, C) in [array(float64, 1d, C)]
    raised from /home/jgbs/.local/lib/python3.5/site-packages/numba/typing/arraydecl.py:71
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at /home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py (32)

File "../../../../.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py", line 32:
def _insert(cells_xyz, cells_rgb, aabmin, inv_aabb_size, cell_count, xyz, rgb, spacing, shift, force=False):
    <source elided>

        return xyz[notinserted], rgb[notinserted], needs_balance
        ^

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

Traceback (most recent call last):
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/task/node_process.py", line 157, in run
    result, data = _process(node, octree_metadata, name, filenames, queue, begin, log_file)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/task/node_process.py", line 109, in _process
    node.insert(node_catalog, octree_metadata.scale, data['xyz'], data['rgb'], halt_at_depth == 0)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/node.py", line 73, in insert
    self._split(node_catalog, scale)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/node.py", line 155, in _split
    self.insert(node_catalog, scale, xyz, rgb)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/node.py", line 80, in insert
    self.aabb[0], self.inv_aabb_size, xyz, rgb)
  File "/home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py", line 62, in insert
    int(self.cell_count[0] - 1).bit_length(), force)
  File "/home/jgbs/.local/lib/python3.5/site-packages/numba/dispatcher.py", line 401, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/home/jgbs/.local/lib/python3.5/site-packages/numba/dispatcher.py", line 344, in error_rewrite
    reraise(type(e), e, None)
  File "/home/jgbs/.local/lib/python3.5/site-packages/numba/six.py", line 668, in reraise
    raise value.with_traceback(tb)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (array(float32, 2d, C), array(float64, 1d, C))
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
In definition 2:
    All templates rejected with literals.
In definition 3:
    All templates rejected without literals.
In definition 4:
    All templates rejected with literals.
In definition 5:
    All templates rejected without literals.
In definition 6:
    All templates rejected with literals.
In definition 7:
    All templates rejected without literals.
In definition 8:
    All templates rejected with literals.
In definition 9:
    All templates rejected without literals.
In definition 10:
    All templates rejected with literals.
In definition 11:
    All templates rejected without literals.
In definition 12:
    TypeError: unsupported array index type array(float64, 1d, C) in [array(float64, 1d, C)]
    raised from /home/jgbs/.local/lib/python3.5/site-packages/numba/typing/arraydecl.py:71
In definition 13:
    TypeError: unsupported array index type array(float64, 1d, C) in [array(float64, 1d, C)]
    raised from /home/jgbs/.local/lib/python3.5/site-packages/numba/typing/arraydecl.py:71
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at /home/jgbs/.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py (32)

File "../../../../.local/lib/python3.5/site-packages/py3dtiles/points/points_grid.py", line 32:
def _insert(cells_xyz, cells_rgb, aabmin, inv_aabb_size, cell_count, xyz, rgb, spacing, shift, force=False):
    <source elided>

        return xyz[notinserted], rgb[notinserted], needs_balance
        ^

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

Could you shed any light on this error? Thank you.

autra commented 4 years ago

Hi @JoaoGuibs. Thanks for the report. Yes I've seen this already (but can't remember in which situatino). Can you give me your os / python version and the output of pip freeze please?

I guess I have some maintenance work to do to adapt to the new breaking change of numba.

Also we absolutely need to restrict the range of version of our dependencies or this is bound to happen again.

JoaoGuibs commented 4 years ago

Hi, thanks for the quick reply. My python version (I am running inside a conda environment) is 3.7.3, and the output of pip freeze is:

pip freeze
attrs==19.2.0
backcall==0.1.0
bleach==3.1.0
certifi==2019.9.11
cycler==0.10.0
Cython==0.29.14
decorator==4.4.0
defusedxml==0.6.0
dtcwt==0.12.0
entrypoints==0.3
importlib-metadata==0.23
ipykernel==5.1.2
ipython==7.8.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.15.1
Jinja2==2.10.3
jsonschema==3.1.1
jupyter-client==5.3.3
jupyter-core==4.5.0
kiwisolver==1.1.0
libLAS==1.8.1
llvmlite==0.31.0
MarkupSafe==1.1.1
matplotlib==3.1.1
mistune==0.8.4
more-itertools==7.2.0
nbconvert==5.6.0
nbformat==4.4.0
notebook==6.0.1
numba==0.47.0
numpy==1.17.2
olefile==0.46
open3d==0.8.0.0
pandocfilters==1.4.2
parso==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.2.0
prometheus-client==0.7.1
prompt-toolkit==2.0.10
psycopg2-binary==2.8.4
ptyprocess==0.6.0
py3dtiles==1.0.2
Pygments==2.4.2
pyparsing==2.4.2
pyproj==2.4.2.post1
pyrsistent==0.15.4
python-dateutil==2.8.0
pytz==2019.3
pyzmq==18.1.0
scipy==1.3.1
Send2Trash==1.5.0
six==1.12.0
terminado==0.8.2
testpath==0.4.2
tornado==6.0.3
traitlets==4.3.3
triangle==20190115.3
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==3.5.1
zipp==0.6.0

The numba version is the latest release version of the package.

autra commented 4 years ago

py3dtiles==1.0.2

I'm quite surprised by that: you are using py3dtiles convert, which is not in 1.0.2 and AFAIK not released on pypi yet. Are you trying to run py3dtiles master in a env containing py3dtiles 1.0.2? If that's the case maybe you can try to uninstall 1.0.2, update dependencies and see what happens?

Apart from that, I guess we should release a new version on pypi, that might help you.

JoaoGuibs commented 4 years ago

Hi, you are indeed right! I had installed py3dtiles through pip the first time and when I realised there was no convert script, I installed it again through pip install git+[git_url] so there was some kind of clash between versions. Now the conversion fully works! Thank you again for the quick reply!

Oslandia / py3dtiles

convert fails with numba error #65