Open odidev opened 4 years ago
Thanks for the report, this is related to the discussion here:
A first step would be to try to build and test VTK on aarch64. Did you have any success doing so ?
@jcfr , I have build the VTK wheel on aarch64 architecture but not sure how to test it. Can you please help me here?
@jcfr is there anything else required from my side.
@odidev if you built the wheel, you can just run python -m pip install /path/to/vtk-xyz.whl
to install it.
@martinken @tjcorona @mathstuf
I doubt we're going to be able to provide aarch64 wheels as long as we lack hardware for testing and building them regularly.
I have build the VTK wheel on aarch64 architecture but not sure how to test it. Can you please help me here?
is there anything else required from my side.
@odidev This is exciting and thanks for sharing updates and thanks for your patience :pray:
As @mathstuf mentioned, we do not have the infrastructure to test here but we shouldn't let that stop us from having wheels for aarch64. I am sure we could find community members willing to help support this.
Now, would you like to document the steps used to build the VTK wheel ?
Also, since the process to build the wheel has been completely revamped (see here), would be great to try these.
@jcfr, I explored the link (https://gitlab.kitware.com/vtk/vtk/-/blob/master/Documentation/dev/build.md#python-wheels) suggested by you. Please find below steps to build VTK aarch64 wheel.
apt install -y build-essential mesa-common-dev mesa-utils freeglut3-dev ninja-build git cmake
git clone --recursive https://gitlab.kitware.com/vtk/vtk.git
mkdir build
cd build
cmake -GNinja -DVTK_WHEEL_BUILD=ON -DVTK_WRAP_PYTHON=ON ../vtk/
ninja
python setup.py bdist_wheel
root@b605f0ff0d7c:~# pip install /build/dist/vtk-9.0.0-cp37-cp37m-linux_aarch64.whl
Processing /build/dist/vtk-9.0.0-cp37-cp37m-linux_aarch64.whl
Installing collected packages: vtk
Successfully installed vtk-9.0.0
root@b605f0ff0d7c:~# python
Python 3.7.7 (default, Apr 21 2020, 08:59:39)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
>>> print(vtk.vtkVersion.GetVTKSourceVersion())
vtk version 9.0.0
>>>
@jcfr, can you please let me know if I can help with anything else.
We don't do aarch64 testing of the codebase, so without that, I'm hesitant to make official releases of builds. Is there some way to mark uploads to PyPI as experimental or the like (not per-version, but per artifact)?
@mathstuf - thanks for this. I work with @odidev on the AArch64 enablement. I think your asking about marking the aarch64 wheel as experimental in PyPI - something that I don't believe is supported.
We may be able to help by adding some automatic testing. As a simple end-to-end case, I guess running something like https://vtk.org/Wiki/VTK/Examples/Python/Cylinder with xvfb and comparing the result to a known good rendering would be a start. Would that provide sufficient confidence to move forward with an AArch64 wheel?
I'd like a full test suite run on aarch64 (with Python enaled). This would be a normal (non-wheel) build of VTK since our wheel build configuration doesn't work with the test suite (due to the library relocations). You should be able to submit a run to our CDash with ctest -D Experimental
once VTK is configured. You can also run the steps individually, but the CTest docs are better for that. A CTest script may also be more reproducible if it isn't just a simple run.
Thanks for this @mathstuf, I'll work with @odidev to look into this.
Hi @mathstuf, I tried running tests but it looks like the tests are not getting picked up. Please find below steps which I followed to run the tests. Also, can you please point it out if I am missing anything here?
apt install -y build-essential mesa-common-dev mesa-utils freeglut3-dev ninja-build git cmake
git clone --recursive https://gitlab.kitware.com/vtk/vtk.git
mkdir build
cd build
cmake -GNinja -DVTK_WRAP_PYTHON=ON ../vtk/ # Non-wheel, python enabled build
ninja
root@fc2408cb3eaa:~/build# ctest -D Experimental
Site: fc2408cb3eaa
Build name: Linux-c++
Create new tag: 20200609-1139 - Experimental
Configure project
Each . represents 1024 bytes of output
. Size of output: 0K
Build project
Each symbol represents 1024 bytes of output.
'!' represents an error and '*' a warning.
. Size of output: 0K
0 Compiler errors
0 Compiler warnings
Test project /root/build
No tests were found!!!
Performing coverage
Cannot find any coverage files. Ignoring Coverage request.
Submit files
SubmitURL: http://open.cdash.org/submit.php?project=VTK
Uploaded: /root/build/Testing/20200609-1139/Configure.xml
Uploaded: /root/build/Testing/20200609-1139/Build.xml
Uploaded: /root/build/Testing/20200609-1139/Test.xml
Uploaded: /root/build/Testing/20200609-1139/Done.xml
Submission successful
root@fc2408cb3eaa:~/build#
The tests are off by default. Should be VTK_BUILD_TESTING=WANT
.
OK. Let me try that. Thanks for the quick response.
@mathstuf , I tried configuring VTK by "cmake -GNinja VTK_BUILD_TESTING=WANT -DVTK_WRAP_PYTHON=ON ../vtk/" command but even this is not helping me. Please find below logs-
root@fc2408cb3eaa:~/build# ctest -D Experimental
Site: fc2408cb3eaa
Build name: Linux-c++
Create new tag: 20200609-1320 - Experimental
Configure project
Each . represents 1024 bytes of output
. Size of output: 0K
Build project
Each symbol represents 1024 bytes of output.
'!' represents an error and '*' a warning.
... Size of output: 3K
0 Compiler errors
0 Compiler warnings
Test project /root/build
No tests were found!!!
Performing coverage
Cannot find any coverage files. Ignoring Coverage request.
Submit files
SubmitURL: http://open.cdash.org/submit.php?project=VTK
Uploaded: /root/build/Testing/20200609-1320/Configure.xml
Uploaded: /root/build/Testing/20200609-1320/Build.xml
Uploaded: /root/build/Testing/20200609-1320/Test.xml
Uploaded: /root/build/Testing/20200609-1320/Done.xml
Submission successful
root@fc2408cb3eaa:~/build#
@mathstuf, can you please suggest what needs to be done here?
For reference, build options for VTK are documented here: https://github.com/Kitware/VTK/blob/master/Documentation/dev/build.md#build-settings
Instead of running ctest -D Experimental
, I suggest you try to directly use ctest
to run the test. What do you get doing the following:
git clone --recursive https://gitlab.kitware.com/vtk/vtk.git
mkdir build
cd build
cmake -GNinja -DVTK_WRAP_PYTHON=ON -DVTK_BUILD_TESTING=WANT ../vtk/
ninja
ctest -N
Running ctest -N
should list all tests that have been enabled. See https://cmake.org/cmake/help/latest/manual/ctest.1.html#options
cmake -GNinja VTK_BUILD_TESTING=WANT
It looks like you missed -D
on the testing argument.
@mathstuf Thanks for your support. I am able to run the test cases on x86 machine but few test cases are failing. I investigated about the possible reasons for the failures and I think that they are failing because of some system dependency. Could you please look into the logs and point the possible reasons for the failed test cases. Attaching the test logs for your reference. VTK_Text_logs.txt
(I'm working with @odidev). @odidev - I guess the good news is that all the failures look the same: 'numerical' :)
Can you work on reproducing a single test with a complete example of how to reproduce the single test failure. Also, please can you output more verbose logging information as this may reveal the source of the problem.
I have attached the test file TestConeLayoutStrategy.txt which looks to be a minimal program to reproduce the issue.
I have ran example program to display the cylinder on both aarch64 and x86_64 architecture. here is the test program: test.txt
Please see blow outputs on both the architectures
On aarch64
On x86_64
@mathstuf, I ran single test case with debug logs enabled using command ctest -R TestGPURayCastSlicePlane --debug 2>&1 | tee debug-log-aarch64.txt
. Attached the debug log file here for your reference. Could you please help me debugging this test failure? Thanks in advance.
I have no idea how to figure out rendering/OpenGL failures, sorry. @martinken?
@mathstuf
Out of many issues in VTK testing on Linux/ARM64 machines, the most prominent issue seems to be the import error, while importing the module: ‘vtkCommonCore’. Please find the error logs below:
/home/ubuntu/cmake-3.20.0-rc3/Source/CTest/cmCTestRunTest.cxx:41 24: File "/home/ubuntu/01_VTK_Project/build/lib/python3.8/site-packages/vtkmodules/__init__.py", line 53, in <module>
/home/ubuntu/cmake-3.20.0-rc3/Source/CTest/cmCTestRunTest.cxx:41 24: from . import vtkCommonCore
/home/ubuntu/cmake-3.20.0-rc3/Source/CTest/cmCTestRunTest.cxx:41 24: ImportError: /home/ubuntu/01_VTK_Project/build/lib/python3.8/site-packages/vtkmodules/vtkCommonCore.cpython-38-aarch64-linux-gnu.so: undefined symbol: _Py_NotImplementedStruct
/home/ubuntu/cmake-3.20.0-rc3/Source/CTest/cmCTestRunTest.cxx:41 24: During handling of the above exception, another exception occurred:
/home/ubuntu/cmake-3.20.0-rc3/Source/CTest/cmCTestRunTest.cxx:41 24: File "/home/ubuntu/01_VTK_Project/vtk/Utilities/vtkTclTest2Py/rtImageTest.py", line 12, in <module>
/home/ubuntu/cmake-3.20.0-rc3/Source/CTest/cmCTestRunTest.cxx:41 24: import vtk
Above logs are the result of following test command:
ctest -R PythonContext2DPython-testPythonItem --debug
The above error logs are the same in all the test cases, which shows the import error.
According to the logs, ‘vtkCommonCore’ is not getting imported, which results in an undefined symbol error for ‘_Py_NotImplementedStruct’.
I have tried following resolutions to fix this error:
import os
modules_rough = "/home/ubuntu/01_VTK_Project/build/lib/python3.8/site-packages/vtkmodules"
os.environ['LD_LIBRARY_PATH'] = modules_rough
try:
from . import vtkCommonCore
except ImportError:
import _vtkmodules_static
But the error remains the same.
Can you please provide me with some pointers to this import issue during VTK testing on Linux/ARM64 machines? Once this error gets resolved, I will look into the rendering issue in other test cases.
That's odd. Can you try with -DVTK_PYTHON_OPTIONAL_LINK=OFF
?
Though it is used in the generated code and is in the Python headers. Is it not in libpython
?
@mathstuf
Thank you for the suggestion. However, using -DVTK_PYTHON_OPTIONAL_LINK=OFF
flag along with cmake brought dangerous relocation: unsupported relocation
error during ninja command.
I suspected that the python import issue must be hitting because of some varied system configuration. So, I moved to another ARM64 server, and built and tested VTK. The results are much better than before, and there are no python import issues.
95% tests are passing now, only 112 tests out of 2156 tests are failing.
glBlitFramebuffer
operation. Below are the exact error logs picked while running single test in debug mode:/io/src/Source/CTest/cmCTestRunTest.cxx:41 706: ( 30.212s) [main thread ]vtkOpenGLFramebufferObj:1390 WARN| failed at glBlitFramebuffer 1 OpenGL errors detected
/io/src/Source/CTest/cmCTestRunTest.cxx:41 706: 0 : (1282) Invalid operation
…
…
…
Failed Image Test ( TestSeedWidget2.png ) : 487.931
This error comes from this line in the VTK project.
According to the documentation here, it seems that the glBlitFramebuffer
function copies the block of pixels from one texture buffer object to another. However, I don’t have a Graphics card/GPU attached to my server machine. I was suspecting that the test fails to copy the pixels from objects, as there is no GPU available.
I again ran the test suite on a AMD64 server machine(accessed via mobxterm), and found that out of above 92 test cases failing in ARM64, only 50 tests showed the same issue (mentioned above) on AMD64 server machine. And on a local AMD64 Desktop machine, just 16 out of the above 92 test cases failed. Hence, I am doubtful if this issue is related to GPU, or there are some other complications involved. Also, there are certain test cases that are only failing on x64 server machine with the above error, but passing on ARM64 server machine.
/io/src/Source/CTest/cmCTestRunTest.cxx:41 705: ( 12.898s) [main thread ] vtkTextureObject.cxx:1025 ERR| vtkTextureObject (0xaaab19090980): Attempt to use a texture buffer exceeding your hardware's limits. This can happen when trying to color by cell data with a large dataset. Hardware limit is 65536 values while 146404 was requested.
The error logs come from this line in VTK project.
vtkglGetIntegerv(GL_MAX_TEXTURE_BUFFER_SIZE, &maxSize)
function is used to fetch the maximum texture buffer size and place it in maxSize
variable. As I do not have a GPU, I am suspecting that 65536 is taken as a default buffer size, and is compared to the requested numValues
variable. As 65536 is smaller than the requested value, the check fails and hence the test fails.
11 tests are failing on ARM64 server machine due to this issue. However these tests are passing on the local AMD64 desktop machine.
As the test results are varied on different machines, can you please provide some pointers and check if these issues are related to unavailability of GPU or not? Are there any possibilities to pass these tests alternatively? This can help me progress in this activity.
Also, may I please know if any of the above test failures can be ignored or not?
Please let me know if you need any further information from me.
@martinken Any thoughts on the OpenGL failures/differences here?
The texture buffer limit is a mesa issue and we have a MR in to fix it so fee free to ignore those. Haven't had a chance to look at the others.
@martinken @mathstuf
I have analysed that some tests are written for the x86 architecture, and out of them, few are not correct for the ARM64 architecture. This is because the C/Cpp compilers treat CHAR differently for AMD64 and ARM64 machines.
For example: Let’s run the test CommonCoreCxx-UnitTestMath
On ARM64 machines, the C/Cpp compiler reads the CHAR’s range from 0 to 255. On the other hand, AMD64 C/Cpp compilers treat CHAR from -128 to +127. Hence, running above test case gives the below error on ARM64 machine:
Bad fitting range for VTK_UNSIGNED_CHAR
This is because, VTK_CHAR_MIN and VTK_UNSIGNED_CHAR_MIN are both assigned the value ‘0’, and VTK_CHAR_MAX and VTK_UNSIGNED_CHAR_MAX have been assigned the value 255, for ARM64 machines. This creates data mismatch in the array check defined in the function definition of “vtkMath::GetScalarTypeFittingRange()” function here : https://github.com/Kitware/VTK/blob/master/Common/Core/vtkMath.cxx#L2842.
Both CHAR and UNSIGNED CHAR are treated the same way. Thus, for the ‘unsigned char’ test case here: https://github.com/Kitware/VTK/blob/master/Common/Core/Testing/Cxx/UnitTestMath.cxx#L3282, the macro ‘VTK_UNSIGNED_CHAR’ (value hardcoded to 3), is compared to the function returning value 2 (defined for the range of CHAR). This fails the test case for ARM64.
This test case would have been failing for AMD64 as well, if there were a case of SIGNED_CHAR used along with CHAR, as AMD64 C/Cpp compilers treat both CHAR and SIGNED CHAR from -128 to +127. Hence, the test would have compared VTK_SIGNED_CHAR (value hardcoded to 15) with function returning 2 (defined for the range of CHAR).
Can you please suggest if we can do something for such test cases?
Can you please suggest if we can do something for such test cases?
I have no idea how prevalent the char == unsigned char
assumption is. There does seem to be support for signed char
(VTK_CHAR_IS_SIGNED
) which is derived from KWIML's KWIML_ABI_CHAR_IS_SIGNED
. Is this information (in vtkType.h
) correct at least in an aarch64 build? If so, it seems to be an issue with vtkMath
and/or its test suite.
Interestingly, the Python vtkConstants
has VTK_CHAR_MIN
as signed (unconditionally) and has been since 2002. I don't know what to do here though.
I have build the VTK wheel on aarch64 architecture but not sure how to test it. Can you please help me here?
is there anything else required from my side.
@odidev This is exciting and thanks for sharing updates and thanks for your patience pray
As @mathstuf mentioned, we do not have the infrastructure to test here but we shouldn't let that stop us from having wheels for aarch64. I am sure we could find community members willing to help support this.
Now, would you like to document the steps used to build the VTK wheel ?
Also, since the process to build the wheel has been completely revamped (see here), would be great to try these.
I'm (trying) to build vtk python modules on an arm64 system as well. I'm using the instructions you referenced, here: https://gitlab.kitware.com/vtk/vtk/-/blob/master/Documentation/dev/build.md#python-wheels
Everything runs fine, and if I am inside the build directory, and run python followed by a VTK import, it works well.
But the python3 setup.py bdist_wheel command doesn't exist. The VTK directory itself is ~4GB, so I'd like to somehow get the vtk python module into a portable format I can distribute to another identical system. My understanding was that building a wheel was the best way to do this, but in the absence of the bdist_wheel command, what would you recommend doing?
Standard commands:
build build everything needed to install
build_py "build" pure Python modules (copy to build directory)
build_ext build C/C++ and Cython extensions (compile/link to build directory)
build_clib build C/C++ libraries used by Python extensions
build_scripts "build" scripts (copy and fixup #! line)
clean clean up temporary files from 'build' command
install install everything from build directory
install_lib install all Python modules (extensions and pure Python)
install_headers install C/C++ header files
install_scripts install scripts (Python or otherwise)
install_data install data files
sdist create a source distribution (tarball, zip file, etc.)
register register the distribution with the Python package index
bdist create a built (binary) distribution
bdist_dumb create a "dumb" built distribution
bdist_rpm create an RPM distribution
bdist_wininst create an executable installer for MS Windows
check perform some checks on the package
upload upload binary package to PyPI
Do you have the wheel
package installed in the right environment?
I did not, thanks! Everything is working now.
Interestingly, even once it is installed, it's not shown in the help files. I guess to more software-oriented people it's obvious, but for me, without wheel installed the command failed like so:
ubuntu@server:/data/vtk2/build$ python setup.py bdist_wheel
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'bdist_wheel'
So I then checked the help for setup.py, which showed no such "bdist_wheel" option. The combination of "invalid command" and it not existing in the help sort of leads the user away from concluding that the issue is a missing package rather than an issue in setup.py itself. Replacing invalid command with "wheel module missing" (or mentioning wheel must be installed in the guide) might clear this up?
ubuntu@server:/data/vtk2/build$ python -m wheel version
wheel 0.37.0
ubuntu@server:/data/vtk2/build$ python setup.py --help-commands
Standard commands:
build build everything needed to install
build_py "build" pure Python modules (copy to build directory)
build_ext build C/C++ extensions (compile/link to build directory)
build_clib build C/C++ libraries used by Python extensions
build_scripts "build" scripts (copy and fixup #! line)
clean clean up temporary files from 'build' command
install install everything from build directory
install_lib install all Python modules (extensions and pure Python)
install_headers install C/C++ header files
install_scripts install scripts (Python or otherwise)
install_data install data files
sdist create a source distribution (tarball, zip file, etc.)
register register the distribution with the Python package index
bdist create a built (binary) distribution
bdist_dumb create a "dumb" built distribution
bdist_rpm create an RPM distribution
bdist_wininst create an executable installer for MS Windows
check perform some checks on the package
upload upload binary package to PyPI
The combination of "invalid command" and it not existing in the help sort of leads the user away from concluding that the issue is a missing package rather than an issue in setup.py itself. Replacing invalid command with "wheel module missing" (or mentioning wheel must be installed in the guide) might clear this up?
Welcome to my world of "list of problems with distutils / setuptools" :) . I can add a note to the VTK docs, but for --help-commands
, upstream would need to resolve commands available through whatever mechanism wheel
makes its stuff available in the first place.
I added a binary wheel for python3.10 here: https://github.com/finsberg/vtk-aarch64/releases/tag/vtk-9.2.6-cp310, which you can install with
python3 -m pip install "https://github.com/finsberg/vtk-aarch64/releases/download/vtk-9.2.6-cp310/vtk-9.2.6.dev0-cp310-cp310-linux_aarch64.whl"
Thanks a lot @finsberg for providing a whl file. I was able to build VTK python wheel following the instructions provided in the thread, however I'm getting much bigger .so files (see screenshot, left is my whl and right is yours).
I'm curious on how you got something way lighter: did you compile with different instructions? (In terms of installation and execution in Python, both whl files seem to be working fine)
That looks like a Debug
build rather than Release
.
@thewtex I think I just followed the recipe here: https://docs.vtk.org/en/latest/advanced/build_python_wheels.html and I belive I also set CMAKE_BUILD_TYPE
to RelWithDebInfo
. You can also find more info her: https://github.com/Kitware/VTK/blob/master/.gitlab/os-linux.yml
Ah, the debug info could definitely be a problem here. Do things look better if you run strip
on the libraries?
Awesome, thanks a lot @mathstuf and @finsberg , indeed it was the debug info (RelWithDebInfo
) that was making the libs quite big => using the Release
without debug info was the right instruction:
cmake -GNinja -DVTK_WHEEL_BUILD=ON -DVTK_WRAP_PYTHON=ON -DCMAKE_BUILD_TYPE=Release ../vtk/
Thanks again and happy new year!
New wheel for python3.12 is available here: https://github.com/finsberg/vtk-aarch64/releases/tag/vtk-9.3.0-cp312
VTK installation using pip on aarch64 machine fails with below issue: