Closed TylerGubala closed 3 years ago
I've tried to build it on Arch Linux but without success so far. Log attched log.txt
The link to build instructions does not work, btw.
Just tried manual build steps on Ubuntu server Linux, got the following result:
I am building now on Ubuntu Server with the script shown here
Support for Ubuntu wheels are in the works.
under debian 10 I have to add -DWITH_MEM_JEMALLOC=OFF
to the cmake command on line 15 of your script.
.blenderpy/v2.80/build_linux_bpy_custom_test/venv/bin/python3.7 -c "import bpy; bpy.ops.render.render(write_still=True)"
Color management: using fallback mode for management
Color management: Error could not find role data role.
Color management: scene view "Filmic" not found, setting default "Standard".
[1] 19185 segmentation fault .blenderpy/v2.80/build_linux_bpy_custom_test/venv/bin/python3.7 -c
I am not worried about the segfault as this is "normal" on blender exit on my system (regardless of linux distro/version)
otherwise the importing bpy will fail with
.blenderpy/v2.80/build_linux_bpy_custom_test/venv/bin/python3.7 -c "import bpy; bpy.ops.render.render(write_still=True)"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: /lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block
hope this helps, even as a work around, with making the script compatible with more systems and an automated installation.
@Kuro-Maii Do you know a good way to detect linux distributions?
What does
distro.linux_distribution(full_distribution_name=False)
report for you?
(Note, requires installing python distro module)
Is the CMake option `-DWITH_PYTHON' necessary?
Just tried manual build steps on Ubuntu server Linux, got the following result:
/usr/bin/ld.gold: error: /home/tylergubala/.blenderpy/master/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libx264.a(quant-a-10.o): requires dynamic R_X86_64_PC32 reloc against 'x264_decimate_table4' which may overflow at runtime; recompile with -fPIC /usr/bin/ld.gold: error: /home/tylergubala/.blenderpy/master/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libx264.a(trellis-64-8.o): requires dynamic R_X86_64_PC32 reloc against 'x264_cabac_entropy' which may overflow at runtime; recompile with -fPIC /usr/bin/ld.gold: error: /home/tylergubala/.blenderpy/master/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libx264.a(trellis-64-10.o): requires dynamic R_X86_64_PC32 reloc against 'x264_cabac_entropy' which may overflow at runtime; recompile with -fPIC /usr/bin/ld.gold: error: /home/tylergubala/.blenderpy/master/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libxvidcore.a(qpel_mmx.o): requires dynamic R_X86_64_PC32 reloc against 'xvid_FIR_14_3_2_1' which may overflow at runtime; recompile with -fPIC collect2: error: ld returned 1 exit status source/creator/CMakeFiles/blender.dir/build.make:682: recipe for target 'bin/bpy.so' failed make[2]: *** [bin/bpy.so] Error 1 CMakeFiles/Makefile2:7825: recipe for target 'source/creator/CMakeFiles/blender.dir/all' failed make[1]: *** [source/creator/CMakeFiles/blender.dir/all] Error 2 Makefile:162: recipe for target 'all' failed make: *** [all] Error 2 tylergubala@ubuntubox:~/.blenderpy/master/build_linux_bpy_custom$
I am also failing with this error. All other steps look really great and thanks by the way. How did you end up resolving these ld.gold errors? @TylerGubala
@zehao-sean-huang you should check out this Blender dev talk post that's related but essentially the ld.gold linker errors went completely away by performing install_deps.sh
If you are using my script in the wiki exactly, especially on Ubuntu 18.4 LTS then you should try building both master and the 2.80 branches.
I can check when I get home but 2.80 was the latest that would build on my system when I last tried.
@zehao-sean-huang the ld.gold errors usually result from cloning from SVN in my experience. The updated setup.py script should avoid errors with this.
Wheels for Linux will come out once pypi updates the file size limit for bpy
, bpy-cuda
and bpy-optix
Building on Debian should be better for you now @Kuro-Maii ; if you get around to it try building from the repo with the prerequisites installed.
Sorry for the very late reply, I was severely distracted with a few other projects...
@Kuro-Maii Do you know a good way to detect linux distributions?
What does
distro.linux_distribution(full_distribution_name=False)
report for you?
(Note, requires installing python distro module)
Python 3.7.3 (default, Dec 20 2019, 18:57:59)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distro
>>> distro.linux_distribution(full_distribution_name=False)
('debian', '10', 'buster')
>>>
I am reading thru the other comments and going to test the the rest.
I tried to run the command pip3 install bpy && bpy_post_install
at first and got the following output that resulted in a fail:
CMake Error at CMakeLists.txt:834 (message):
Cycles OSL requires WITH_LLVM, the library may not have been found.
Configure LLVM or disable WITH_CYCLES_OSL
installing with the script found here I am coming to the same conclusion as my earlier comment from december
this is regardless of the flags: --precompiled-libs
, --compile-libs
or neither
I only rested this with ./build_bpy_new.sh v2.80 python3.7
using the master
branch fails in the same way as with installing from pip3
as listed above
I just found an interesting error after running the script again for v2.80
after having it done for master
...
as the test then fails with
-- Installing: /home/angel/.blenderpy/v2.80/build_linux_bpy_custom/bin/2.80/datafiles/locale/zh_TW/LC_MESSAGES/blender.mo
mkdir: cannot create directory ‘../build_linux_bpy_custom_test’: File exists
--running test--
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: libIlmImf-2_4.so.24: cannot open shared object file: No such file or directory
doing a find reuslts in the following:
find / -iname "libIlmImf*"
/usr/lib/x86_64-linux-gnu/libIlmImf.a
/usr/lib/x86_64-linux-gnu/libIlmImfUtil.a
/usr/lib/x86_64-linux-gnu/libIlmImf-2_2.so.23
/usr/lib/x86_64-linux-gnu/libIlmImf.so
/usr/lib/x86_64-linux-gnu/libIlmImfUtil.so
/usr/lib/x86_64-linux-gnu/libIlmImfUtil-2_2.so.23.0.0
/usr/lib/x86_64-linux-gnu/libIlmImfUtil-2_2.so.23
/usr/lib/x86_64-linux-gnu/libIlmImf-2_2.so.23.0.0
/home/angel/.blenderpy/v2.80/blender/linux_centos7_x86_64/openexr/lib/libIlmImfUtil.a
/home/angel/.blenderpy/v2.80/blender/linux_centos7_x86_64/openexr/lib/libIlmImf.a
this was not a problem before master was used as the branch to compile
Hi @Kuro-Maii ,
I apologize for the confusion, but I believe that this is not necessarily an issue with the master
branch.
I think that if you
rm -Rf /home/angel/.blenderpy
You will be able to build using the build script again.
What I believe happened was that you built from the build script, and the .blenderpy
folder was created, and the script attempted to build Blender as a Python module (note that script you linked is for linux Ubuntu, not Debian)
Then, if you built from master
(using python setup.py sdist bdist_wheel
or something similar)... how did that go for you by the way? I added to the build script in bpy-build:master
a check to disable JEMALLOC
for Debian and Raspbian, which I can do more easily in Python. I am not experienced enough with bash to know how to do that in bash.
After you built the first time with the build bash script it created the files and folders; I think that is what is preventing you from building a second time. I have gotten similar errors when trying to build twice in a row from the bash script so I would suggest you perform the following step:
rm -Rf /home/angel/.blenderpy
Hi, I am tempting an install on Ubuntu. Sorry if the question seems silly but why bpy==1.2.3
in requirements.txt won't work? Only valid version seems to be "0.0.0a0". The requirements.txt
has been provided to me by devs working on Windows.
Edit: also pip install bpy --no-binary
asks me a param for --no-binary, so I guess the correct command is pip install bpy --no-binary future_fstrings
.
Script seems to take forever to run for me without much logs.
Edit 2: to get up to date cmake on Ubuntu:
sudo apt remove --purge --auto-remove cmake # remove preinstalled version, will not be high enough (>= 3.13.5)
sudo snap install cmake --classic
Edit 3: what is the state of the limit issue? https://github.com/pypa/pypi-support/issues/362 Is there currently a way to get a simpler install?
Edit 4: I haven't managed to build at this point, having a message:
If you're using CMake add this to your configuration flags:
-D WITH_CODEC_SNDFILE=ON
-D PYTHON_VERSION=3.7
-D WITH_OPENCOLORIO=ON
-D OPENCOLORIO_ROOT_DIR=/opt/lib/ocio
-D WITH_OPENIMAGEIO=ON
-D OPENIMAGEIO_ROOT_DIR=/opt/lib/oiio
-D WITH_CYCLES_OSL=ON
-D WITH_LLVM=ON
-D LLVM_VERSION=6.0
-D OSL_ROOT_DIR=/opt/lib/osl
-D WITH_OPENSUBDIV=ON
-D OPENSUBDIV_ROOT_DIR=/opt/lib/osd
-D WITH_OPENVDB=ON
-D WITH_OPENVDB_BLOSC=ON
-D OPENVDB_ROOT_DIR=/opt/lib/openvdb
-D BLOSC_ROOT_DIR=/opt/lib/blosc
-D WITH_ALEMBIC=ON
-D ALEMBIC_ROOT_DIR=/opt/lib/alembic
-D WITH_CODEC_FFMPEG=ON
-D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;x264;openjp2'
Or even simpler, just run (in your blender-source dir):
make -j6 BUILD_CMAKE_ARGS="-U *SNDFILE* -U *PYTHON* -U *BOOST* -U *Boost* -U *OPENCOLORIO* -U *OPENEXR* -U *OPENIMAGEIO* -U *LLVM* -U *CYCLES* -U *OPENSUBDIV* -U *OPENVDB* -U *COLLADA* -U *FFMPEG* -U *ALEMBIC* -D WITH_CODEC_SNDFILE=ON -D PYTHON_VERSION=3.7 -D WITH_OPENCOLORIO=ON -D OPENCOLORIO_ROOT_DIR=/opt/lib/ocio -D WITH_OPENIMAGEIO=ON -D OPENIMAGEIO_ROOT_DIR=/opt/lib/oiio -D WITH_CYCLES_OSL=ON -D WITH_LLVM=ON -D LLVM_VERSION=6.0 -D OSL_ROOT_DIR=/opt/lib/osl -D WITH_OPENSUBDIV=ON -D OPENSUBDIV_ROOT_DIR=/opt/lib/osd -D WITH_OPENVDB=ON -D WITH_OPENVDB_BLOSC=ON -D OPENVDB_ROOT_DIR=/opt/lib/openvdb -D BLOSC_ROOT_DIR=/opt/lib/blosc -D WITH_ALEMBIC=ON -D ALEMBIC_ROOT_DIR=/opt/lib/alembic -D WITH_CODEC_FFMPEG=ON -D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;x264;openjp2'"
Or in all your build directories:
cmake -U *SNDFILE* -U *PYTHON* -U *BOOST* -U *Boost* -U *OPENCOLORIO* -U *OPENEXR* -U *OPENIMAGEIO* -U *LLVM* -U *CYCLES* -U *OPENSUBDIV* -U *OPENVDB* -U *COLLADA* -U *FFMPEG* -U *ALEMBIC* -D WITH_CODEC_SNDFILE=ON -D PYTHON_VERSION=3.7 -D WITH_OPENCOLORIO=ON -D OPENCOLORIO_ROOT_DIR=/opt/lib/ocio -D WITH_OPENIMAGEIO=ON -D OPENIMAGEIO_ROOT_DIR=/opt/lib/oiio -D WITH_CYCLES_OSL=ON -D WITH_LLVM=ON -D LLVM_VERSION=6.0 -D OSL_ROOT_DIR=/opt/lib/osl -D WITH_OPENSUBDIV=ON -D OPENSUBDIV_ROOT_DIR=/opt/lib/osd -D WITH_OPENVDB=ON -D WITH_OPENVDB_BLOSC=ON -D OPENVDB_ROOT_DIR=/opt/lib/openvdb -D BLOSC_ROOT_DIR=/opt/lib/blosc -D WITH_ALEMBIC=ON -D ALEMBIC_ROOT_DIR=/opt/lib/alembic -D WITH_CODEC_FFMPEG=ON -D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;x264;openjp2' .
This information has been written to /tmp/pip-install-s0rqjx0w/bpy/BUILD_NOTES.txt
cmake -DWITH_PLAYER=OFF -DWITH_PYTHON_INSTALL=OFF -DWITH_PYTHON_MODULE=ON -DPYTHON_VERSION=3.7 -DWITH_AUDASPACE=OFF -S/tmp/pip-install-s0rqjx0w/bpy/build/temp.linux-x86_64-3.7/blender -B/tmp/pip-install-s0rqjx0w/bpy/build/temp.linux-x86_64-3.7/build
CMake Error: The source directory "/tmp/pip-install-s0rqjx0w/bpy" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
error: command 'cmake' failed with exit status 1
(but also maybe errors higher up? it's not clear what is failing)
@eric-burel
Sorry if the question seems silly but why
bpy==1.2.3
in requirements.txt won't work?
My apologies but 1.2.3
has been removed. The recommendation is to install the version that you actually want, such as 2.81
.
also
pip install bpy --no-binary
asks me a param for --no-binary, so I guess the correct command ispip install bpy --no-binary future_fstrings
The parameter that should be used is :all:
, as shown in the readme
For example:
python3.7 -m pip install bpy --no-binary :all:
Edit 3: what is the state of the limit issue? pypa/pypi-support#362 Is there currently a way to get a simpler install?
The upload limit issue is resolved; I have the wheels built and placed on pypi for Windows and MacOS.
I can't do the same for linux as it's not as completely straightforward due to limitations on how you can upload wheel
s to pypi. The wheel
would have to be using the manylinux
tag from PEP 513, which requires me to learn Docker.
Until then, I have a place on my Google Drive containing some wheel
files that should be relatively straightforward to install. You are more than welcome to give that a shot in the meantime.
So as far as installation from pypi... Not until I figure out what manylinux
i.e PEP 513 is all about. Never used Docker before.
it's not clear what is failing
I can see in the log something of interest:
CMake Error: The source directory "/tmp/pip-install-s0rqjx0w/bpy" does not appear to contain CMakeLists.txt.
Which would imply that you are using an outdated version of cmake
. I would suggest you uninstall your current version and try building a later version of cmake
Script seems to take forever to run for me without much logs.
When installing from pip
, logging is opt-in as far as I am aware. See how to log using pip
on this stackoverflow.com answer
Sadly I have the right version of Cmake (3.17). However I installed it through snap, as it's a bit difficult to get a version to cmake upper to 3.10 on Ubuntu, which may explain the issue. I had similar troubles with Chromium installed as snap unable to find tmp
. Will investigate a bit more later.
Thanks a lot for the wheels on Google Driver in the meantime, you are awesome!
I'll try to spend some time on the Docker thing, though on my side it's been a very long time since I tried to build smth myself from sources...
Hi, I've tried the following Dockerfile, still no success:
FROM python:3.7
WORKDIR /app
COPY . /app # suppose you have a working app
RUN pip install -r requirements.txt # suppose you have future_strings here
RUN wget -qO- "https://cmake.org/files/v3.17/cmake-3.17.0-Linux-x86_64.tar.gz" | tar --strip-components=1 -xz -C /usr/local # cmake 3.17. With manylinux images, you need curl instead, and the install still failed in the end (missing deps)
RUN pip install bpy --no-binary :all:
I get the following error (after multiple messages):
xml.etree.ElementTree.ParseError: syntax error: line 1, column 0
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys,
setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-lvph4jx1/bpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-lvph4jx1/bpy/setup.py'"'"';
f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read()
.replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
install --record /tmp/pip-record-gviry0lr/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/bpy Check the logs for full command output.
The command '/bin/sh -c pip install bpy --no-binary :all:' returned a non-zero code: 1
Regarding building wheels, I think I understand the concept of manylinux, but it's not completely clear in the provided examples how you are supposed to setup the build environment. You don't really need to learn Docker per se, more the CLI, as in the end they seem to use it as some blackbox script to build the wheel. The relevant issue is probably this one, to avoid using Travis: https://github.com/pypa/manylinux/issues/73
Edit: actually you seem to need to setup a Docker image actually, based on the manylinux image. At the moment I havent figured out how to properly install a recent cmake ^^
Sorry for the very late reply, I was severely distracted with a few other projects...
@Kuro-Maii Do you know a good way to detect linux distributions? What does
distro.linux_distribution(full_distribution_name=False)
report for you? (Note, requires installing python distro module)
Python 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import distro >>> distro.linux_distribution(full_distribution_name=False) ('debian', '10', 'buster') >>>
I am reading thru the other comments and going to test the the rest.
I get ImportError: libjemalloc.so.1: cannot open shared object file: No such file or directory
when trying to build with Debian 10 Buster too (which is used by Python 3.7 image), no way to bypass it.
And I still can't build the source in the Dockerfile (see post just above).
So atm only installing on Ubuntu with provided wheels worked for me.
@eric-burel
I get
ImportError: libjemalloc.so.1: cannot open shared object file: No such file or directory
when trying to build with Debian 10 Buster too
Wow, that is so odd to me, because I could have sworn that issue went away when I committed to the bpy-build
repo the commit to set JEMALLOC off for Raspbian and Debian.
I was able to build this on Raspbian - only using the old 2.79 32-bit release though, now that 32-bis is discontinued.
So atm only installing on Ubuntu with provided wheels worked for me.
At least it's not totally broken! Just kidding, I'm sure it's frustrating not being able to install this onto your system of choice.
Could you try cloning the bpy-build
repo
git clone https://github.com/TylerGubala/bpy-build.git
And stepping through the get_configure_commands
function to see where it is hanging up?
I don't have a Debian machine available.
I have the same error
ImportError: libjemalloc.so.1: cannot open shared object file: No such file or directory
At the line import bpy
. I installed bpy via the wheel provided in your Gdrive : bpy-2.82-cp37-cp37m-linux_x86_64.whl and then ran bpy_post_install.
Here are my specs :
$ python --version
Python 3.7.8
>>> distro.linux_distribution()
('Ubuntu', '20.04', 'focal')
Edit: I got the same result under ubuntu 16.04.
Are you able to do
sudo apt install libjemalloc-dev
And see if that resolves it?
On Tue, Jun 30, 2020, 10:17 AM Apollinaire Lecocq notifications@github.com wrote:
I have the same error ImportError: libjemalloc.so.1: cannot open shared object file: No such file or directory
At the line import bpy. I installed bpy via the wheel provided in your Gdrive : bpy-2.82-cp37-cp37m-linux_x86_64.whl and then ran bpy_post_install.
Here are my specs :
$ python --version Python 3.7.8
distro.linux_distribution() ('Ubuntu', '20.04', 'focal')
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TylerGubala/blenderpy/issues/1#issuecomment-651820922, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFLEYRZM6YHGHX2CFW4V2RLRZHXW3ANCNFSM4E5OMKZA .
Same error after installing lijemalloc-dev.
That is very bizarre. I will have to look into it over the weekend and get
back to you, maybe that is coming from install_deps.sh
or something and I
have it and don't get the error because I have already run that...? I'm not
sure.
On Wed, Jul 1, 2020, 5:32 AM Apollinaire Lecocq notifications@github.com wrote:
Same error after installing lijemalloc-dev.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TylerGubala/blenderpy/issues/1#issuecomment-652307921, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFLEYR7E7JOWB6CI2KE7UJDRZL7DHANCNFSM4E5OMKZA .
Hi, progressing a bit! I try to repro an install in a Docker image (still not really investigating build, just looking for a quick solution until things get sorted out for Linux)
So, it seems that the "libjemalloc.so.1" thing is only solvable using Ubuntu 18.04 as the base image. Other distros (debian etc.), or later versions (eg 19) won't have "libjemalloc1" package and will fail. This is the (simplified) image:
FROM ubuntu:18.04
WORKDIR /app
COPY . /app
RUN apt-get -y update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends apt-utils
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install python3.7
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install python3-pip
# Use python3.7
RUN python3.7 -m pip install -r requirements.txt
# Dependency for blender python
RUN apt-get -y install libtbb-dev
RUN apt-get install libjemalloc1 libjemalloc-dev
RUN python3.7 -m pip install "/app/bpy-2.82-cp37-cp37m-linux_x86_64.whl"
RUN bpy_post_install
Now I have this error:
ImportError: libfftw3.so.3: cannot open shared object file: No such file or directory
But at least no jemalloc stuff.
Edit: solved after installing libfftw3-3
, now I have ImportError: libosdGPU.so.3.4.0: cannot open shared object file: No such file or directory
. The lib I have to install for this one is less obvious, I keep looking.
Second edit: "libosdgpu3.4.0", "libosdcpu3.4.0", "libosd-dev" seems to exist on Ubuntu 20, not 18... so I don't get why it works on my machine in the first place (Ubuntu 18, I do have those libosdGPU.so.3.4.0 on my machine), and I still can't run in a Docker.
Edit 2: we can install https://github.com/PixarAnimationStudios/OpenSubdiv/tree/rc/v3_4_0
in older distro. It needs to install a bunch of deps however to be built (cmake etc. etc.) so the build is quite long. Still testing stuff, not yet done.
Edit 3: gave up at liboslcomp.so.1.9
(OpenShadingLanguage), because I need an higher version of cmake, which is very slow to get in Ubuntu. I don't get why there are so many missing deps, and basically no way to install them, while everything was fine on my own Ubuntu install. I've probably installed another program, maybe even when trying to build blenderpy?
Edit again: it seems that I need to run Blender's "install_deps.sh", even with the wheel, because there is a ton of shared libraries to have installed in the first place. The wheel simply allows me to skip the build (or at least that's what I understand) You seem to have come to the same conclusion in #38 However install_deps.sh tends to fail in Docker image or even locally
Some devs are working on Docker images:
CI / Docker is in development on the manylinux-ci
branch so hopefully hopefully generally compatible wheel
availability soon.
@TylerGubala Hopeful I am! By development you mean you run the image or you are creating it? Btw I've managed to have an image that does build.
It weights no less than 9.1 gb (!), takes 2 hours to build, buuuut... it goes as far as installing deps.
FROM ubuntu:18.04
ENV os=ubuntu:18.04
ENV TERM=xterm
ENV maintainer "Egidijus Ligeika"
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get dist-upgrade -qy && \
apt-get install -yq \
gcc libc6-dev make curl automake \
bash-completion \
python3 python3-setuptools python3-dev \
libxml2-utils \
locales \
lsb-release \
build-essential \
python python-setuptools python-dev \
nvidia-cuda-dev python3-pycuda python-pycuda \
sudo libboost-tools-dev magics++\
vim mlocate apt-utils \
liblld-6.0-dev llvm-6.0-dev \
libtbb-dev \
git \
wget
# cmake 3.12 @see https://www.claudiokuenzler.com/blog/796/install-upgrade-cmake-3.12.1-ubuntu-14.04-trusty-alternatives
RUN wget http://www.cmake.org/files/v3.12/cmake-3.12.1.tar.gz && tar -xvzf cmake-3.12.1.tar.gz
RUN cd cmake-3.12.1/ && ./configure && make && sudo make install
RUN sudo update-alternatives --install /usr/bin/cmake cmake /usr/local/bin/cmake 1 --force
RUN echo "source /etc/profile.d/bash_completion.sh" >> ~/.bashrc
ADD blender-src /build/blender-src
WORKDIR /build/blender-src
RUN ./build_files/build_environment/install_deps.sh --show-deps && \
./build_files/build_environment/install_deps.sh --with-all --no-confirm --force-boost --force-python --build-llvm --build-osl --force-osl
Mostly taken from https://github.com/egidijus/blender-compile (you need to follow the doc to have the correct folder structure), just fixed a dep and the cmake install. It lacks some reproducbility as it clones the latest blender.
Cmake build is extremely slow, being able to use a snap or prebuilt version would allow to gain a few dozen minutes.
Working commit of Blender is "2de5de57c58521862e0fecc95fc474ea347b7468" It also adds only Python 3.6.9
By development you mean you run the image or you are creating it?
What's it matter? Maybe you can answer that question for yourself based on what I am doing:
travis.yml
from the manylinux
demo projectIt weights no less than 9.1 gb (!), takes 2 hours to build
That is a major pain point with this whole process, yeah.
Cmake build is extremely slow, being able to use a snap or prebuilt version would allow to gain a few dozen minutes
If you have any recommendations feel free to submit a PR; any gains or speedups to be had with this process would be tremendous.
Honestly I do not think that I will be able to create a manylinux
compatible wheel
file because glibc-2.9
is not available on Centos 5, upon which Blender depends.
Do you think it would be doable to have at least a slightly better support of other major linux distros? Debian likes, latest Ubuntu, Fedora etc.? Can we help by running some specific code on our machines? If the process of rebuilding Blender wasn't so complicated/unstable/unpredictable we could definitely give a hand on this.
@eric-burel
Do you think it would be doable to have at least a slightly better support of other major linux distros? Debian likes, latest Ubuntu, Fedora etc.?
I'm pretty sure that's the exact problem that will be solved by the manylinux
wheel
files.
I just pushed up a commit to add the Dockerfile that I am currently working on, the one in the .docker/manylinux
folder. It uses an older version of Centos (requirement for manylinux) and I at least have it up to the point of searching for compatible Blender versions online.
I think that if you can get Blender building following the manual instructions then those instructions can be automated. That is essentially the process I have followed to get to this point.
If the process of rebuilding Blender wasn't so complicated/unstable/unpredictable we could definitely give a hand on this.
Have you tried asking for help on Blender's Dev Talk Forum? If you have issues building Blender itself if you post the issue you are having and the operating system you are on there you should find some good help.
Thanks for sharing the Dockerfile, that's a great basis. Maybe it could be reused on specific systems by changing the "FROM" to another distro (+ tweaking the syntax a bit).
Yeah I've tried the Dev Talk Forum, on post related to Docker, but everybody is acting like Blender build perfectly all the time on their machine. But Blender build is really hard to reproduce depending on certain machine, or within a Dockerfile, a lot of underlying dependencies seems to miss in the list they publish on Blender documentation. Every Dockerfiles I've found are not reproducible, outdated or never actually worked in the first place.
Honestly I do not think that I will be able to create a manylinux compatible wheel file because glibc-2.9 is not available on Centos 5, upon which Blender depends.
Note that there are much more recent manylinux specifications than manylinux1
, including manylinux2014
which uses CentOS 7 as a base and includes glibc 2-17. In the projects I've worked on, updating from an older manylinux tag to a newer one has just required changing the base docker image I depend on (I mostly use manylinux 2010, e.g., quay.io/pypa/manylinux2010_x86_64).
@jamadden it has gotten a lot better after attempting to use the manylinux2014
image, yeah. There is just a lot to track down. I think it might be serendipity that Blender's own svn
libraries are built off the CentOS7 platform.
@eric-burel I know that it can be a bit painful to go into the Blender Dev Talk forum but by BF's own admission it is the best resource for getting advice on building Blender in any environment.
Also some progress has been made in the Docker department, I have made some VS Code tasks to automate the build of those environments as well.
If you have insight into Docker (I am still very much a noobie at it) could you clone my repo, download VS Code and go:
View -> Command Palette -> Tasks: Run Task -> Docker - Build Ubuntu Manual
And let me know what you think?
PRs also welcomed and appreciated; this is a lot to track down admittedly, especially for multi platform.
The VS code tasks are a nice addition, thank you. Here is what I get for "Build Ubuntu Manual":
When compiling the Python module, they don't link correctly to OpenMP I think.
That could be solved by patching CMakeLists.txt as suggested below and following up to make sure the build still works.
https://devtalk.blender.org/t/centos-7-manylinux-build-difficulties/15007/6
On Wed, Aug 26, 2020, 3:29 AM Eric Burel notifications@github.com wrote:
The VS code tasks are a nice addition, thank you. Here is what I get for "Build Ubuntu Manual":
Scanning dependencies of target cycles_bvh [100%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh.cpp.o ... [100%] Linking CXX static library ../../../lib/libcycles_bvh.a [100%] Built target cycles_bvh Scanning dependencies of target blender [100%] Building C object source/creator/CMakeFiles/blender.dir/creator.c.o [100%] Building C object source/creator/CMakeFiles/blender.dir/creator_signals.c.o [100%] Building C object source/creator/CMakeFiles/blender.dir/creator_args.c.o [100%] Building C object source/creator/CMakeFiles/blender.dir/buildinfo.c.o [100%] Linking CXX shared module ../../bin/bpy.so [100%] Linking CXX shared module ../../bin/bpy.so /usr/bin/ld.gold: error: /usr/lib/gcc/x86_64-linux-gnu/10/libgomp.a(team.o): unsupported reloc 23 against global symbol gomp_tls_data /usr/bin/ld.gold: error: /usr/lib/gcc/x86_64-linux-gnu/10/libgomp.a(team.o): unsupported reloc 23 against global symbol gomp_tls_data
[here I got multiple times the same error message]
/usr/bin/ld.gold: error: /usr/lib/gcc/x86_64-linux-gnu/10/libgomp.a(affinity.o): requires dynamic R_X86_64_PC32 reloc against 'stderr' which may overflow at runtime; recompile with -fPIC [here I got multiple times the same error message] /usr/bin/ld.gold: error: /usr/lib/gcc/x86_64-linux-gnu/10/libgomp.a(oacc-init.o): unsupported reloc 23 against global symbol goacc_tls_data /usr/bin/ld.gold: error: /usr/lib/gcc/x86_64-linux-gnu/10/libgomp.a(affinity-fmt.o): requires dynamic R_X86_64_PC32 reloc against 'stderr' which may overflow at runtime; recompile with -fPIC /usr/bin/ld.gold: error: /usr/lib/gcc/x86_64-linux-gnu/10/libgomp.a(oacc-profiling.o): requires dynamic R_X86_64_PC32 reloc against 'acc_prof_lookup' which may overflow at runtime; recompile with -fPIC /usr/bin/ld.gold: error: /usr/lib/gcc/x86_64-linux-gnu/10/libgomp.a(env.o): requires dynamic R_X86_64_PC32 reloc against 'stderr' which may overflow at runtime; recompile with -fPIC /usr/bin/ld.gold: error: /usr/lib/gcc/x86_64-linux-gnu/10/libgomp.a(error.o): requires dynamic R_X86_64_PC32 reloc against 'stderr' which may overflow at runtime; recompile with -fPIC collect2: error: ld returned 1 exit status make[3]: [source/creator/CMakeFiles/blender.dir/build.make:414: bin/bpy.so] Error 1 make[2]: [CMakeFiles/Makefile2:6419: source/creator/CMakeFiles/blender.dir/all] Error 2 make[1]: [Makefile:163: all] Error 2 make: [GNUmakefile:313: all] Error 2 The command '/bin/sh -c cd blender && make bpy' returned a non-zero code: 2 The terminal process "/bin/bash '-c', 'docker build -t blenderpy/ubuntu-manual -f .docker/ubuntu-manual/Dockerfile .'" terminated with exit code: 2.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TylerGubala/blenderpy/issues/1#issuecomment-680710247, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFLEYR6HWWYXOLDPELQK5VLSCS2U5ANCNFSM4E5OMKZA .
@eric-burel It should be working now.
I agree it worked on Ubuntu 20.04 on the HEAD of master, I had to specify cmake option PNG_LIBRARY_RELEASE
to make the compilation work, otherwise I had an error with libpng :
cmake -C../blender/build_files/cmake/config/bpy_module.cmake -DPNG_LIBRARY_RELEASE=~/Documents/blender-git/lib/linux_centos7_x86_64/png/lib/libpng16.a ../blender/
I finally have manylinux
wheel
files building.
Hopefully I will be able to add these to PYPI soon, the built files are quite large (184Mb)
See also (I got issue number 666 just in time for halloween)
For linux installs please use the wheel
files that are supplied on the Releases page.
If you continue to have issues post the error in a new issue. Thanks.
My project using Blenderpy is sadly not active anymore, but thanks a lot for all your work, I am looking forward for using it in the future!
Currently the setup script has only been tested under Windows
The current build script may work for Ubuntu though, or may be close per this:
https://wiki.blender.org/wiki/Building_Blender/Linux/Ubuntu#Installing_CMake
More testing and more support is needed for Fedora, Gentoo, Arch, and OpenSUSE.
Build instructions can be found here:
https://wiki.blender.org/wiki/Building_Blender/Linux
To do: