Closed Popolon closed 3 years ago
Hello,
I had a look on your pkgbuild. I think I found the problems. :)
The first thing is that you do not need to download assimp sources at all, it is included in the tarball you download from PyPI:
$ wget https://files.pythonhosted.org/packages/source/y/yoga/yoga-1.0.0.tar.gz
[...]
$ tar -xvzf yoga-1.0.0.tar.gz
[...]
$ ls -lah yoga-1.0.0/assimp
total 80K
drwxr-xr-x 7 fabien fabien 4,0K avril 27 20:13 .
drwxr-xr-x 5 fabien fabien 4,0K avril 27 20:13 ..
-rw-r--r-- 1 fabien fabien 345 avril 27 20:13 assimp.pc.in
drwxr-xr-x 2 fabien fabien 4,0K avril 27 20:13 cmake
-rw-r--r-- 1 fabien fabien 31K avril 27 20:13 CMakeLists.txt
drwxr-xr-x 2 fabien fabien 4,0K avril 27 20:13 cmake-modules
drwxr-xr-x 9 fabien fabien 4,0K avril 27 20:13 code
drwxr-xr-x 16 fabien fabien 4,0K avril 27 20:13 contrib
-rw-r--r-- 1 fabien fabien 4,5K avril 27 20:13 CREDITS
drwxr-xr-x 3 fabien fabien 4,0K avril 27 20:13 include
-rw-r--r-- 1 fabien fabien 3,8K avril 27 20:13 LICENSE
-rw-r--r-- 1 fabien fabien 933 avril 27 20:13 revision.h.in
The python3 setup.py build_py
command seems to be useless here because it only compiles the library "in place", but the generated .so file will not be copied by the python3 setup.py install
command (I do not know why setuptools have this behaviour).
As the library (.so) seems to be only compilable during the install
command, you should remove the --skip-build
from the command.
Here are all the command I used to generated an installation in a ./output
folder:
wget https://files.pythonhosted.org/packages/source/y/yoga/yoga-1.0.0.tar.gz
tar -xvzf yoga-1.0.0.tar.gz
cd yoga-1.0.0
python3 setup.py install --root=./output --optimize=1
You should be able to reduce your pkgbuild to something like I think:
pkgname=('yoga')
_module='yoga'
pkgver='1.0.0'
pkgrel=4
pkgdesc="Yummy Optimizer for Gorgeous Assets"
url="https://github.com/wanadev/yoga"
depends=(python3 python-cffi python-pillow python-pycparser python-unidecode python-pyguetzli python-zopflipy)
makedepends=('python-setuptools')
license=('BSD')
arch=('x86_64')
source=("https://files.pythonhosted.org/packages/source/${_module::1}/$_module/$_module-$pkgver.tar.gz")
sha256sums=('a4e7010969602b42af2ca74c95126ee533e3cf9ea9fdaa0c5c316eec7d985f53')
package() {
depends+=()
cd "${srcdir}/${_module}-${pkgver}"
python setup.py install --root="${pkgdir}" --optimize=1
}
Note:
any
as there is compiled parts (and this is the same for python-pyguetzli
and python-zopflipy
; only yoga-image-optimizer
has no compiled parts in it)Hum also, I do not know arch packaging so maybe it is assumed it is present, but please note that cmake
is required to build assimp :)
If I don't get the source, it doesn't compile at all. At least it work with default assimp in my case with main account.
"any" is not as in Debian here. It means in a PKGBUILD it can be compiled in any architectures, else makepkg will throw an error and stop unless the architecture is present or -A flag is added to makepkg, this is the habit on most pyhon packages. As these package are architecture independant, it works fine.
Here are the error when I compile with included assimp:
running build_py
-- The C compiler identification is GNU 11.1.0
-- The CXX compiler identification is GNU 11.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Shared libraries disabled
-- compiling zlib from sources
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Enabled importer formats: AMF 3DS AC ASE ASSBIN B3D BVH COLLADA DXF CSM HMP IRRMESH IRR LWO LWS M3D MD2 MD3 MD5 MDC MDL NFF NDO OFF OBJ OGRE OPENGEX PLY MS3D COB BLEND IFC XGL FBX Q3D Q3BSP RAW SIB SMD STL TERRAGEN 3D X X3D GLTF 3MF MMD
-- Disabled importer formats:
-- Enabled exporter formats: OBJ OPENGEX PLY 3DS ASSBIN ASSXML M3D COLLADA FBX STL X X3D GLTF 3MF PBRT ASSJSON STEP
-- Disabled exporter formats:
-- Configuring done
-- Generating done
-- Build files have been written to: /data/pacaur/yoga/src/yoga-1.0.0/assimp/build
[ 0%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/compress.c.o
[ 0%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/adler32.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 1%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/crc32.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 1%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/deflate.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 2%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/gzclose.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 2%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/gzlib.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 3%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/gzread.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 3%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/gzwrite.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 3%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/inflate.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 4%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/infback.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 4%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/inftrees.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 5%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/inffast.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 5%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/trees.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 6%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/uncompr.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 6%] Building C object contrib/zlib/CMakeFiles/zlibstatic.dir/zutil.c.o
cc1: attention: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C
[ 7%] Linking C static library libzlibstatic.a
[ 7%] Built target zlibstatic
[ 7%] Building CXX object code/CMakeFiles/assimp.dir/CApi/CInterfaceIOWrapper.cpp.o
[ 7%] Building CXX object code/CMakeFiles/assimp.dir/Common/Assimp.cpp.o
[ 8%] Building CXX object code/CMakeFiles/assimp.dir/Common/BaseImporter.cpp.o
[ 8%] Building CXX object code/CMakeFiles/assimp.dir/Common/BaseProcess.cpp.o
[ 9%] Building CXX object code/CMakeFiles/assimp.dir/Common/PostStepRegistry.cpp.o
[ 9%] Building CXX object code/CMakeFiles/assimp.dir/Common/ImporterRegistry.cpp.o
[...] I cut these part here as everntyhing goes without error
[ 77%] Building CXX object code/CMakeFiles/assimp.dir/AssetLib/X3D/X3DImporter.cpp.o
[ 77%] Building CXX object code/CMakeFiles/assimp.dir/AssetLib/glTF/glTFCommon.cpp.o
[ 78%] Building CXX object code/CMakeFiles/assimp.dir/AssetLib/glTF/glTFImporter.cpp.o
[ 78%] Building CXX object code/CMakeFiles/assimp.dir/AssetLib/glTF2/glTF2Importer.cpp.o
Dans le fichier inclus depuis /data/pacaur/yoga/src/yoga-1.0.0/assimp/code/AssetLib/glTF/glTFAsset.h:1159,
depuis /data/pacaur/yoga/src/yoga-1.0.0/assimp/code/AssetLib/glTF/glTFImporter.cpp:45:
/data/pacaur/yoga/src/yoga-1.0.0/assimp/code/AssetLib/glTF/glTFAsset.inl: Dans la fonction membre « void glTF::Asset::Load(const string&, bool) »:
/data/pacaur/yoga/src/yoga-1.0.0/assimp/code/AssetLib/glTF/glTFAsset.inl:1304:29: erreur: écrit 1 octet dans une région de taille 0 [-Werror=stringop-overflow=]
1304 | sceneData[mSceneLength] = '\0';
cc1plus : tous les avertissements sont traités comme des erreurs
make[2]: *** [code/CMakeFiles/assimp.dir/build.make:2372 : code/CMakeFiles/assimp.dir/AssetLib/glTF/glTFImporter.cpp.o] Erreur 1
make[2]: *** Attente des tâches non terminées....
make[1]: *** [CMakeFiles/Makefile2:169 : code/CMakeFiles/assimp.dir/all] Erreur 2
make: *** [Makefile:136 : all] Erreur 2
and then it try to make the python setup and fail too.
I coded a lot in C++ during several years, until 20 years ago, so the language growed, but basis are the same :).
In my point of view, problem is here:
// read the scene data
std::vector<char> sceneData(mSceneLength + 1);
sceneData[mSceneLength] = '\0';
std::vector<char>
This glTFAsset.inl file isnt present int he original assimp package.
We're not in java. The variable constructor is called with the provided size. My feeling is that the gcc11 warning is wrong.
I never programed in java, don't know it.
Sorry for my message that looks a bit aggressive now I read it gain. What I meant is that the c++ seems correct. Can You try disabling that GCC warning?
I read more doc about std::vector class and examples and you are true, sorry for my stupid statement I forgot this part of C++ with 20 years without practising, just remembered new form of allocator. More strangely, if I add this useless assignment it works:
std::vector<char> sceneData(mSceneLength + 1);
sceneData[0]= 'a';
sceneData[mSceneLength] = '\0';
But if fail again for another warning, warning level should probably lowered in python build, but I don't know python build system at all.
It looks like flags should be changed here in setup.py, (line 32). which flag do you suggest with today c++ standards?
if ccompiler.get_default_compiler() == "unix":
os.environ["CPPFLAGS"] = "--std=c++11"
Output of the next fail:
[ 9%] Building CXX object code/CMakeFiles/assimp.dir/AssetLib/M3D/M3DExporter.cpp.o
Dans le fichier inclus depuis /data/pacaur/yoga/src/yoga-1.0.0/assimp/code/AssetLib/M3D/M3DWrapper.h:59,
depuis /data/pacaur/yoga/src/yoga-1.0.0/assimp/code/AssetLib/M3D/M3DExporter.cpp:69:
/data/pacaur/yoga/src/yoga-1.0.0/assimp/code/AssetLib/M3D/m3d.h: Dans la fonction « unsigned char* m3d_save(m3d_t*, int, int, unsigned int*) »:
/data/pacaur/yoga/src/yoga-1.0.0/assimp/code/AssetLib/M3D/m3d.h:5523:46: erreur: array subscript « uint64_t {aka long unsigned int}[0] » is partly outside array bounds of « unsigned char [4] » [-Werror=array-bounds]
5523 | *((uint64_t *)out) = *((uint64_t *)&weights[0]);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
/data/pacaur/yoga/src/yoga-1.0.0/assimp/code/AssetLib/M3D/m3d.h:4349:43: note: while referencing « weights »
4349 | unsigned char *out = NULL, *z = NULL, weights[M3D_NUMBONE], *norm = NULL;
| ^~~~~~~
cc1plus : tous les avertissements sont traités comme des erreurs
make[2]: *** [code/CMakeFiles/assimp.dir/build.make:2568 : code/CMakeFiles/assimp.dir/AssetLib/M3D/M3DExporter.cpp.o] Erreur 1
make[1]: *** [CMakeFiles/Makefile2:169 : code/CMakeFiles/assimp.dir/all] Erreur 2
make: *** [Makefile:136 : all] Erreur 2
For more information, the problem is with gcc 11.1.0
For more information, the current problem is with gcc 11.1.0 here is the changelog, but it looks more like a bug than a willing change in behavior https://gcc.gnu.org/gcc-11/porting_to.html
You can try:
os.environ["CPPFLAGS"] = "--std=c++11 -Wno-error=stringop-overflow -Wno-error=array-bounds"
By the way, (if above works...) can you try CXXFLAGS
instead of CPPFLAGS
?
I wonder why for, size_t (c stdlib) is used instead of std::size_t (c++). Anyway I tried both in a simple example and they are compiled without problem (using size_t or std::size_t with the same GCC version.
#include <vector>
#include "stdlib.h"
int main()
{
size_t length=150;
std::vector<char> test(length+1);
test[length]='\0';
}
Worked both with g++ -o test_vector --std=c++11 test_vector.cpp
and g++ -o test_vector --std=c++11 -Lstdlib test_vector.cpp
It will not be easy to make a bug report, if this is really a bug, but not related to a problem detected elsewhere.
It pass previous errors, but fail again here (the sameway with CPPFLAGS/CXXFLAGS. I believe they are mutually interchangable in gcc. Forcing g++ instead of gcc could resolve the issue.
[ 89%] Building C object code/CMakeFiles/assimp.dir/AssetLib/Assjson/cencode.c.o cc1: erreur: l'option de la ligne de commande « -std=c++11 » est valable pour C++/ObjC++ mais pas pour C [-Werror]
Beside this problem, good news I was able to build yoga on GCC10.2.1 on RISC-V 64 architecture
Works if removing --std=c++11 and adding the two parameters you give to avoid the warnings.
sed -i 's/--std=c++11/-Wno-error=stringop-overflow -Wno-error=array-bounds/' setup.py
python3 setup.py build_py
python3 yoga/model/assimp_build.py
Strangely with other users or root accounts, I still have this error:
$ yoga
Traceback (most recent call last):
File "/usr/bin/yoga", line 33, in <module>
sys.exit(load_entry_point('yoga==1.0.0', 'console_scripts', 'yoga')())
File "/usr/bin/yoga", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 855, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/usr/lib/python3.9/site-packages/yoga/__init__.py", line 2, in <module>
from . import model # noqa
File "/usr/lib/python3.9/site-packages/yoga/model/__init__.py", line 174, in <module>
from .assimp import assimp_import_from_bytes, assimp_export_to_bytes
File "/usr/lib/python3.9/site-packages/yoga/model/assimp.py", line 6, in <module>
from ._assimp import lib, ffi
ModuleNotFoundError: No module named 'yoga.model._assimp'
Ok I think I have the right commands to build now:
# Get source
wget https://files.pythonhosted.org/packages/source/y/yoga/yoga-1.0.0.tar.gz
tar -xvzf yoga-1.0.0.tar.gz
cd yoga-1.0.0
# Build
python3 setup.py build
# Install in ./output/usr
python3 setup.py install --prefix=./output/usr --optimize=1 --skip-build
I wonder why for, size_t (c stdlib) is used instead of std::size_t (c++). Anyway I tried both in a simple example and they are compiled without problem (using size_t or std::size_t with the same GCC version.
std::size_t is an alias to "c-style" size_t. They are purely equivalent. One can be assigned to the other without a cast as there is no strong typing.
#include <vector> #include "stdlib.h" int main() { size_t length=150; std::vector<char> test(length+1); test[length]='\0'; }
Worked both with
g++ -o test_vector --std=c++11 test_vector.cpp
andg++ -o test_vector --std=c++11 -Lstdlib test_vector.cpp
It will not be easy to make a bug report, if this is really a bug, but not related to a problem detected elsewhere.
Add -Wall -Wextra -Werror
to your gcc command line.
It pass previous errors, but fail again here (the sameway with CPPFLAGS/CXXFLAGS. I believe they are mutually interchangable in gcc. Forcing g++ instead of gcc could resolve the issue.
They're different. They are used by the build system, not GCC directly.
Ok I think I have the right commands to build now: (...)
Nice!
I just don't know why your build with GCC11.1 seems to have -Wall -Werror
enabled by default, but it looks like it is what is causing all your troubles.
I started a documentation to help packagers: https://wanadev.github.io/yoga/contributing.html#packaging-yoga
@Breush Thank you for all these details.
@flozz python3-pil is missing for Bullseye (tested on RISC-V flavor) in the documentation.
Worked both with
g++ -o test_vector --std=c++11 test_vector.cpp
andg++ -o test_vector --std=c++11 -Lstdlib test_vector.cpp
It will not be easy to make a bug report, if this is really a bug, but not related to a problem detected elsewhere.Add
-Wall -Wextra -Werror
to your gcc command line.
No error either. The whole line I used:
g++ -o test_vector --std=c++11 -Wall -Wextra -Werror test_vector.cpp
@flozz python3-pil is missing for Bullseye (tested on RISC-V flavor) in the documentation.
For source install it is not a system dependency as it should be installed from PyPI (like all other Python dependencies). But I understand that on non-intel platform, installing PIL/Pillow from PyPI could be problematic as there is probably no binary distribution (wheel) so you will have to compile it (and need to install all PIL / Pillow build deps)
When I said from source, I mean without pip, yes. So it could be a good idea to add this in documentation for non x86 platforms if no binary are available for non-x86 platform.
@Popolon I added some links in the documentation to help :)
I close this issue as the AUR pkg seems to be ok now. Please let me know if there is more issues ;)
How the ArchLinux package is compiled: Depend on assimp, because unable to install it following instruction. The assimp source code is still needed to compile yoga. So
python3 setup.py build_py
build_py is the only one that don't make error, and need this lib. The other documented build process don't work on Arch. The files are then placed in the package dir using:
python setup.py install --root="${pkgdir}" --optimize=1 --skip-build
After installing the made package, it works fine with the user that build the package, but with other user, there is strangely the following error. could be due to some parts that are installed only for the compiling user during the build process?