search

sighingnow / libclang

(Unofficial) Release libclang (clang.cindex) on pypi.
https://pypi.org/project/libclang
Other
81 stars 21 forks source link

libclang package does not seem to exist on MacOS #7

Closed CJxD closed 2 years ago

CJxD commented 3 years ago

I just tried installing libclang on two different machines, and got two different results.

Linux happily accepted pip3 install libclang, but on MacOS, I get:

Collecting libclang
ERROR: Could not find a version that satisfies the requirement libclang (from versions: none)
ERROR: No matching distribution found for libclang

However, both OSes will accept pip3 install clang. The problem then comes when trying to use libclang, which results in the error:

Linux:

libclang-11.so: cannot open shared object file: No such file or directory. To provide a path to libclang use Config.set_library_path() or Config.set_library_file().

MacOS:

clang.cindex.LibclangError: dlopen(libclang.dylib, 6): image not found. To provide a path to libclang use Config.set_library_path() or Config.set_library_file().

I have been able to repeat this on three different Mac machines, and two different Linux machines.

sighingnow commented 3 years ago

Hi @CJxD,

You have encountered the same trouble with #8.

The clang is another package and doesn't bundle the shared library. To install this library please try pip3 install libclang instead.

but on MacOS, I get:

May I know the version of your OS? The published wheel on pypi should works for ABI that compatible with macosx_10_9_x86_64.

CJxD commented 3 years ago

Hey, thanks for getting back promptly.

We have three Macs which all do the same:

However, we've found one Mac that is installing it OK: x86_64 Catalina 10.9

In which case, please could you run a build for arm64 and Big Sur? :)

sighingnow commented 3 years ago

There's no release for arm64 yet. I'm wondering if it is a M1-chip backed Mac?

sighingnow commented 3 years ago

There's no release for arm64 yet. I'm wondering if it is a M1-chip backed Mac?

I don't have access to such laptop, so, helps are welcome 😆

CJxD commented 3 years ago

Yes, it's an M1 :)

Happy to help build or test (although your pipeline badge is showing it's building on macosx-arm64). Send through instructions and I'll make it happen :)

sighingnow commented 3 years ago

Yes, it's an M1 :)

Happy to help build or test (although your pipeline badge is showing it's building on macosx-arm64). Send through instructions and I'll make it happen :)

@CJxD Thank you in advance!

Just run the following command, and then uploaded the generated libclang.dylib here should be enough, then I could prepare the wheel package and upload it to pypi.

LLVM_VER=11.1.0
brew install gcc

wget https://github.com/llvm/llvm-project/releases/download/llvmorg-$LLVM_VER/llvm-project-$LLVM_VER.src.tar.xz
tar xf llvm-project-$LLVM_VER.src.tar.xz
mv llvm-project-$LLVM_VER.src llvm-project-$LLVM_VER

mkdir -p llvm-project-$LLVM_VER/build
cd llvm-project-$LLVM_VER/build
cmake ../llvm -DLLVM_ENABLE_PROJECTS=clang -DBUILD_SHARED_LIBS=OFF -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=MinSizeRel -DCMAKE_CXX_FLAGS_MINSIZEREL="-Os -s -DNDEBUG -static-libgcc -static-libstdc++" -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8 -DCMAKE_OSX_DEPLOYMENT_TARGET=10.9
make libclang -j$(sysctl -n hw.ncpu)

Then there should be a libclang.dylib under the lib folder.

sighingnow commented 3 years ago

@CJxD I have updated the installation command for gcc: please use brew install gcc instead.

CJxD commented 3 years ago

Thanks @sighingnow - GCC still doesn't quite want to install for me - I think I have to emulate x86_64 to run it.

I've managed to build it with clang though - is that still valid?

Changed CMAKE_OSX_DEPLOYMENT_TARGET to 11.2, and LLVM_TARGETS_TO_BUILD to AArch64.

I also changed the build flags a bit: removed -Os as it should be implied, removed -s as it doesn't do anything in clang, removed -static-libgcc and -static-libstdc++ as it will automatically use libc and libc++ (although not sure if it did this statically, will double check after lunch).

Here's the interim result, but haven't run any tests on it just yet: https://we.tl/t-PY8rT0HX27

sighingnow commented 3 years ago

Thanks, I have packed a wheel here:

libclang-11.1.0-py2.py3-none-macosx_11_2_arm64.whl.zip

Could you please helps me to test it?

CJxD commented 3 years ago

Will do! I'm now busy for the rest of the day, but will test ASAP

CJxD commented 3 years ago

Unfortunately the wheel didn't work. I got the following error:

pip install libclang-11.1.0-py2.py3-none-macosx_11_2_arm64.whl 
ERROR: libclang-11.1.0-py2.py3-none-macosx_11_2_arm64.whl is not a supported wheel on this platform.

I'm trying to build it again with gcc, but even after installing gcc, it's aliased to clang so it's not statically linking the standard libraries. Working on it.

sighingnow commented 3 years ago

Unfortunately the wheel didn't work. I got the following error:

pip install libclang-11.1.0-py2.py3-none-macosx_11_2_arm64.whl 
ERROR: libclang-11.1.0-py2.py3-none-macosx_11_2_arm64.whl is not a supported wheel on this platform.

I'm trying to build it again with gcc, but even after installing gcc, it's aliased to clang so it's not statically linking the standard libraries. Working on it.

Could you please try to rename the file as libclang-11.1.0-py2.py3-none-macosx_11_2_aarch64.whl and try again?

CJxD commented 3 years ago

Hey, sorry for the delay. That didn't work either I'm afraid! I think it might actually be due to the binary. I've been busy for the past day but I'll run a new build as soon as possible

sighingnow commented 3 years ago

Thanks @CJxD, I think the "is not a supported wheel on this platform." error comes from the wheel name. It doesn't touch the binary during installation.

CJxD commented 3 years ago

Got it, I'll look into what the name should be

CJxD commented 2 years ago

Hey, sorry for the radio silence. I think I worked out what the issue is!

There is no macosx_11_2: the closest one is macosx_11_0 which would then be compatbile with Big Sur and above which is what we want.

Try setting the OS tags to that when you package it. Here's the complete list of tags I have on my Mac M1 running 11.2:

['cp38-cp38-macosx_11_0_arm64', 'cp38-cp38-macosx_11_0_universal2', 'cp38-cp38-macosx_10_16_universal2', 'cp38-cp38-macosx_10_15_universal2', 'cp38-cp38-macosx_10_14_universal2', 'cp38-cp38-macosx_10_13_universal2', 'cp38-cp38-macosx_10_12_universal2', 'cp38-cp38-macosx_10_11_universal2', 'cp38-cp38-macosx_10_10_universal2', 'cp38-cp38-macosx_10_9_universal2', 'cp38-cp38-macosx_10_8_universal2', 'cp38-cp38-macosx_10_7_universal2', 'cp38-cp38-macosx_10_6_universal2', 'cp38-cp38-macosx_10_5_universal2', 'cp38-cp38-macosx_10_4_universal2', 'cp38-abi3-macosx_11_0_arm64', 'cp38-abi3-macosx_11_0_universal2', 'cp38-abi3-macosx_10_16_universal2', 'cp38-abi3-macosx_10_15_universal2', 'cp38-abi3-macosx_10_14_universal2', 'cp38-abi3-macosx_10_13_universal2', 'cp38-abi3-macosx_10_12_universal2', 'cp38-abi3-macosx_10_11_universal2', 'cp38-abi3-macosx_10_10_universal2', 'cp38-abi3-macosx_10_9_universal2', 'cp38-abi3-macosx_10_8_universal2', 'cp38-abi3-macosx_10_7_universal2', 'cp38-abi3-macosx_10_6_universal2', 'cp38-abi3-macosx_10_5_universal2', 'cp38-abi3-macosx_10_4_universal2', 'cp38-none-macosx_11_0_arm64', 'cp38-none-macosx_11_0_universal2', 'cp38-none-macosx_10_16_universal2', 'cp38-none-macosx_10_15_universal2', 'cp38-none-macosx_10_14_universal2', 'cp38-none-macosx_10_13_universal2', 'cp38-none-macosx_10_12_universal2', 'cp38-none-macosx_10_11_universal2', 'cp38-none-macosx_10_10_universal2', 'cp38-none-macosx_10_9_universal2', 'cp38-none-macosx_10_8_universal2', 'cp38-none-macosx_10_7_universal2', 'cp38-none-macosx_10_6_universal2', 'cp38-none-macosx_10_5_universal2', 'cp38-none-macosx_10_4_universal2', 'cp37-abi3-macosx_11_0_arm64', 'cp37-abi3-macosx_11_0_universal2', 'cp37-abi3-macosx_10_16_universal2', 'cp37-abi3-macosx_10_15_universal2', 'cp37-abi3-macosx_10_14_universal2', 'cp37-abi3-macosx_10_13_universal2', 'cp37-abi3-macosx_10_12_universal2', 'cp37-abi3-macosx_10_11_universal2', 'cp37-abi3-macosx_10_10_universal2', 'cp37-abi3-macosx_10_9_universal2', 'cp37-abi3-macosx_10_8_universal2', 'cp37-abi3-macosx_10_7_universal2', 'cp37-abi3-macosx_10_6_universal2', 'cp37-abi3-macosx_10_5_universal2', 'cp37-abi3-macosx_10_4_universal2', 'cp36-abi3-macosx_11_0_arm64', 'cp36-abi3-macosx_11_0_universal2', 'cp36-abi3-macosx_10_16_universal2', 'cp36-abi3-macosx_10_15_universal2', 'cp36-abi3-macosx_10_14_universal2', 'cp36-abi3-macosx_10_13_universal2', 'cp36-abi3-macosx_10_12_universal2', 'cp36-abi3-macosx_10_11_universal2', 'cp36-abi3-macosx_10_10_universal2', 'cp36-abi3-macosx_10_9_universal2', 'cp36-abi3-macosx_10_8_universal2', 'cp36-abi3-macosx_10_7_universal2', 'cp36-abi3-macosx_10_6_universal2', 'cp36-abi3-macosx_10_5_universal2', 'cp36-abi3-macosx_10_4_universal2', 'cp35-abi3-macosx_11_0_arm64', 'cp35-abi3-macosx_11_0_universal2', 'cp35-abi3-macosx_10_16_universal2', 'cp35-abi3-macosx_10_15_universal2', 'cp35-abi3-macosx_10_14_universal2', 'cp35-abi3-macosx_10_13_universal2', 'cp35-abi3-macosx_10_12_universal2', 'cp35-abi3-macosx_10_11_universal2', 'cp35-abi3-macosx_10_10_universal2', 'cp35-abi3-macosx_10_9_universal2', 'cp35-abi3-macosx_10_8_universal2', 'cp35-abi3-macosx_10_7_universal2', 'cp35-abi3-macosx_10_6_universal2', 'cp35-abi3-macosx_10_5_universal2', 'cp35-abi3-macosx_10_4_universal2', 'cp34-abi3-macosx_11_0_arm64', 'cp34-abi3-macosx_11_0_universal2', 'cp34-abi3-macosx_10_16_universal2', 'cp34-abi3-macosx_10_15_universal2', 'cp34-abi3-macosx_10_14_universal2', 'cp34-abi3-macosx_10_13_universal2', 'cp34-abi3-macosx_10_12_universal2', 'cp34-abi3-macosx_10_11_universal2', 'cp34-abi3-macosx_10_10_universal2', 'cp34-abi3-macosx_10_9_universal2', 'cp34-abi3-macosx_10_8_universal2', 'cp34-abi3-macosx_10_7_universal2', 'cp34-abi3-macosx_10_6_universal2', 'cp34-abi3-macosx_10_5_universal2', 'cp34-abi3-macosx_10_4_universal2', 'cp33-abi3-macosx_11_0_arm64', 'cp33-abi3-macosx_11_0_universal2', 'cp33-abi3-macosx_10_16_universal2', 'cp33-abi3-macosx_10_15_universal2', 'cp33-abi3-macosx_10_14_universal2', 'cp33-abi3-macosx_10_13_universal2', 'cp33-abi3-macosx_10_12_universal2', 'cp33-abi3-macosx_10_11_universal2', 'cp33-abi3-macosx_10_10_universal2', 'cp33-abi3-macosx_10_9_universal2', 'cp33-abi3-macosx_10_8_universal2', 'cp33-abi3-macosx_10_7_universal2', 'cp33-abi3-macosx_10_6_universal2', 'cp33-abi3-macosx_10_5_universal2', 'cp33-abi3-macosx_10_4_universal2', 'cp32-abi3-macosx_11_0_arm64', 'cp32-abi3-macosx_11_0_universal2', 'cp32-abi3-macosx_10_16_universal2', 'cp32-abi3-macosx_10_15_universal2', 'cp32-abi3-macosx_10_14_universal2', 'cp32-abi3-macosx_10_13_universal2', 'cp32-abi3-macosx_10_12_universal2', 'cp32-abi3-macosx_10_11_universal2', 'cp32-abi3-macosx_10_10_universal2', 'cp32-abi3-macosx_10_9_universal2', 'cp32-abi3-macosx_10_8_universal2', 'cp32-abi3-macosx_10_7_universal2', 'cp32-abi3-macosx_10_6_universal2', 'cp32-abi3-macosx_10_5_universal2', 'cp32-abi3-macosx_10_4_universal2', 'py38-none-macosx_11_0_arm64', 'py38-none-macosx_11_0_universal2', 'py38-none-macosx_10_16_universal2', 'py38-none-macosx_10_15_universal2', 'py38-none-macosx_10_14_universal2', 'py38-none-macosx_10_13_universal2', 'py38-none-macosx_10_12_universal2', 'py38-none-macosx_10_11_universal2', 'py38-none-macosx_10_10_universal2', 'py38-none-macosx_10_9_universal2', 'py38-none-macosx_10_8_universal2', 'py38-none-macosx_10_7_universal2', 'py38-none-macosx_10_6_universal2', 'py38-none-macosx_10_5_universal2', 'py38-none-macosx_10_4_universal2', 'py3-none-macosx_11_0_arm64', 'py3-none-macosx_11_0_universal2', 'py3-none-macosx_10_16_universal2', 'py3-none-macosx_10_15_universal2', 'py3-none-macosx_10_14_universal2', 'py3-none-macosx_10_13_universal2', 'py3-none-macosx_10_12_universal2', 'py3-none-macosx_10_11_universal2', 'py3-none-macosx_10_10_universal2', 'py3-none-macosx_10_9_universal2', 'py3-none-macosx_10_8_universal2', 'py3-none-macosx_10_7_universal2', 'py3-none-macosx_10_6_universal2', 'py3-none-macosx_10_5_universal2', 'py3-none-macosx_10_4_universal2', 'py37-none-macosx_11_0_arm64', 'py37-none-macosx_11_0_universal2', 'py37-none-macosx_10_16_universal2', 'py37-none-macosx_10_15_universal2', 'py37-none-macosx_10_14_universal2', 'py37-none-macosx_10_13_universal2', 'py37-none-macosx_10_12_universal2', 'py37-none-macosx_10_11_universal2', 'py37-none-macosx_10_10_universal2', 'py37-none-macosx_10_9_universal2', 'py37-none-macosx_10_8_universal2', 'py37-none-macosx_10_7_universal2', 'py37-none-macosx_10_6_universal2', 'py37-none-macosx_10_5_universal2', 'py37-none-macosx_10_4_universal2', 'py36-none-macosx_11_0_arm64', 'py36-none-macosx_11_0_universal2', 'py36-none-macosx_10_16_universal2', 'py36-none-macosx_10_15_universal2', 'py36-none-macosx_10_14_universal2', 'py36-none-macosx_10_13_universal2', 'py36-none-macosx_10_12_universal2', 'py36-none-macosx_10_11_universal2', 'py36-none-macosx_10_10_universal2', 'py36-none-macosx_10_9_universal2', 'py36-none-macosx_10_8_universal2', 'py36-none-macosx_10_7_universal2', 'py36-none-macosx_10_6_universal2', 'py36-none-macosx_10_5_universal2', 'py36-none-macosx_10_4_universal2', 'py35-none-macosx_11_0_arm64', 'py35-none-macosx_11_0_universal2', 'py35-none-macosx_10_16_universal2', 'py35-none-macosx_10_15_universal2', 'py35-none-macosx_10_14_universal2', 'py35-none-macosx_10_13_universal2', 'py35-none-macosx_10_12_universal2', 'py35-none-macosx_10_11_universal2', 'py35-none-macosx_10_10_universal2', 'py35-none-macosx_10_9_universal2', 'py35-none-macosx_10_8_universal2', 'py35-none-macosx_10_7_universal2', 'py35-none-macosx_10_6_universal2', 'py35-none-macosx_10_5_universal2', 'py35-none-macosx_10_4_universal2', 'py34-none-macosx_11_0_arm64', 'py34-none-macosx_11_0_universal2', 'py34-none-macosx_10_16_universal2', 'py34-none-macosx_10_15_universal2', 'py34-none-macosx_10_14_universal2', 'py34-none-macosx_10_13_universal2', 'py34-none-macosx_10_12_universal2', 'py34-none-macosx_10_11_universal2', 'py34-none-macosx_10_10_universal2', 'py34-none-macosx_10_9_universal2', 'py34-none-macosx_10_8_universal2', 'py34-none-macosx_10_7_universal2', 'py34-none-macosx_10_6_universal2', 'py34-none-macosx_10_5_universal2', 'py34-none-macosx_10_4_universal2', 'py33-none-macosx_11_0_arm64', 'py33-none-macosx_11_0_universal2', 'py33-none-macosx_10_16_universal2', 'py33-none-macosx_10_15_universal2', 'py33-none-macosx_10_14_universal2', 'py33-none-macosx_10_13_universal2', 'py33-none-macosx_10_12_universal2', 'py33-none-macosx_10_11_universal2', 'py33-none-macosx_10_10_universal2', 'py33-none-macosx_10_9_universal2', 'py33-none-macosx_10_8_universal2', 'py33-none-macosx_10_7_universal2', 'py33-none-macosx_10_6_universal2', 'py33-none-macosx_10_5_universal2', 'py33-none-macosx_10_4_universal2', 'py32-none-macosx_11_0_arm64', 'py32-none-macosx_11_0_universal2', 'py32-none-macosx_10_16_universal2', 'py32-none-macosx_10_15_universal2', 'py32-none-macosx_10_14_universal2', 'py32-none-macosx_10_13_universal2', 'py32-none-macosx_10_12_universal2', 'py32-none-macosx_10_11_universal2', 'py32-none-macosx_10_10_universal2', 'py32-none-macosx_10_9_universal2', 'py32-none-macosx_10_8_universal2', 'py32-none-macosx_10_7_universal2', 'py32-none-macosx_10_6_universal2', 'py32-none-macosx_10_5_universal2', 'py32-none-macosx_10_4_universal2', 'py31-none-macosx_11_0_arm64', 'py31-none-macosx_11_0_universal2', 'py31-none-macosx_10_16_universal2', 'py31-none-macosx_10_15_universal2', 'py31-none-macosx_10_14_universal2', 'py31-none-macosx_10_13_universal2', 'py31-none-macosx_10_12_universal2', 'py31-none-macosx_10_11_universal2', 'py31-none-macosx_10_10_universal2', 'py31-none-macosx_10_9_universal2', 'py31-none-macosx_10_8_universal2', 'py31-none-macosx_10_7_universal2', 'py31-none-macosx_10_6_universal2', 'py31-none-macosx_10_5_universal2', 'py31-none-macosx_10_4_universal2', 'py30-none-macosx_11_0_arm64', 'py30-none-macosx_11_0_universal2', 'py30-none-macosx_10_16_universal2', 'py30-none-macosx_10_15_universal2', 'py30-none-macosx_10_14_universal2', 'py30-none-macosx_10_13_universal2', 'py30-none-macosx_10_12_universal2', 'py30-none-macosx_10_11_universal2', 'py30-none-macosx_10_10_universal2', 'py30-none-macosx_10_9_universal2', 'py30-none-macosx_10_8_universal2', 'py30-none-macosx_10_7_universal2', 'py30-none-macosx_10_6_universal2', 'py30-none-macosx_10_5_universal2', 'py30-none-macosx_10_4_universal2', 'py38-none-any', 'py3-none-any', 'py37-none-any', 'py36-none-any', 'py35-none-any', 'py34-none-any', 'py33-none-any', 'py32-none-any', 'py31-none-any', 'py30-none-any']

CJxD commented 2 years ago

Furthermore, I just tried to install libclang on my Alpine Linux build server, which also had this issue. You have set the Linux versions to use manylinux1 which doesn't contain some distros. I think, however, that you can just use linux as libclang doesn't use any OS-specific features.

CJxD commented 2 years ago

And here's libclang 12.0.0 for Mac M1: https://we.tl/t-SLdOcqo6Ea

sighingnow commented 2 years ago

Thank you @CJxD, I have made a wheel package for arm64 based on the libclang.dylib above:

dist.zip

sighingnow commented 2 years ago

Furthermore, I just tried to install libclang on my Alpine Linux build server, which also had this issue. You have set the Linux versions to use manylinux1 which doesn't contain some distros. I think, however, that you can just use linux as libclang doesn't use any OS-specific features.

Thanks for the suggestion, I will try it and if it works I will upload a linux wheel to pypi as well.

sighingnow commented 2 years ago

Furthermore, I just tried to install libclang on my Alpine Linux build server, which also had this issue. You have set the Linux versions to use manylinux1 which doesn't contain some distros. I think, however, that you can just use linux as libclang doesn't use any OS-specific features.

Thanks for the suggestion, I will try it and if it works I will upload a linux wheel to pypi as well.

I have build a wheel with linux (rather than manylinux1) abi, and successfully installed but failed to run a test program on alpine, as the missing of mallinfo symbol.

CJxD commented 2 years ago

Thank you @CJxD, I have made a wheel package for arm64 based on the libclang.dylib above:

dist.zip

Great! That's working fine on my Mac now.

Annoyingly though, it won't install in my venv because it's missing the py3-...-11_0 tag.

My virtualenv was setup with python3 -m venv .venv, and installation with .venv/bin/pip3 install libclang-12.0.0-py2.py3-none-macosx_11_0_arm64.whl.

Seems the closest it can get there is py38-...-11_0. I presume this is a bug in the packaging package, but perhaps it's worth putting py37 in the compatible pythons or something? It has all second digits from 0-8, here's the full list:

['cp38-cp38-macosx_11_0_arm64', 'cp38-cp38-macosx_11_0_universal2', 'cp38-cp38-macosx_10_16_universal2', 'cp38-cp38-macosx_10_15_universal2', 'cp38-cp38-macosx_10_14_universal2', 'cp38-cp38-macosx_10_13_universal2', 'cp38-cp38-macosx_10_12_universal2', 'cp38-cp38-macosx_10_11_universal2', 'cp38-cp38-macosx_10_10_universal2', 'cp38-cp38-macosx_10_9_universal2', 'cp38-cp38-macosx_10_8_universal2', 'cp38-cp38-macosx_10_7_universal2', 'cp38-cp38-macosx_10_6_universal2', 'cp38-cp38-macosx_10_5_universal2', 'cp38-cp38-macosx_10_4_universal2', 'cp38-abi3-macosx_11_0_arm64', 'cp38-abi3-macosx_11_0_universal2', 'cp38-abi3-macosx_10_16_universal2', 'cp38-abi3-macosx_10_15_universal2', 'cp38-abi3-macosx_10_14_universal2', 'cp38-abi3-macosx_10_13_universal2', 'cp38-abi3-macosx_10_12_universal2', 'cp38-abi3-macosx_10_11_universal2', 'cp38-abi3-macosx_10_10_universal2', 'cp38-abi3-macosx_10_9_universal2', 'cp38-abi3-macosx_10_8_universal2', 'cp38-abi3-macosx_10_7_universal2', 'cp38-abi3-macosx_10_6_universal2', 'cp38-abi3-macosx_10_5_universal2', 'cp38-abi3-macosx_10_4_universal2', 'cp38-none-macosx_11_0_arm64', 'cp38-none-macosx_11_0_universal2', 'cp38-none-macosx_10_16_universal2', 'cp38-none-macosx_10_15_universal2', 'cp38-none-macosx_10_14_universal2', 'cp38-none-macosx_10_13_universal2', 'cp38-none-macosx_10_12_universal2', 'cp38-none-macosx_10_11_universal2', 'cp38-none-macosx_10_10_universal2', 'cp38-none-macosx_10_9_universal2', 'cp38-none-macosx_10_8_universal2', 'cp38-none-macosx_10_7_universal2', 'cp38-none-macosx_10_6_universal2', 'cp38-none-macosx_10_5_universal2', 'cp38-none-macosx_10_4_universal2', 'cp37-abi3-macosx_11_0_arm64', 'cp37-abi3-macosx_11_0_universal2', 'cp37-abi3-macosx_10_16_universal2', 'cp37-abi3-macosx_10_15_universal2', 'cp37-abi3-macosx_10_14_universal2', 'cp37-abi3-macosx_10_13_universal2', 'cp37-abi3-macosx_10_12_universal2', 'cp37-abi3-macosx_10_11_universal2', 'cp37-abi3-macosx_10_10_universal2', 'cp37-abi3-macosx_10_9_universal2', 'cp37-abi3-macosx_10_8_universal2', 'cp37-abi3-macosx_10_7_universal2', 'cp37-abi3-macosx_10_6_universal2', 'cp37-abi3-macosx_10_5_universal2', 'cp37-abi3-macosx_10_4_universal2', 'cp36-abi3-macosx_11_0_arm64', 'cp36-abi3-macosx_11_0_universal2', 'cp36-abi3-macosx_10_16_universal2', 'cp36-abi3-macosx_10_15_universal2', 'cp36-abi3-macosx_10_14_universal2', 'cp36-abi3-macosx_10_13_universal2', 'cp36-abi3-macosx_10_12_universal2', 'cp36-abi3-macosx_10_11_universal2', 'cp36-abi3-macosx_10_10_universal2', 'cp36-abi3-macosx_10_9_universal2', 'cp36-abi3-macosx_10_8_universal2', 'cp36-abi3-macosx_10_7_universal2', 'cp36-abi3-macosx_10_6_universal2', 'cp36-abi3-macosx_10_5_universal2', 'cp36-abi3-macosx_10_4_universal2', 'cp35-abi3-macosx_11_0_arm64', 'cp35-abi3-macosx_11_0_universal2', 'cp35-abi3-macosx_10_16_universal2', 'cp35-abi3-macosx_10_15_universal2', 'cp35-abi3-macosx_10_14_universal2', 'cp35-abi3-macosx_10_13_universal2', 'cp35-abi3-macosx_10_12_universal2', 'cp35-abi3-macosx_10_11_universal2', 'cp35-abi3-macosx_10_10_universal2', 'cp35-abi3-macosx_10_9_universal2', 'cp35-abi3-macosx_10_8_universal2', 'cp35-abi3-macosx_10_7_universal2', 'cp35-abi3-macosx_10_6_universal2', 'cp35-abi3-macosx_10_5_universal2', 'cp35-abi3-macosx_10_4_universal2', 'cp34-abi3-macosx_11_0_arm64', 'cp34-abi3-macosx_11_0_universal2', 'cp34-abi3-macosx_10_16_universal2', 'cp34-abi3-macosx_10_15_universal2', 'cp34-abi3-macosx_10_14_universal2', 'cp34-abi3-macosx_10_13_universal2', 'cp34-abi3-macosx_10_12_universal2', 'cp34-abi3-macosx_10_11_universal2', 'cp34-abi3-macosx_10_10_universal2', 'cp34-abi3-macosx_10_9_universal2', 'cp34-abi3-macosx_10_8_universal2', 'cp34-abi3-macosx_10_7_universal2', 'cp34-abi3-macosx_10_6_universal2', 'cp34-abi3-macosx_10_5_universal2', 'cp34-abi3-macosx_10_4_universal2', 'cp33-abi3-macosx_11_0_arm64', 'cp33-abi3-macosx_11_0_universal2', 'cp33-abi3-macosx_10_16_universal2', 'cp33-abi3-macosx_10_15_universal2', 'cp33-abi3-macosx_10_14_universal2', 'cp33-abi3-macosx_10_13_universal2', 'cp33-abi3-macosx_10_12_universal2', 'cp33-abi3-macosx_10_11_universal2', 'cp33-abi3-macosx_10_10_universal2', 'cp33-abi3-macosx_10_9_universal2', 'cp33-abi3-macosx_10_8_universal2', 'cp33-abi3-macosx_10_7_universal2', 'cp33-abi3-macosx_10_6_universal2', 'cp33-abi3-macosx_10_5_universal2', 'cp33-abi3-macosx_10_4_universal2', 'cp32-abi3-macosx_11_0_arm64', 'cp32-abi3-macosx_11_0_universal2', 'cp32-abi3-macosx_10_16_universal2', 'cp32-abi3-macosx_10_15_universal2', 'cp32-abi3-macosx_10_14_universal2', 'cp32-abi3-macosx_10_13_universal2', 'cp32-abi3-macosx_10_12_universal2', 'cp32-abi3-macosx_10_11_universal2', 'cp32-abi3-macosx_10_10_universal2', 'cp32-abi3-macosx_10_9_universal2', 'cp32-abi3-macosx_10_8_universal2', 'cp32-abi3-macosx_10_7_universal2', 'cp32-abi3-macosx_10_6_universal2', 'cp32-abi3-macosx_10_5_universal2', 'cp32-abi3-macosx_10_4_universal2', 'py38-none-macosx_11_0_arm64', 'py38-none-macosx_11_0_universal2', 'py38-none-macosx_10_16_universal2', 'py38-none-macosx_10_15_universal2', 'py38-none-macosx_10_14_universal2', 'py38-none-macosx_10_13_universal2', 'py38-none-macosx_10_12_universal2', 'py38-none-macosx_10_11_universal2', 'py38-none-macosx_10_10_universal2', 'py38-none-macosx_10_9_universal2', 'py38-none-macosx_10_8_universal2', 'py38-none-macosx_10_7_universal2', 'py38-none-macosx_10_6_universal2', 'py38-none-macosx_10_5_universal2', 'py38-none-macosx_10_4_universal2', 'py3-none-macosx_11_0_arm64', 'py3-none-macosx_11_0_universal2', 'py3-none-macosx_10_16_universal2', 'py3-none-macosx_10_15_universal2', 'py3-none-macosx_10_14_universal2', 'py3-none-macosx_10_13_universal2', 'py3-none-macosx_10_12_universal2', 'py3-none-macosx_10_11_universal2', 'py3-none-macosx_10_10_universal2', 'py3-none-macosx_10_9_universal2', 'py3-none-macosx_10_8_universal2', 'py3-none-macosx_10_7_universal2', 'py3-none-macosx_10_6_universal2', 'py3-none-macosx_10_5_universal2', 'py3-none-macosx_10_4_universal2', 'py37-none-macosx_11_0_arm64', 'py37-none-macosx_11_0_universal2', 'py37-none-macosx_10_16_universal2', 'py37-none-macosx_10_15_universal2', 'py37-none-macosx_10_14_universal2', 'py37-none-macosx_10_13_universal2', 'py37-none-macosx_10_12_universal2', 'py37-none-macosx_10_11_universal2', 'py37-none-macosx_10_10_universal2', 'py37-none-macosx_10_9_universal2', 'py37-none-macosx_10_8_universal2', 'py37-none-macosx_10_7_universal2', 'py37-none-macosx_10_6_universal2', 'py37-none-macosx_10_5_universal2', 'py37-none-macosx_10_4_universal2', 'py36-none-macosx_11_0_arm64', 'py36-none-macosx_11_0_universal2', 'py36-none-macosx_10_16_universal2', 'py36-none-macosx_10_15_universal2', 'py36-none-macosx_10_14_universal2', 'py36-none-macosx_10_13_universal2', 'py36-none-macosx_10_12_universal2', 'py36-none-macosx_10_11_universal2', 'py36-none-macosx_10_10_universal2', 'py36-none-macosx_10_9_universal2', 'py36-none-macosx_10_8_universal2', 'py36-none-macosx_10_7_universal2', 'py36-none-macosx_10_6_universal2', 'py36-none-macosx_10_5_universal2', 'py36-none-macosx_10_4_universal2', 'py35-none-macosx_11_0_arm64', 'py35-none-macosx_11_0_universal2', 'py35-none-macosx_10_16_universal2', 'py35-none-macosx_10_15_universal2', 'py35-none-macosx_10_14_universal2', 'py35-none-macosx_10_13_universal2', 'py35-none-macosx_10_12_universal2', 'py35-none-macosx_10_11_universal2', 'py35-none-macosx_10_10_universal2', 'py35-none-macosx_10_9_universal2', 'py35-none-macosx_10_8_universal2', 'py35-none-macosx_10_7_universal2', 'py35-none-macosx_10_6_universal2', 'py35-none-macosx_10_5_universal2', 'py35-none-macosx_10_4_universal2', 'py34-none-macosx_11_0_arm64', 'py34-none-macosx_11_0_universal2', 'py34-none-macosx_10_16_universal2', 'py34-none-macosx_10_15_universal2', 'py34-none-macosx_10_14_universal2', 'py34-none-macosx_10_13_universal2', 'py34-none-macosx_10_12_universal2', 'py34-none-macosx_10_11_universal2', 'py34-none-macosx_10_10_universal2', 'py34-none-macosx_10_9_universal2', 'py34-none-macosx_10_8_universal2', 'py34-none-macosx_10_7_universal2', 'py34-none-macosx_10_6_universal2', 'py34-none-macosx_10_5_universal2', 'py34-none-macosx_10_4_universal2', 'py33-none-macosx_11_0_arm64', 'py33-none-macosx_11_0_universal2', 'py33-none-macosx_10_16_universal2', 'py33-none-macosx_10_15_universal2', 'py33-none-macosx_10_14_universal2', 'py33-none-macosx_10_13_universal2', 'py33-none-macosx_10_12_universal2', 'py33-none-macosx_10_11_universal2', 'py33-none-macosx_10_10_universal2', 'py33-none-macosx_10_9_universal2', 'py33-none-macosx_10_8_universal2', 'py33-none-macosx_10_7_universal2', 'py33-none-macosx_10_6_universal2', 'py33-none-macosx_10_5_universal2', 'py33-none-macosx_10_4_universal2', 'py32-none-macosx_11_0_arm64', 'py32-none-macosx_11_0_universal2', 'py32-none-macosx_10_16_universal2', 'py32-none-macosx_10_15_universal2', 'py32-none-macosx_10_14_universal2', 'py32-none-macosx_10_13_universal2', 'py32-none-macosx_10_12_universal2', 'py32-none-macosx_10_11_universal2', 'py32-none-macosx_10_10_universal2', 'py32-none-macosx_10_9_universal2', 'py32-none-macosx_10_8_universal2', 'py32-none-macosx_10_7_universal2', 'py32-none-macosx_10_6_universal2', 'py32-none-macosx_10_5_universal2', 'py32-none-macosx_10_4_universal2', 'py31-none-macosx_11_0_arm64', 'py31-none-macosx_11_0_universal2', 'py31-none-macosx_10_16_universal2', 'py31-none-macosx_10_15_universal2', 'py31-none-macosx_10_14_universal2', 'py31-none-macosx_10_13_universal2', 'py31-none-macosx_10_12_universal2', 'py31-none-macosx_10_11_universal2', 'py31-none-macosx_10_10_universal2', 'py31-none-macosx_10_9_universal2', 'py31-none-macosx_10_8_universal2', 'py31-none-macosx_10_7_universal2', 'py31-none-macosx_10_6_universal2', 'py31-none-macosx_10_5_universal2', 'py31-none-macosx_10_4_universal2', 'py30-none-macosx_11_0_arm64', 'py30-none-macosx_11_0_universal2', 'py30-none-macosx_10_16_universal2', 'py30-none-macosx_10_15_universal2', 'py30-none-macosx_10_14_universal2', 'py30-none-macosx_10_13_universal2', 'py30-none-macosx_10_12_universal2', 'py30-none-macosx_10_11_universal2', 'py30-none-macosx_10_10_universal2', 'py30-none-macosx_10_9_universal2', 'py30-none-macosx_10_8_universal2', 'py30-none-macosx_10_7_universal2', 'py30-none-macosx_10_6_universal2', 'py30-none-macosx_10_5_universal2', 'py30-none-macosx_10_4_universal2', 'py38-none-any', 'py3-none-any', 'py37-none-any', 'py36-none-any', 'py35-none-any', 'py34-none-any', 'py33-none-any', 'py32-none-any', 'py31-none-any', 'py30-none-any']

CJxD commented 2 years ago

Furthermore, I just tried to install libclang on my Alpine Linux build server, which also had this issue. You have set the Linux versions to use manylinux1 which doesn't contain some distros. I think, however, that you can just use linux as libclang doesn't use any OS-specific features.

Thanks for the suggestion, I will try it and if it works I will upload a linux wheel to pypi as well.

I have build a wheel with linux (rather than manylinux1) abi, and successfully installed but failed to run a test program on alpine, as the missing of mallinfo symbol.

That's a shame :( I wonder if it would do the same if the build actually ran on Alpine. The very fact that the clang-libs package contains a working libclang.so means it's surely possible.

CJxD commented 2 years ago

And for your convenience, here's the build script I made from your sample commands. It actually doesn't matter that I'm building libclang with clang and not gcc (and using libc++ not libstdc++) it seems. LLVM's CMake script is built to handle it.

#!/usr/bin/env bash
set -e

LLVM_VER=${LLVM_VER:-12.0.0}
# TARGETS doesn't really matter too much for libclang, this can be anything
TARGETS=${TARGETS:-X86}
NUM_JOBS=4

case "$(uname -s)" in
Darwin)
    OUTFILE="libclang.dylib"
    echo "-------------------------------------"
    echo "Compiling $OUTFILE for macOS"
    echo "-------------------------------------"
    OSX_TARGET=${OSX_TARGET:-11.0}
    PLATFORM_FLAGS="-DCMAKE_OSX_DEPLOYMENT_TARGET=$OSX_TARGET"
    NUM_JOBS=$(sysctl -n hw.ncpu)
    ;;
Linux)
    OUTFILE="libclang.so"
    echo "-------------------------------------"
    echo "Compiling $OUTFILE for Linux"
    echo "-------------------------------------"
    NUM_JOBS=$(nproc)
    ;;
CYGWIN*|MINGW32*|MSYS*|MINGW*)
    OUTFILE="libclang.dll"
    echo "-------------------------------------"
    echo "Compiling $OUTFILE for Windows"
    echo "-------------------------------------"
    NUM_JOBS=$(nproc)
    ;;
*)
    OUTFILE="libclang"
    echo "-------------------------------------"
    echo "Compiling $OUTFILE for an unknown OS"
    echo "-------------------------------------"
    ;;
esac

project_dir="llvm-project-$LLVM_VER"
mkdir -p "$project_dir"

echo "Downloading LLVM"
wget -c -q --show-progress "https://github.com/llvm/llvm-project/releases/download/llvmorg-$LLVM_VER/$project_dir.src.tar.xz"

echo "Extracting LLVM"
if [ ! -f "$project_dir/.extracted" ]; then
    tar xf "$project_dir.src.tar.xz" -C "$project_dir"
    touch "$project_dir/.extracted"
fi

pushd "$project_dir" > /dev/null
mkdir -p build
pushd build > /dev/null

echo "Configuring the LLVM project"
cmake "../$project_dir.src/llvm" -DLLVM_ENABLE_PROJECTS=clang -DBUILD_SHARED_LIBS=OFF -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_TARGETS_TO_BUILD=$TARGETS -DCMAKE_BUILD_TYPE=MinSizeRel $PLATFORM_FLAGS

echo "Compiling libclang"
cmake --build . --target libclang -- -j$NUM_JOBS

popd > /dev/null
popd > /dev/null

echo "Successfully built to $project_dir/build/lib/$OUTFILE"
sighingnow commented 2 years ago

Thanks a lot for these information！

Great! That's working fine on my Mac now.

Annoyingly though, it won't install in my venv because it's missing the py3-...-11_0 tag.

Do you mean it works with general pip environment but doesn't work when the pip is inside the venv?

sighingnow commented 2 years ago

And for your convenience, here's the build script I made from your sample commands. It actually doesn't matter that I'm building libclang with clang and not gcc (and using libc++ not libstdc++) it seems. LLVM's CMake script is built to handle it.

#!/usr/bin/env bash
set -e

LLVM_VER=${LLVM_VER:-12.0.0}
# TARGETS doesn't really matter too much for libclang, this can be anything
TARGETS=${TARGETS:-X86}
NUM_JOBS=4

case "$(uname -s)" in
Darwin)
    OUTFILE="libclang.dylib"
    echo "-------------------------------------"
    echo "Compiling $OUTFILE for macOS"
    echo "-------------------------------------"
    OSX_TARGET=${OSX_TARGET:-11.0}
    PLATFORM_FLAGS="-DCMAKE_OSX_DEPLOYMENT_TARGET=$OSX_TARGET"
    NUM_JOBS=$(sysctl -n hw.ncpu)
    ;;
Linux)
    OUTFILE="libclang.so"
    echo "-------------------------------------"
    echo "Compiling $OUTFILE for Linux"
    echo "-------------------------------------"
    NUM_JOBS=$(nproc)
    ;;
CYGWIN*|MINGW32*|MSYS*|MINGW*)
    OUTFILE="libclang.dll"
    echo "-------------------------------------"
    echo "Compiling $OUTFILE for Windows"
    echo "-------------------------------------"
    NUM_JOBS=$(nproc)
    ;;
*)
    OUTFILE="libclang"
    echo "-------------------------------------"
    echo "Compiling $OUTFILE for an unknown OS"
    echo "-------------------------------------"
    ;;
esac

project_dir="llvm-project-$LLVM_VER"
mkdir -p "$project_dir"

echo "Downloading LLVM"
wget -c -q --show-progress "https://github.com/llvm/llvm-project/releases/download/llvmorg-$LLVM_VER/$project_dir.src.tar.xz"

echo "Extracting LLVM"
if [ ! -f "$project_dir/.extracted" ]; then
    tar xf "$project_dir.src.tar.xz" -C "$project_dir"
    touch "$project_dir/.extracted"
fi

pushd "$project_dir" > /dev/null
mkdir -p build
pushd build > /dev/null

echo "Configuring the LLVM project"
cmake "../$project_dir.src/llvm" -DLLVM_ENABLE_PROJECTS=clang -DBUILD_SHARED_LIBS=OFF -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_TARGETS_TO_BUILD=$TARGETS -DCMAKE_BUILD_TYPE=MinSizeRel $PLATFORM_FLAGS

echo "Compiling libclang"
cmake --build . --target libclang -- -j$NUM_JOBS

popd > /dev/null
popd > /dev/null

echo "Successfully built to $project_dir/build/lib/$OUTFILE"

Thanks for the script!

sighingnow commented 2 years ago

Furthermore, I just tried to install libclang on my Alpine Linux build server, which also had this issue. You have set the Linux versions to use manylinux1 which doesn't contain some distros. I think, however, that you can just use linux as libclang doesn't use any OS-specific features.

Thanks for the suggestion, I will try it and if it works I will upload a linux wheel to pypi as well.

I have build a wheel with linux (rather than manylinux1) abi, and successfully installed but failed to run a test program on alpine, as the missing of mallinfo symbol.

That's a shame :( I wonder if it would do the same if the build actually ran on Alpine. The very fact that the clang-libs package contains a working libclang.so means it's surely possible.

Yes, it is definitely doable. The current version doesn't work, because we build libclang.so on Linux, which links to glibc, on alpine, after installing glibc there are still symbol not found issue.

CJxD commented 2 years ago

Thanks a lot for these information！

Great! That's working fine on my Mac now. Annoyingly though, it won't install in my venv because it's missing the py3-...-11_0 tag.

Do you mean it works with general pip environment but doesn't work when the pip is inside the venv?

Yes, exactly. It works in general, but not in the venv.

CJxD commented 2 years ago

Furthermore, I just tried to install libclang on my Alpine Linux build server, which also had this issue. You have set the Linux versions to use manylinux1 which doesn't contain some distros. I think, however, that you can just use linux as libclang doesn't use any OS-specific features.

Thanks for the suggestion, I will try it and if it works I will upload a linux wheel to pypi as well.

I have build a wheel with linux (rather than manylinux1) abi, and successfully installed but failed to run a test program on alpine, as the missing of mallinfo symbol.

That's a shame :( I wonder if it would do the same if the build actually ran on Alpine. The very fact that the clang-libs package contains a working libclang.so means it's surely possible.

Yes, it is definitely doable. The current version doesn't work, because we build libclang.so on Linux, which links to glibc, on alpine, after installing glibc there are still symbol not found issue.

Ah I see. I might try compiling on Alpine to see what happens! :)

sighingnow commented 2 years ago

Ah I see. I might try compiling on Alpine to see what happens! :)

Build the libclang.so inside a alpine docker container does works. I have checked in a github action yaml to do that: Ah I see. I might try compiling on Alpine to see what happens! :)

I will update the whl packages for linux then.

sighingnow commented 2 years ago

@CJxD Seems that pypi rejects the general linux_x86_64 abi tag wheels. And I cannot replace the ubuntu-built version with the alpine built version, as it requires libc.musl-x86_64.so.1, which is not available on many generic Linux distributions.

You could find the libclang.so built by alpine on the github action page, if needed.

CJxD commented 2 years ago

That's a shame! Thanks for the attempt though, much appreciated.

sighingnow commented 2 years ago

@CJxD Seems that pypi rejects the general linux_x86_64 abi tag wheels. And I cannot replace the ubuntu-built version with the alpine built version, as it requires libc.musl-x86_64.so.1, which is not available on many generic Linux distributions.

You could find the libclang.so built by alpine on the github action page, if needed.

Hi @CJxD,

I have realized the correct abi tag should be musl_1_2_x86_64 on pypi for alpine linux. I have upload such wheels to pypi for version v13.0.0. You now can install libclang in your alpine linux just with pip3 install libclang.

I have tested it in docker image python:3.9-alpine3.14.

CJxD commented 2 years ago

Awesome stuff @sighingnow , thanks for the update!