noirello / pyorc

Python module for Apache ORC file format
Apache License 2.0
64 stars 20 forks source link

Have a universal2 wheel be created for macos in addition to the standard x86_64 wheel. #55

Closed dbaxa closed 2 years ago

dbaxa commented 2 years ago

Signed-off-by: David Black dblack@atlassian.com

dbaxa commented 2 years ago

This is now working :-) - so I'll clean this up.

dbaxa commented 2 years ago

This works as tested on a mac os 12 ec2 m1 instance ->

brew install pipenv 
brew install python@3.10

# setup a pipenv for 3.10
# here ^^^^^

# download wheels from ci 
curl --output wheels.zip "https://noirello.visualstudio.com/d41b569e-7f1c-42e1-91a5-ea56fcfc7ea2/_apis/build/builds/485/artifacts?artifactName=wheels&api-version=7.0&%24format=zip"

unzip wheels.zip
pip install wheels/pyorc-0.7.0-cp310-cp310-macosx_10_9_universal2.whl

git clone https://github.com/noirello/pyorc.git
cd pyorc

CERT_PATH=$(python -m certifi)
export SSL_CERT_FILE=${CERT_PATH}
export REQUESTS_CA_BUNDLE=${CERT_PATH}

pip install pybind11

python setup.py build_ext --download-only
cd deps/orc-1.7.5
cmake -DCMAKE_BUILD_TYPE=DEBUG -DBUILD_JAVA=OFF  -DBUILD_LIBHDFSPP=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON
make -j 8 package 

cp -a examples ../
cp -a ./_CPack_Packages/Darwin/TGZ/ORC-1.7.5-Darwin/ ../
cd ../../

pip install -U pytest pytest-cov "pytest-xdist[psutil]" coverage codecov
pytest -vs --junitxml=./test-results.xml tests/ -n auto

which should have

====================================================================== 261 passed in 73.19s (0:01:13) =======================================================================

as the output.

You can also install the python3.8 & 3.9 wheel files & test them in a similar fashion (e.g. pipenv install --python /System/Volumes/Data/Library/Developer/CommandLineTools/usr/bin/python, pip install pybind11 & then pip install wheels/pyorc-0.7.0-cp38-cp38-macosx_10_9_universal2.whl. You do not need to re-build the orc dep and can run the tests in the pyorc repository after installing the required test deps)

dbaxa commented 2 years ago

Additionally, the universal2 wheel files will also work on amd64 macs (I tested using a similar setup as found in the above comment on an ec2 mac amd64 system. I tested 3.10 & the system provided 3.8.9 ).


The different wheels that were generated as part of the CI for this branch have been tested and they work on/with

✅ mac os 12 m1 (system) 3.8.9 ✅ mac os 12 m1 (brew) 3.9 ✅ mac os 12 m1 (brew) 3.10

✅ mac os 12 amd64 (system) 3.8.9 ✅ mac os 12 amd64 (brew) 3.9 ✅ mac os 12 amd64 (brew) 3.10

dbaxa commented 2 years ago

@noirello can we get this merged & mac m1 compatible wheel files made available :-) ?