kmilos
commented
2 years ago Duplicate of #167 and #170?
Closed clearlon closed 3 months ago
kmilos
commented
2 years ago Duplicate of #167 and #170?
JQuezada0
commented
2 years ago @clearlon Not sure if you're still having this issue, and to be clear, I have no idea what I'm doing. But I ran into the same exact issue while trying to install something else that listed rawpy as a requirement in a requirements.txt and ultimately managed to get it installed.
My machine info:
macOS Moneterey - 12.2.1 MacBook Pro (14-inch, 2021) Apple M1 Pro
I'm going to be intentionally super verbose for anyone else scouring google for answers.
I got this error first
ERROR: Could not find a version that satisfies the requirement rawpy (from versions: none)
ERROR: No matching distribution found for rawpyWhich then led to this issue I believe https://github.com/letmaik/rawpy/issues/167. And then that led to these instructions: https://github.com/letmaik/rawpy#installation-from-source-on-linuxmacos, where I ended up getting some behavior similar to you, but also different
Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from rawpy==0.17.1) (1.22.4)
Using legacy 'setup.py install' for rawpy, since package 'wheel' is not installed.
Installing collected packages: rawpy
Running setup.py install for rawpy ... done
Successfully installed rawpyIt says successful, but in reality rawpy could not be found or used. But there's a peculiar line in there about wheel
Using legacy 'setup.py install' for rawpy, since package 'wheel' is not installed.So I went ahead and installed wheel via pip3 install wheel. That went well, so I went ahead and tried installing from the rawpy repo again pip3 install ., and that's how I got to the same exact error as you
Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from rawpy==0.17.1) (1.22.4)
Building wheels for collected packages: rawpy
Building wheel for rawpy (setup.py) ... done
WARNING: Legacy build of wheel for 'rawpy' created no files.
Command arguments: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -u -c '
exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
# import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
# setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
# manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize
try:
import setuptools
except ImportError as error:
print(
"ERROR: Can not execute `setup.py` since setuptools is not available in "
"the build environment.",
file=sys.stderr,
)
sys.exit(1)
__file__ = %r
sys.argv[0] = __file__
if os.path.exists(__file__):
filename = __file__
with tokenize.open(__file__) as f:
setup_py_code = f.read()
else:
filename = "<auto-generated setuptools caller>"
setup_py_code = "from setuptools import setup; setup()"
exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/Users/johnilquezada/Development/rawpy/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /private/var/folders/pp/h4fhyjq566v4gqvcfxdd2yrw0000gn/T/pip-wheel-_5q21bkb
Command output: [use --verbose to show]
Running setup.py clean for rawpy
Failed to build rawpy
Installing collected packages: rawpy
Running setup.py install for rawpy ... done
Successfully installed rawpySays successful but it actually failed.
Then I happened to find this issue https://github.com/microsoft/vscode-python/issues/15057. I followed the repro steps in there (but I changed the pip command to pip3):
pip3 install numpy cythonsudo TMPDIR=. pip3 install --verbose --no-clean .This one ended in the same error again, but the output was slightly different. Now it mentioned sh: cmake: command not found, so in a different terminal I ran brew install cmake
Once that finished I went back and re-ran sudo TMPDIR=. pip3 install --verbose --no-clean.
Finally that actually truly installed rawpy, and going back to what I was originally trying to install made it past that step.
I'm now stuck on a different part, but hopefully this at least helps with getting rawpy installed on an M1 Mac.
letmaik
commented
2 years ago Keep in mind that there is always the option to use the amd64 version of Python on M1. It may be a little bit slower than the native one but I doubt it's a practical issue in most cases.
JQuezada0
commented
2 years ago @letmaik Thanks for the heads up. I just tried this via Anaconda and it appears to suffer from the same issue. I added the correct channel, and whether it's via the GUI or CLI, it seems to think rawpy doesn't exist.
Edit: I just went with making a docker image. Everything works fine in there.
moebaid
commented
2 years ago I've managed to successfully install from source on Mac M1. Initially, I had the same errors as @JQuezada0 and followed their solutions, up until running brew install cmake. After this I still had errors running sudo make install (the final step of libraw installation) due to issues with OpenMP. It seems others have had issues installing and using OpenMP on Mac M1 -- I ran the following in a new terminal:
brew install llvm
cd /opt/homebrew/opt/libomp/lib
cd /usr/local/lib
sudo ln -s /opt/homebrew/opt/libomp/lib/libomp.dylib libomp.dylibI also ran brew install libomp, though this might have been redundant. After that, I was able to successfully finish installing libraw then rawpy.
weaming
commented
1 year ago I installed universal2 format python3 from https://www.python.org/downloads/release/python-3108, then use /usr/bin/arch -x86_64 /Library/Frameworks/Python.framework/Versions/3.10/bin/pip to install rawpy from pypi.org
kamikaze
commented
1 year ago any news on being able to install via pip install?
letmaik
commented
1 year ago For anyone waiting, please follow this upstream issue: https://github.com/actions/runner-images/issues/2187
avengersassemble
commented
1 year ago I was able to install rawpy following instructions mentioned by @moebaid My machine info:
macOS Moneterey - 12.6.2 MacBook Pro (13-inch, M1, 2020)
brew install llvm
cd /opt/homebrew/opt/libomp/lib
cd /usr/local/lib
sudo ln -s /opt/homebrew/opt/libomp/lib/libomp.dylib libomp.dylib
python3 -m venv venv
source venv/bin/activate/
git clone https://github.com/letmaik/rawpy
cd rawpy
pip install wheel numpy cython
pip install .It successfully installed rawpy in my site-packages. Later, I deleted rawpy directory.
rafalstapinski
commented
1 year ago I was ultimately able to install rawpy with
Adding my experience to the above solutions for installing on M1 with another issue I ran into with compiling the Cython code due to what seems like a change in the numpy C-API.
The Cython compilation issue was in the two places where _rawpy.pyx sets ndarr.base = <PyObject*> self raising the following error Assignment to a read-only property. After looking through this issue, I replaced the two instances of setting ndarr.base with the following call to numpy's C-API:
# ndarr.base = <PyObject*> self
np.PyArray_SetBaseObject(ndarr, self)After making that change, I was able to build/install rawpy with pip install .
letmaik
commented
1 year ago @rafalstapinski Can you open a pull request with the change for numpy? Thanks!
EmersonDove
commented
1 year ago Hey I built some wheels using my m2 mini local runner you can install with
pip install newrawpy
happy to add my runner to this repo if the owners are interested. This was just a quick patch fix on my end.
letmaik
commented
1 year ago @EmersonDove Thanks for trying this out, it's good to know that it works out of the box just by adding another CI matrix entry with arm64 as python arch. I won't be able to add external runners to this repo though as I couldn't provide any guarantees to users on the provenance of the machine and this would become an attack vector to inject malicious code into the macOS wheel. By using standard GitHub Actions runners there is at least some provenance from the fact that the machines are operated by Microsoft directly. Hopefully it won't take too long for M1 runners to arrive, see also https://github.com/github/roadmap/issues/528 which currently states Q4 2023 – Oct-Dec.
EmersonDove
commented
1 year ago True, I think I would have the same reaction if someone offered that to me
donB092
commented
1 year ago need rawpy on m1 air, new to py - how to?
nullus
commented
1 year ago Sort of shoehorning my own solution in here in the form of #209
I had a similar problem awhile ago, and decided to build against the system libraw, installed using Homebrew:
brew install libraw
env RAWPY_USE_SYSTEM_LIBRAW=1 python setup.py installGlad to hear the M1 runners will be available soon!
wing-kit
commented
1 year ago Sort of shoehorning my own solution in here in the form of #209
I had a similar problem awhile ago, and decided to build against the system libraw, installed using Homebrew:
brew install libraw env RAWPY_USE_SYSTEM_LIBRAW=1 python setup.py installGlad to hear the M1 runners will be available soon!
In file included from rawpy/_rawpy.cpp:1137:
rawpy/def_helper.h:1:10: fatal error: 'libraw_version.h' file not found
#include "libraw_version.h"
^~~~~~~~~~~~~~~~~~
1 warning and 1 error generated.
error: command '/usr/bin/clang' failed with exit code 1
[end of output]just tried to follow it on a M1 Mac. Did you experience this? @nullus
nullus
commented
1 year ago No, looks like it can't find the header path though. If you run pkg-config --cflags libraw do you get any output? It's possible that brew install pkg-config will fix it.
Edit: the standard install procedure will correctly build libraw as part of the rawpy installation too! From memory there may be some other build dependencies required: brew install cmake for one.
wing-kit
commented
1 year ago @nullus thanks. brew install pkg-config did the job.
occamz
commented
10 months ago Looked into this a bit.
There are already beta runners (paid): https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/running-jobs-on-larger-runners?platform=mac#available-macos-larger-runners
The next milestone is General Availability (GA). Expected to be ready: Q1 2024 – Jan-Mar: https://github.com/github/roadmap/issues/819
"Once we reach the GA milestone, we will start working on additional plans to support open source, along with new macOS runners." https://github.com/actions/runner-images/issues/8439#issuecomment-1755601587
So might be a while still :)
chbrown
commented
9 months ago Here's a one-liner shot in the dark that happened to work for me in my env (M2, btw):
pip install 'git+https://github.com/letmaik/rawpy.git@v0.19.0#egg=rawpy'You may want to update the tag.
greiginsydney
commented
8 months ago It fails when I try it @chbrown :
DEPRECATION: git+https://github.com/letmaik/rawpy.git@v0.19.0#egg=rawpy' contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at https://github.com/pypa/pip/issues/11617(Environment is Raspberry Pi, 32-bit bookworm)
lufrikas
commented
8 months ago Hello, I'm trying to install rawpy on my M3 iMac. I am following the "Installation from source on Linux/macOS" as others have done. The installation itself works fine. However, when trying to use rawpy, I get the following error upon import rawpy:
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/rawpy/_rawpy.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__ZN6LibRaw11open_bufferEPKvm'
Any ideas?
kgantchev
commented
8 months ago Hopefully it won't take too long for M1 runners to arrive, see also github/roadmap#528 which currently states Q4 2023 – Oct-Dec.
You can use the FlyCI's M1 and M2 runners. Our runners are on average 2x faster and 2x cheaper than GitHub's AND we have a free tier for OSS projects (see below).
Step 1: Install the FlyCI GitHub app.
Step 2: Change the runs-on flag to use the FlyCI label (see below).
Example with M1 runners:
jobs:
ci:
- runs-on: macos-latest
+ runs-on: flyci-macos-large-latest-m1
steps:
- name: 👀 Checkout repo
uses: actions/checkout@v4Example with M2 runners:
jobs:
ci:
- runs-on: macos-latest
+ runs-on: flyci-macos-large-latest-m2
steps:
- name: 👀 Checkout repo
uses: actions/checkout@v4If your repo is public, then FlyCI offers 500 mins/month of free M1 runner usage with the flyci-macos-large-latest-m1 runner.
Best Regards, Kiril Gantchev CEO and co-founder of FlyCI
When I installed on MacOS with M1(arm) according to
Installation from source on Linux/macOS, it showed thatSuccessfully installed rawpy, but rawpy was not available.