search

pacman82 / arrow-odbc-py

Read Apache Arrow batches from ODBC data sources in Python
MIT License
54 stars 5 forks source link

Cannot install `arrow-odbc` on Apple Silicon Macs #54

Closed alexander-beedie closed 7 months ago

alexander-beedie commented 11 months ago

Not sure how much effort would be required to build suitable wheels for distribution (I believe GitHub does offer the necessary runners these days?), but would be great to be able to install on modern Macs without having to compile locally (which I tried, but temporarily gave up on as it was getting a little non-trivial).

pip will attempt to compile it on download, but fails to resolve the relevant paths for linking (assuming you have already downloaded the necessary libraries), etc...😅

pacman82 commented 11 months ago

Hello @alexander-beedie ,

thanks for reporting the issue. Yes, this would be straight forward if GitHub would offer ARM runners. Yet it doesn't. Not to look a gift horse in the mouth though. It is awesome that in this day and age I get this much infrastructure for free anyway.

Sadly I can not see the entire error based on the provided screenshot. The artefact should build nicely on ARM though. The workflow files in this repository should give some hints on what is required to build. On the top of my head:

You need to install unixodbc. You can use brew for this, but last time I had access to an ARM brew put the binaries in different directories on ARM systems, as opposed to other platforms. cargo would not be able to find and link them as a result. This might be outdated knowledge though. I would recommend installing unixodbc from source. See also the ARM section of the odbc-sys Readme: https://github.com/pacman82/odbc-sys. UnixODBC is both a bulid and a runtime dependency.

The second thing you need for the build to work is a rust toolchain, including its build and package manager cargo. Visit https://rustup.rs and follow the instruction. The latest stable toolchain with default options always works.

Feel free to come back here, if you get stuck.

Best, Markus

alexander-beedie commented 11 months ago

thanks for reporting the issue. Yes, this would be straight forward if GitHub would offer ARM runners. Yet it doesn't.

Ahh, I should have been much more specific that the official Apple Silicon runners have only been available (in public beta) for -literally- a few days (I could have sworn they came out earlier, but apparently not) ;)

pacman82 commented 11 months ago

Cool, the blog post mentions some cost for these runners though. I already give my spare time maintaining this. I do not feel inclined to pay on top of it.

alexander-beedie commented 11 months ago

Cool, the blog post mentions some cost for these runners though. I already give my spare time maintaining this. I do not feel inclined to pay on top of it.

Ahh, didn't spot that, my bad - not to worry, entirely understandable! Thanks ;)

pacman82 commented 11 months ago

All good. Improving documentation to build help build locally is an entirely different beast though. Also, recently there has been some effort into making arrow-odbc available via the conda-forge channel. ARM pipeline is currently broken, but it used to work, and is likely to do so again in the future. So if you have anaconda or a similar package manager installed you can use that to install arrow-odbc without building locally yourself.

Best, Markus

pacman82 commented 11 months ago

Hello @alexander-beedie and @ldacey

reopened the issue, because I saw you both mentioning build errors on M1 machines. Sadly I do not have access to one to try myself, but it was one of my core goals to make building and updating arrow-odbc way easier compared to other libraries, by only relying on C-Interfaces to cross the language barrier and not be tied to any C++ ABI or system provided boost version.

With a little bit of help I could probably provide a good description on how to build on M1. If you have access to one of these machines and one of you would be up for a pair programming (well, building) session I would be game.

I am located in the Central European Time zone FYI.

Best, Markus

alexander-beedie commented 11 months ago

With a little bit of help I could probably provide a good description on how to build on M1. If you have access to one of these machines and one of you would be up for a pair programming (well, building) session I would be game.

Sounds like a plan; would be happy to help :) As far as timezones go, I'm only 2 hours off CET, so shouldn't be too challenging to find a good time.

pacman82 commented 11 months ago

@alexander-beedie I invited you to a repository with my contact data in order to connect.

tomsej commented 10 months ago

For anyone who is trying to install arrow-odbc on Apple Silicon. First you need to install unixodbc - brew install unixodbc. Now if you try to install arrow-odbc you will get following error:

    = note: ld: warning: ignoring duplicate libraries: '-lc', '-lm'
            ld: warning: search path '/Users/radek.tomsej/lib' not found
            ld: library 'odbc' not found
            clang: error: linker command failed with exit code 1 (use -v to see invocation)

All you need to do is create a simling to specified search path: sudo ln -s /opt/homebrew/lib /Users/radek.tomsej/lib. Then you are able to install it.

pacman82 commented 10 months ago

Thanks again @tomsej ,

this worked again here: https://github.com/pacman82/arrow-odbc/issues/75#issuecomment-1797394808

pacman82 commented 9 months ago

Added a little section of how to build for ARM Mac to the readme. Thanks for everybodies help in here.

ldacey commented 9 months ago

Nice, thanks!

Successfully built arrow-odbc Installing collected packages: arrow-odbc Successfully installed arrow-odbc-2.0.2

pacman82 commented 9 months ago

Thanks @ldacey for confirming it is working#

radeva commented 8 months ago

@pacman82 In case you need M1 runners in the future, you can try FlyCI's M1 and M2 runners. They are on average 2x faster and 2x cheaper than GitHub's AND we have a free tier for OSS projects (see below).

Install Instructions

  1. Install FlyCI app and
  2. Easily replace one line of code and start using FlyCI runners:
jobs:
 ci:
-    runs-on: macos-latest
+    runs-on: flyci-macos-large-latest-m1
   steps:
   - name: 👀 Checkout repo
     uses: actions/checkout@v4

500 mins/month Free for Public Repos

Since your repo is public, FlyCI offers 500 mins/month of free M1 runner usage with the flyci-macos-large-latest-m1 runner for public projects.

Don't hesitate to contact us in case the free tier doesn't suit your needs or you experience any issues with the runners. Our team is here to support you!

Best Regards, Veselina Radeva Product Manager at FlyCI

pacman82 commented 7 months ago

Hello @radeva ,

I am short on time atm. , yet this is indeed interesting. Soon, as I find myself having some time to spare, I will give it a look.

Best, Markus

pacman82 commented 7 months ago

Hello @radeva ,

thanks to FlyCI's free tier both arrow-odbc and odbc2parquet now enjoy a release for Mac ARM 64. Thank you!

Best, Markus