search

stan-dev / httpstan

HTTP interface to Stan, a package for Bayesian inference.
ISC License
40 stars 15 forks source link

cirrus ci builds for more architectures #645

Closed pdjames closed 3 months ago

pdjames commented 12 months ago

Hi team,

stan appreciator here.

I have some projects for which I use Stan and in particular pystan, frustratingly for me the dev work is macOS M1 and the deployment is raspberry pi. So I got curious to improve my build skills for the purpose of using stand more freely and consistency in my dockerfiles etc.

I think I have some working builds on additional platforms, though not completely tested these are building successfully including the original test scripts and I have been able to install and build on my machines. Here is a link, let me know if you aren't able to see anything there but it should be open; https://cirrus-ci.com/build/4992871254720512

I'm looking now at how to make windows builds work as I see the stanc releases are there, and it seems the makefile and or wsl on cirrus ci is the key to making progress there.

I hope this is beneficial in some way, I can appreciate the difficulties you must have encountered getting the compilation and packaging to work as I have tried several avenues before finding and getting this cirrus ci pipeline to work.

Best regards, Peter

pdjames commented 12 months ago

Just to add, I'm aware this needs some finalisation work for wheels uploading etc, but this commit should be at least testable on your side.

riddell-stan commented 12 months ago

I wonder if there's a simpler route now that Github has M1 runners, https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

pdjames commented 12 months ago

I hadn't seen that yet, that's interesting. They're charging for these machine at the moment.

I was able to get an M1 build to work just now, I needed to do a few of the same things as in this PR wrt the build dependencies, I don't know much about the sphinx errors so just froze them to a version which was compatible.

I don't know what the GitHub actions support for aarch64 is like?

riddell-stan commented 12 months ago

@pdjames Would you be willing to follow (rigidly) the instructions in .github/workflows/main.yml on your Apple Silicon device? If a viable wheel is produced without any changes (or with a couple of changes in the Makefile) I'll find a way to pay Github for the runner.

pdjames commented 11 months ago

Yeah I can give it a test. I was able to get the M1 GitHub runner machine to work, with the same setting as in the cirrus ci build, as there are some problems with using latest versions of the dev dependencies which are remedied by the versions I have frozen in this one. I didn't go into finding out which package is the root cause yet, but could check that out also if necessary. For now, the frozen versions work so unless there are cutting edge improvements that are needed it's fine.

I haven't output a wheel from the GitHub build, though all the tests ran and passed. So it looks promising.

Then I guess aarch64 for raspi is easier than the M1 to configure, and I still want to look at windows as well.

pdjames commented 11 months ago

I'm not sure the cleanest way to do this, but I have a branch with GitHub actions M1 builds producing some artefacts which I've been able to successfully install to venvs on my M1 MacBook and test. This has minimal changes to the .github/workflows/main.yml just those in order to select the machine, get the right environment for build and publish some artefacts to test.

Currently only python 3.10 & 3.11 due to availability in the Github Action: actions/setup-python@v2 on M1 architecture, but this is easy enough to expand out now that the proof of concept of Stan libraries compilation for 3.10 and 3.11 works.

I thought perhaps cibuildwheels is useful for versioning if I can compile libraries on the base runner and copy them into containers to have cibw utilise them in the various implementations.

Perhaps I should put the M1 commit to it's own PR while I keep looking at getting the other os's to work?

riddell-stan commented 11 months ago

If you think you've identified what needs to change for Apple Silicon, a minimal PR would be welcome. I'll look into getting the Apple Silicon runners.

Ultimately, we need to get things working on https://github.com/stan-dev/httpstan-wheels , which uses multibuild. But getting things working here is one place to start.

riddell-stan commented 11 months ago

A new pull request that changes .github/workflows/main.yml so that it builds and run tests using a macos-12-large runner would be welcome. (Apparently you don't need to do anything to sign up for the Apple Silicon beta.)

riddell-stan commented 3 months ago

Closing this. At the moment there's a lack of resources for addressing CI issues and/or adding support for more architectures. Building from source is the only option, I fear.