search

It4innovations / hyperqueue

Scheduler for sub-node tasks for HPC systems with batch scheduling
https://it4innovations.github.io/hyperqueue
MIT License
272 stars 21 forks source link

Update build.yml #728

Open theely opened 1 month ago

theely commented 1 month ago

Builds Python binding on MacOS ARM Relates to: https://github.com/It4innovations/hyperqueue/issues/726

Kobzol commented 1 month ago

Hi, thanks for the PR! I'm still a bit uneasy about providing these packages when we officially don't support them :sweat_smile:, but I guess there's no harm in doing that when we explicitly state that it's a best effort approach.

It looks like HQ_BUILD_VERSION is not propagated into maturin, probably because it is set using docker-options, and Docker isn't used on macOS for the maturin build. Could you try adding HQ_BUILD_VERSION: ${{ needs.set-env.outputs.version }} here?

Kobzol commented 1 month ago

Right, so the macOS Python package was built, but most of the Python API tests (expectedly) fail with it. I'm really unsure whether we want to release packages are unsupported and also fail most tests, especially when interested users can just build the package locally.. @spirali What do you think?

theely commented 1 month ago

Indeed this is less than ideal. Conceptually it makes sense to provide client libs (Python package) built for multiple architectures even if the server binary only runs on x64. Unfortunately, I don't see a way to build a client on one architecture and test it against a server running on a different one.

Kobzol commented 1 month ago

The client should probably work on macOS (not 100% sure though), so it is a valid use-case (although most HQ users just run the client on a supercomputer). But as you said, providing testing of a macOS client and e.g. a Linux/x64 server/worker is highly non-trivial and we do not have the necessary testing infrastructure prepared for that.

spirali commented 1 month ago

I like the idea that we are distributing MacOs client, but without at least partial automatic testing it is quite thin ice and it will be broken somewhere in the future.

Does GitHub CI allow something like docker compose? So we can start linux & macos instance together? Or can be we start linux docker container inside a macos CI instance?

Kobzol commented 1 month ago

Starting a (virtualized) Linux Docker container inside a macOS is probably possible, but I'm not sure if that works on GitHub Actions, I didn't find much information about it online.