search

conda-forge / airflow-feedstock

A conda-smithy repository for airflow.
BSD 3-Clause "New" or "Revised" License
6 stars 17 forks source link

Add support for linux/arm64 and darwin/arm64 #103

Closed Volatus closed 1 year ago

Volatus commented 1 year ago

Signed-off-by: Ismayil Mirzali ismayilmirzeli@gmail.com

Checklist

closes #101

conda-forge-webservices[bot] commented 1 year ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

xylar commented 1 year ago

@Volatus, I tried to do architecture migration in #102. It may be more successful to comment there, though you are welcome to see if you can make more headway here. You need to add several things to the build sections of the builds as in: https://github.com/conda-forge/airflow-feedstock/pull/102/files#diff-f3725a55bf339595bf865fec73bda8ac99f283b0810c205442021f29c06eea9aR25-R27 For now, you will also need to skip airflow-with-ldap for the new architectures because dependencies have not yet been built: https://github.com/conda-forge/airflow-feedstock/pull/102/files#diff-f3725a55bf339595bf865fec73bda8ac99f283b0810c205442021f29c06eea9aR387-R388

Volatus commented 1 year ago

@xylar thanks for the pointers, I'll see what I can do

xylar commented 1 year ago

@conda-forge-admin, please rerender

conda-forge-webservices[bot] commented 1 year ago

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

conda-forge-webservices[bot] commented 1 year ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Volatus commented 1 year ago

@conda-forge-admin, please rerender

github-actions[bot] commented 1 year ago

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/airflow-feedstock/actions/runs/4404255160.

github-actions[bot] commented 1 year ago

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/airflow-feedstock/actions/runs/4404263799.

Volatus commented 1 year ago

@xylar It seems like some issue with yarn but when I tried the yarn install on the Airflow repo everything went fine. Do you think it's possible that the wrapper install script ends up shadowing the real errors underneath? Is there a way I can run the packaging locally to test? (I'm on darwin/arm64)

xylar commented 1 year ago

@Volatus, I also don't understand the yarn issue. It seems to have emerged very recently and since it's appearing in #104, I don't think it's related to anything you changed here.

xylar commented 1 year ago

I asked about it here: https://matrix.to/#/!SOyumkgPRWoXfQYIFH:matrix.org/$1678705789214366sdSaV:matrix.org?via=matrix.org&via=gitter.im&via=gnome.org but no one has responded so far.

Volatus commented 1 year ago

@Volatus, I also don't understand the yarn issue. It seems to have emerged very recently and since it's appearing in #104, I don't think it's related to anything you changed here.

Right, and I thought Airflow broke something recently but it did seem like yarn worked fine for me locally on their repo. How would I try running a local build of this feedstock for my host and arch to test some things?

xylar commented 1 year ago

Hmm, I don't think Airflow changed but it does seem like some of the yarn dependencies aren't working anymore. We probably need to figure out how to patch airflow/www/yarn.lock to fix it but I don't have time or expertise to debug this.

Right, and I thought Airflow broke something recently but it did seem like yarn worked fine for me locally on their repo. How would I try running a local build of this feedstock for my host and arch to test some things?

If you can install docker, you can use: https://github.com/conda-forge/airflow-feedstock/blob/main/build-locally.py to build locally. I can't use docker (it interferes with my ability to connect to my institution's wifi) so I use:

conda mambabuild -m .ci_support/<config>.yaml recipe

That's not foolproof (it can pick up system libraries occasionally) but works pretty well.

xylar commented 1 year ago

You will need to install mamba and boa for this. boa provides mambabuild. I believe there are currently some compatibility issues with the latest conda, mamba and boa so it might be safest to go with a version of conda from earlier this year so that you can have a working installation of conda and boa. (Very inconvenient, so I hope it gets worked out soon!)

xylar commented 1 year ago

@Volatus, it looks like there is a release candidate for 2.5.2: https://github.com/apache/airflow/issues/30028 That suggests to me that there might be a new release in a week or so. I think we should just wait for that and try again. Feel free to debug in the meantime but I won't have time to spend on this until after that release.

xylar commented 1 year ago

@Volatus, if you rebase, you should be able to test again. The fix was to constrain yarn <2.

Volatus commented 1 year ago

@Volatus, if you rebase, you should be able to test again. The fix was to constrain yarn <2.

@xylar Will rebase off of main now.

Volatus commented 1 year ago

@xylar So here's the takeaways:

  1. Not sure why the MacOS ARM builds failed, might be something with conda-build.
  2. linux/aarch64 failed on 3.9 and 3.10, but succeeds for 3.8, which makes perfect sense, due to this change introduced in 3.9. It would seem that passlib, a dependency of sqlalchemy ends up creating a bcrypt hash, which Ubuntu's (the current build environment) glibc cannot handle.
xylar commented 1 year ago

@Volatus, okay, we could wait on OSX-ARM for now (and make a separate PR where we ask for help from the experts).

Regarding aarch64, I don't think there's anything we can do about updating glibc (I've had a discussion with core folks about that on some other feedstock). Anything we can do to patch or bypass this?

Volatus commented 1 year ago

Regarding aarch64, I don't think there's anything we can do about updating glibc (I've had a discussion with core folks about that on some other feedstock). Anything we can do to patch or bypass this?

I was wondering how I can change the Docker image for the aarch64 runs. There is an anvil for aarch64 and it would massively speed up our CI builds. Might even fix the issue with crypt, worth a try I think.

xylar commented 1 year ago

It probably makes sense to skip this test when we're cross-compiling: https://github.com/conda-forge/airflow-feedstock/blob/a1cff27029be376b1a190215c69bcf1c160ad270/recipe/meta.yaml#L114

xylar commented 1 year ago

I don't think we have the ability to change the docker images. That has to be done consistently across feedstocks by the conda-forge/core folks.

xylar commented 1 year ago

@Volatus, I solved the OSX ARM issue by moving two of the package from host (target architecture) to build (native architecture). That got merged in #106 and should be available soon (if not now).

I'm working on the Linux architectures in #105 but certainly am open to suggestions you might have. Feel free to keep working here if you need a testing grounds.