try add osx_arm64 build

[SrNetoChan]

Checklist

• [x] Used a personal fork of the feedstock to propose changes
• [x] Bumped the build number (if the version is unchanged)
• [ ] Reset the build number to 0 (if the version changed)
• [x] Re-rendered with the latest conda-smithy (Use the phrase code>@<space/conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• [ ] Ensured the license file is being packaged.

This is necessary for building QGIS in osx_arm64 and linux_aarch64 https://github.com/conda-forge/qgis-feedstock/pull/289

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[SrNetoChan]

@conda-forge-admin, please rerender

[SrNetoChan]

any ideas @gillins ?

[SrNetoChan]

It give and status 126 Error. Googling around it seems related to some execution permissions on the file Users/runner/miniforge3/conda-bld/qscintilla2_1672709146979/work/conda_build.sh

[gillins]

Thanks @SrNetoChan I think it might be trying to run Python but for the wrong CPU type when cross compiling. I think you might need these lines in the build section:

    - python                                # [build_platform != target_platform]
    - cross-python_{{ target_platform }}    # [build_platform != target_platform]

See https://conda-forge.org/blog/posts/2020-10-29-macos-arm64/#how-to-add-a-osx-arm64-build-to-a-feedstock and the changes I needed to get TuiView working: https://github.com/conda-forge/tuiview-feedstock/pull/51/files#diff-f3725a55bf339595bf865fec73bda8ac99f283b0810c205442021f29c06eea9a

[SrNetoChan]

It's complaining about sip build both in osx_arm and aarch64

[gillins]

Can you bring in the if [[ "${CONDA_BUILD_CROSS_COMPILATION:-}" == "1" ]]; then parts from https://github.com/conda-forge/pyqtwebkit-feedstock/blob/main/recipe/build.sh ? Hopefully that should fix it.

[SrNetoChan]

@conda-forge-admin, please rerender

[github-actions[bot]]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/qscintilla2-feedstock/actions/runs/3829049122.

[gillins]

@conda-forge-admin, please rerender

[github-actions[bot]]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/qscintilla2-feedstock/actions/runs/3833895480.

[gillins]

OK that didn't make much difference. I wonder if you could try syncing up this part of the build.sh: https://github.com/conda-forge/pyqtwebkit-feedstock/blob/main/recipe/build.sh#L9-L34 ? I'm not sure if it will help but at least we will be similar to something that works...

[SrNetoChan]

No change. I will revert that too. Meanwhile, I found this:

https://www.mail-archive.com/qscintilla@riverbankcomputing.com/msg01077.html

Exatly the same error.

[gillins]

@hmaarrfk do you have any thoughts on this one? We are kind of clutching at straws here...

[conda-forge-webservices[bot]]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[SrNetoChan]

The error both in arm or aarch is happens during Building Python bindings

The error is:

build.py: /home/conda/feedstock_root/build_artifacts/qscintilla2_1673030184046/work/Python/sip/qscimodcommon.sip: line 42: column 9: 'QtCore/QtCoremod.sip' could not be found
/home/conda/feedstock_root/build_artifacts/qscintilla2_1673030184046/work/Python/sip/qscimodcommon.sip: line 43: column 9: 'QtGui/QtGuimod.sip' could not be found
/home/conda/feedstock_root/build_artifacts/qscintilla2_1673030184046/work/Python/sip/qscimodcommon.sip: line 44: column 9: 'QtWidgets/QtWidgetsmod.sip' could not be found
/home/conda/feedstock_root/build_artifacts/qscintilla2_1673030184046/work/Python/sip/qscimodcommon.sip: line 46: column 6: 'PyQt_Printer' is not a known qualifier
/home/conda/feedstock_root/build_artifacts/qscintilla2_1673030184046/work/Python/sip/qsciprinter.sip: line 21: column 6: 'PyQt_Printer' is not a known qualifier

Googling for the error I found some references [1][2] to exactly the same error while building Qscintilla2, but no solution, only the hint that this kind of error happens when the builder is using the wrong python/sip within the system. In other cases it's suggested to install also the pyqt5-dev tools

https://stackoverflow.com/questions/18922009/pyqt-unable-to-find-qtcoremod-sip https://www.mail-archive.com/qscintilla@riverbankcomputing.com/msg01077.html https://github.com/qgis/QGIS/issues/19466

@conda-forge/arm-arch team any ideas?

[hmaarrfk]

@conda-forge-admin please rerender

basically cross compilation is just hard with qt.... so I avoid it as much as I can.

[hmaarrfk]

support is very limited for cross compilation in 5.X from qt.

[gillins]

Thanks @hmaarrfk, very interesting. So I guess osx_arm64 is likely to be out of reach for now... Our original goal was to build qgis which takes about 4.5 hours ATM so we were hoping cross compiling would be possible. I think it also uses sip.

[SrNetoChan]

I don't have an Mac M1 but maybe we could find someone with one to try building it locally?

[SrNetoChan]

What would I need to run? Something like this?

conda mambabuild ./recipe -m ./.ci_support/osx_arm64_python3.9.____cpython.yaml

[SrNetoChan]

@gillins I got access to a MacOs M1 machine and built this feedstock there. The files are here:

https://anaconda.org/afneto/qscintilla2/files

If we get someone to upload these and merge the https://github.com/conda-forge/qjson-feedstock/pull/15 we can try to build QGIS again.

[hmaarrfk]

cross compiling works a little better for osx-arch than it does for linux aarch. I wouldn't give up just yet. maybe @jschueller has better insights.

[SrNetoChan]

It's interesting that the error was exactly the same in aarch and arm.

Also, it was nice to see that the recipe worked fine building locally on a mac m1.

We will wait for your insites

[gillins]

Yep would be good to try a few more things and see if we can get it going. Building locally each time there is an update required is going to be a pain.... esp. for qgis.

[hmaarrfk]

@pkgw do you think they might be hitting the same bug you addressed for cross python:

2023-01-11T16:46:40.6713780Z + sip-build --no-make --qsci-features-dir ../src/features --qsci-include-dir ../src --qsci-library-dir ../src --api-dir /Users/runner/miniforge3/conda-bld/qscintilla2_1673454887107/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/qsci/api/python
2023-01-11T16:46:40.6883700Z /Users/runner/miniforge3/conda-bld/qscintilla2_1673454887107/_build_env/bin/sip-build: line 4: import: command not found
2023-01-11T16:46:40.6898830Z /Users/runner/miniforge3/conda-bld/qscintilla2_1673454887107/_build_env/bin/sip-build: line 5: import: command not found
2023-01-11T16:46:40.7054270Z from: can't read /var/mail/sipbuild.tools.build
2023-01-11T16:46:41.4955060Z /Users/runner/miniforge3/conda-bld/qscintilla2_1673454887107/_build_env/bin/sip-build: line 10: syntax error near unexpected token `('
2023-01-11T16:46:41.4956860Z /Users/runner/miniforge3/conda-bld/qscintilla2_1673454887107/_build_env/bin/sip-build: line 10: `    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])'

[hmaarrfk]

FYI, it might be related to: https://github.com/conda-forge/cross-python-feedstock/pull/65

[SrNetoChan]

@hmaarrfk should I revert the last commit? It seems that the current error is fixed by it, although we will hit the next one.

[hmaarrfk]

I think you can revert, and maybe add a reference to the issue I linked to. it shouldn't be necessary, but you did well in forcing the correct flags.

[hmaarrfk]

I'm really not sure what the best way forward for you all is.

1. Invoking cfep03 https://github.com/conda-forge/cfep/blob/main/cfep-03.md isn't the "easiest" thing. It also isn't meant for cases like this where we are simply having trouble cross compiling.
2. Wait to get help from somebody more knowledgable.

I'm sorry i can't help anymore.

[SrNetoChan]

Interesting that the problem was exactly the same both in aarch64 and arm64. I would say it's needed some patch on sip builder configure.py.

[SrNetoChan]

Since the error in cross compiling for OSX arm64 was the same as when trying cross compilation on linux, I was trying to build aarch64 locally on my kubuntu 22.04 machine. But I was stopped even before that error appears. I am getting this:

++ qmake -v
++ sed -n 's/.*Qt version \([0-9])*\).*/\1/p'
/home/aneto/mambaforge/envs/debug_qscintilla2/conda-bld/qscintilla2_1673738274569/work/conda_build.sh: line 14: /home/aneto/mambaforge/envs/debug_qscintilla2/conda-bld/qscintilla2_1673738274569/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin/qmake: cannot execute binary file: Exec format error
+ QT_MAJOR_VER=
Traceback (most recent call last):
  File "/home/aneto/mambaforge/envs/debug_qscintilla2/bin/conda-mambabuild", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/aneto/mambaforge/envs/debug_qscintilla2/lib/python3.11/site-packages/boa/cli/mambabuild.py", line 256, in main
    call_conda_build(action, config)
  File "/home/aneto/mambaforge/envs/debug_qscintilla2/lib/python3.11/site-packages/boa/cli/mambabuild.py", line 228, in call_conda_build
    result = api.build(
             ^^^^^^^^^^
  File "/home/aneto/mambaforge/envs/debug_qscintilla2/lib/python3.11/site-packages/conda_build/api.py", line 180, in build
    return build_tree(
           ^^^^^^^^^^^
  File "/home/aneto/mambaforge/envs/debug_qscintilla2/lib/python3.11/site-packages/conda_build/build.py", line 3097, in build_tree
    packages_from_this = build(metadata, stats,
                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aneto/mambaforge/envs/debug_qscintilla2/lib/python3.11/site-packages/conda_build/build.py", line 2209, in build
    utils.check_call_env(cmd, env=env, rewrite_stdout_env=rewrite_env,
  File "/home/aneto/mambaforge/envs/debug_qscintilla2/lib/python3.11/site-packages/conda_build/utils.py", line 402, in check_call_env
    return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aneto/mambaforge/envs/debug_qscintilla2/lib/python3.11/site-packages/conda_build/utils.py", line 382, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-o', 'errexit', '/home/aneto/mambaforge/envs/debug_qscintilla2/conda-bld/qscintilla2_1673738274569/work/conda_build.sh']' returned non-zero exit status 126.

Seems that bash is not being able to run qmake because it does not consider qmake to have execution permissions.

I am running the following command inside a pre created-environment:

conda mambabuild ./recipe -m ./.ci_support/linux_aarch64_python3.8.____cpython.yaml

[gillins]

I think this is because we are building aarch64 natively now so it is probably trying to run arm binaries on your x86 machine... I wonder if you set aarch64 back to azure and re-render it might work better. Also I think you can just run the feedstock script build_locally.py. @hmaarrfk will know more.

[SrNetoChan]

Ah! Always learning. Makes sense. I will try it tomorrow (monday).

[SrNetoChan]

@gillins @hmaarrfk still the same error or similar after changing to cross compiling and rerendering. Seems I am failing to create the exact same scenario as in azure. :-(

[hmaarrfk]

still the same error or similar after changing to cross compiling and rerendering. Seems I am failing to create the exact same scenario as in azure. :-(

i don't understand. what are you trying to compile? Linux, OSX? can you provide the exact commands you are running.

[SrNetoChan]

still the same error or similar after changing to cross compiling and rerendering. Seems I am failing to create the exact same scenario as in azure. :-(

i don't understand. what are you trying to compile? Linux, OSX? can you provide the exact commands you are running.

I run a linux machine. Since the error currently in osx arm64 was exactly the same in linux aarch64 before we buildi it natively. I was trying to going back to cross'compiling on my local machine to try figure out waht may be hapening on the OSX front.

I also have access to a osx arm computer, where I was able to build osx arm64 without problems, but I think it ended up building natevely, so I donºt know if I can replicate the environment of azure osx

[hmaarrfk]

@conda-forge-admin please rerender

[hmaarrfk]

Can you try now?

If you can generate the log

python build-locally.py 2>&1 | tee log.txt

that would be helpful

[hmaarrfk]

you can also try:

conda debug -m .ci_support/THE_BUILD_YOUWANT recipe

[SrNetoChan]

This was what I got with

python build-locally.py 2>&1 | tee log.txt

log.txt

[hmaarrfk]

On linux, because we can, we typically build with "emulation" which means that the binaries in the PREFIX environment are also available to the build system.

It is likely trying to run the aarch executable.

On ubuntu, I think you need to install qemu-user-static. It might be a similar package on other platforms.

[SrNetoChan]

So I should install that on my machine? Using apt for example?

[hmaarrfk]

yes, something like

sudo apt install qemu-user-static

[hmaarrfk]

basically, docker looks for those files, then uses those files to emulate "arch" when it detects binaries with that format.

[SrNetoChan]

sudo apt install qemu-user-static

@hmaarrfk great that did the difference, now I am in the same situation as azure.. Let's see if I can leverage that to figure out what might be wrong.

[SrNetoChan]

So what I have found so far. I run the debug session and started the build_env that takes me to this folder:

/home/aneto/mambaforge/conda-bld/debug_1674134804089/work

with the following active environemnt:

/home/aneto/mambaforge/conda-bld/debug_1674134804089/_build_env

I tried to run the conda_build.sh file and got the exact same error as before. Something like:

column 9: 'QtCore/QtCoremod.sip' could not be found
column 9: 'QtGui/QtGuimod.sip' could not be found
column 9:  'QtWidgets/QtWidgetsmod.sip' could not be found
column 6: 'PyQt_Printer' is not a known qualifier
column 6: 'PyQt_Printer' is not a known qualifier

These are all import in the following file:

/home/aneto/mambaforge/conda-bld/debug_1674134804089/work/Python/sip/qscimodcommon.sip

I have tried to find there QtCore/QtCoremod.sip files and the are all in this folder:

/home/aneto//mambaforge/conda-bld/debug_1674134804089/_build_env/lib/python3.10/site-packages/PyQt5/bindings/QtCore

Nevertheless, when running conda_build.sh the folder set for SIP_DIR is that one, but

SIP_DIR=/home/aneto/mambaforge/conda-bld/debug_1674134804089/_h_env/lib/python3.10/site-packages/PyQt5/bindings

which does not exist... actually the _h_env/lib/python3.10/site-packages/ folder is completely empty

notice the _h_env instead of _build_env

Are these the host environment and the build environment? So, I wonder if we need to install something extra in the host if the SIP_DIR path is wrong.

[hmaarrfk]

Maybe @conda-forge/pyqt can help

[SrNetoChan]

@conda-forge-admin, please rerender

[SrNetoChan]

Not pretty, but it workd locally for linux, let's see if it does the trick for OSX arm too.