Arm migrator with single output recipe

[stuarteberg]

Maybe the problems in #73 with osx_arm64 migration are somehow related to the fact that this recipe has multi-outputs, as introduced in #57 and #72.

Maybe there's some weird quirk of the ARM cross-compilation support that doesn't work properly for multi-output recipes? Let's find out.

As an experiment, this is a version of #73 that reverts to a single-output recipe.

Checklist

• [x] Used a personal fork of the feedstock to propose changes
• [x] Bumped the build number (if the version is unchanged)
• [x] Re-rendered with the latest conda-smithy (Use the phrase code>@<space/conda-forge-admin, please rerender in a comment in this PR for automated rerendering)

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but it appears we have a merge conflict. Please try to merge or rebase with the base branch to resolve this conflict.

Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

[stuarteberg]

@conda-forge-admin, please rerender

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

• The recipe must have some tests.
• The build section contained an unexpected subsection name. build is not a valid subsection name.
• The build section contained an unexpected subsection name. requirements is not a valid subsection name.
• The build section contained an unexpected subsection name. test is not a valid subsection name.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[stuarteberg]

@conda-forge-admin, please rerender

[stuarteberg]

This idea didn't work. Same error as in #73

ls: ${PREFIX}/lib/python3.8/site-packages/numpy: No such file or directory

[count0]

The build tag says “numpy1.19”, but the log says:

No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16

What does this “default” mean, and what is the consequence of this?

Also, does the path:

_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.8/site-packages

make any sense to you?

[stuarteberg]

What does this “default” mean, and what is the consequence of this?

This does look suspicious, but I think it's harmless. The numpy version should be pinned via the config file. And that file is provided via the -m argument, as seen in the log just above that warning:

+ /Users/runner/miniforge3/bin/conda mambabuild ./recipe -m ./.ci_support/osx_arm64_numpy1.19python3.8.____cpython.yaml --suppress-variables --no-test --clobber-file ./.ci_support/clobber_osx_arm64_numpy1.19python3.8.____cpython.yaml
Updating build index: /Users/runner/miniforge3/conda-bld

No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
Adding in variants from internal_defaults
Adding in variants from /Users/runner/work/1/s/.ci_support/osx_arm64_numpy1.19python3.8.____cpython.yaml
INFO:conda_build.variants:Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from /Users/runner/work/1/s/.ci_support/osx_arm64_numpy1.19python3.8.____cpython.yaml
Attempting to finalize metadata for graph-tool
INFO:conda_build.metadata:Attempting to finalize metadata for graph-tool

From the log, it appears that the code path which emits that warning is being hit multiple times. In the first pass, the config file hasn't actually been loaded yet. Then it says Adding in variants from ..., which should be the place where the pin is configured. I don't know why it has to take multiple passes like that; seems a bit awkward, especially since it generates spurious warnings.

[stuarteberg]

Also, does the path ... make any sense to you?

Yes... Are you just asking why it's such a ridiculously long directory name? Here's the explanation, as I understand it:

• Some software libraries embed the location of the user's build prefix within the final binary artifact(s) as a hard-coded string. In theory, that path could be important to the functioning of the library.
• But users are permitted to install their conda environments anywhere on their hard drive, not to a predefined location like /usr/lib or whatever.
• To allow that, conda post-processes the binary artifacts at install time. It overwrites any occurrence of the original build prefix with the user's local install prefix.
• That works as long as the user's install location is a shorter string than the original build prefix -- otherwise, the span of bytes it needs to replace won't fit within the space allotted for it within the binary.
• To ensure that there is always enough space in the binary to swap in an arbitrary install prefix, conda-build performs all of its builds within a directory that has a ridiculously long name.

I'm leaving out some details, e.g. how conda decides whether to use "text" mode replacement vs. "binary" mode replacement. Anyway, I don't think any of that applies to our build, since our .so artifacts use relative RPATH to avoid embedding a hard-coded prefix anyway.

[ndevenish]

I've never understood exactly where that numpy message comes from, but have seen it on all my conda-build builds also. I don't think it affects the actual output/numpy pinning that conda-forge does.

It sounds like you are running into the same issue I did. This:

ls: ${PREFIX}/lib/python3.8/site-packages/numpy: No such file or directory

is a familiar symptom. When using cross-python, it wipes the site-packages here and moves them all to the build prefix. I eventually solved this in my own feedstock with: https://github.com/conda-forge/dxtbx-feedstock/pull/10/commits/962fa0dd08f8db0053428993e59e8743bf7147ec

[ndevenish]

Here's the explanation, as I understand it:

And yes, this is exactly the reason it does this long prefix name. IIRC compilers and linkers do have ways to specifically pad it, but it's non-portable and has some gotchas, so it's more reliable to just use a really really long folder name.

It doesn't do this prefix_prefix on windows though, because it runs into directory length limits.

[count0]

It sounds like you are running into the same issue I did. This:

ls: ${PREFIX}/lib/python3.8/site-packages/numpy: No such file or directory

is a familiar symptom. When using cross-python, it wipes the site-packages here and moves them all to the build prefix. I eventually solved this in my own feedstock with: conda-forge/dxtbx-feedstock@962fa0d

Many thanks, this sounds promising!

[count0]

However, even if this fix works, I wonder if we will ever manage to build within the time limits, given that cross compilation is so much slower.

[stuarteberg]

@ndevenish Thanks again!

@count0 I will have to step away now. If this fix works here, we should probably apply it to #73 (after updating it to use graph-tool 2.45), rather than merging this PR.

[ndevenish]

Were these terminations from build timeouts or other reasons?

[stuarteberg]

Were these terminations from build timeouts or other reasons?

Other reasons. We did get the build to pass, but I screwed up the install location, so it's rebuilding one more time, in #73.