conda-build recursive dependency resolution still fails

[jhunkeler]

Most of the problems are fixed regarding dependency resolution in conda-4.0.x and conda-build-1.20.0, however it seems the issue ( #807 ) still effects building metapackages:

$ conda build --numpy=1.10 --python=3.5 stsci-hst | grep Missing
Missing dependency acstools, but found recipe directory, so building acstools first
Missing dependency astrolib.coords, but found recipe directory, so building astrolib.coords first
Missing dependency calcos, but found recipe directory, so building calcos first
Missing dependency costools, but found recipe directory, so building costools first
Missing dependency crds, but found recipe directory, so building crds first
Missing dependency drizzlepac, but found recipe directory, so building drizzlepac first
Missing dependency fitsblender, but found recipe directory, so building fitsblender first
Missing dependency hstcal, but found recipe directory, so building hstcal first
Missing dependency nictools, but found recipe directory, so building nictools first
Missing dependency pyregion, but found recipe directory, so building pyregion first
Missing dependency pysynphot, but found recipe directory, so building pysynphot first
Missing dependency pywcs, but found recipe directory, so building pywcs first
Missing dependency reftools, but found recipe directory, so building reftools first
Missing dependency stistools, but found recipe directory, so building stistools first
Missing dependency stsci.convolve, but found recipe directory, so building stsci.convolve first
Missing dependency stsci.image, but found recipe directory, so building stsci.image first
Missing dependency stsci.imagemanip, but found recipe directory, so building stsci.imagemanip first
Missing dependency stsci.imagestats, but found recipe directory, so building stsci.imagestats first
Missing dependency stsci.ndimage, but found recipe directory, so building stsci.ndimage first
Missing dependency stsci.numdisplay, but found recipe directory, so building stsci.numdisplay first
Missing dependency stsci.sphinxext, but found recipe directory, so building stsci.sphinxext first
Missing dependency stsci.stimage, but found recipe directory, so building stsci.stimage first
Missing dependency stsci.skypac, but found recipe directory, so building stsci.skypac first
Missing dependency stsci.sphere, but found recipe directory, so building stsci.sphere first
Missing dependency stwcs, but found recipe directory, so building stwcs first
Missing dependency wfpc2tools, but found recipe directory, so building wfpc2tools first
Missing dependency wfc3tools, but found recipe directory, so building wfc3tools first
 Package missing in current linux-64 channels:
  - stsci.imagestats

The package discovers itself as a dependency and then skips along on to the next recipe in the list:

Missing dependency acstools, but found recipe directory, so building acstools first

The metapackage builds perfectly fine under conda-3.19.1 with conda-build-1.18.1 (prior to the solver changes introduced in v4.0.0). Strangely enough, the stsci-data-analysis recipe builds successfully under the latest release of conda-build... so it doesn't effect every metapackage, but only ones with large dependency chains.

Here's a link to the recipe: https://github.com/astroconda/astroconda-contrib/blob/master/stsci-hst/meta.yaml

Any ideas?

[jakirkham]

Did you try conda-build 1.19.2, as well?

[jhunkeler]

Here's the full output from 1.19.2: http://pastebin.com/ggqZg2PB

^ Managed to build one package before going haywire

And 1.20.0: http://pastebin.com/jyCZF3Di

^ Fails instantly

[jhunkeler]

Taking the idea from #864 -- If I move my metapackage recipes into a sub-directory and call conda build --numpy=1.10 metapackages/stsci-hst, the dependencies are resolved correctly.

Interesting.

[msarahan]

so you move your recipe into the same folder as your metapackage? I don't know what the exact rule is, but conda build's search path is extremely limited. If you want more of a search path, please propose the behavior you would like, in terms of:

• is there a recipes environment variable or condarc setting, for folders that should always be searched?
• how are duplicates handled? First one found is the winner? Do versions come into play?
• Should directories be searched recursively?
• should you be able to specify other directories to search either on the command line or in some config file?

We have operated primarily on flat hierachies up to now, so we haven't explored this space much. If you have concrete workflows or use cases, that will help define what this feature might need to be, and how much effort it might take.

[jhunkeler]

Sorry, I was merely trying something different out of morbid curiosity, because this problem is driving me nuts. That's not how I normally build my packages. This is my workflow...

Directory structure:

|- top-level
\- recipe1
   | - build.sh
   | - bld.bat
   | - meta.yaml
\- recipe2
   | - build.sh
   | - bld.bat
   | - meta.yaml
\- recipe[...]
   | - [...]
\- metapackage1
   | - meta.yaml # Pretend it provides/depends ~30 recipes in the working directory
\- metapackage2
   | -  meta.yaml # Pretend it provides/depends ~10 recipes in the working directory
\- metapackage[...]
   | - [...]

My usage:

cd top-level
conda build --python=3.5 --numpy=1.10 metapackage1
conda build --python=3.5 --numpy=1.10 metapackage2

The desired result: The recipes listed under requirements/build are scanned and resolved automatically, built in the correct order, and metapackage1-1.0.0-np110py35_0 comes out the other end of the pipe.

The current result: The recipes listed under requirements/build are partially resolved, and if conda-build decides to build anything, its in no particular order, and then a build failure occurs.

Why?

For me it feels more reasonable to define a set of recipes in a metapackage and build everything in one place, with one command. Since this stopped working for me after the release of conda 4.0.0, I've been building recipes in a very strict order (i.e. listed in a text file, built individually by a bash script) because the dependency resolution is very broken. My metapackages are not only the type of recipe effected by this, either. Even seemingly benign recipes with maybe two additional build dependencies other than setuptools or python will fail to resolve in the same way. The solver simply cannot find the dependencies even when they are in the current working directory (staring it right in the face).

The interesting part

To clarify the sub-directory test I performed the other day.... The solver tends to find the dependencies successfully if the directory structure looks more like this:

|- top-level
\- recipe1
   | - build.sh
   | - bld.bat
   | - meta.yaml
\- recipe2
   | - build.sh
   | - bld.bat
   | - meta.yaml
\- recipe[...]
   | - [...]
\- metapackages
   \- metapackage1
      | - meta.yaml # Pretend it provides/depends ~30 recipes in the working directory
   \- metapackage2
      | -  meta.yaml # Pretend it provides/depends ~10 recipes in the working directory
   \- metapackage[...]
      | - [...]

And conda-build is invoked as follows:

cd top-level
conda build --python=3.5 --numpy=1.10 metapackages/metapackage1
conda build --python=3.5 --numpy=1.10 metapackages/metapackage2

This bizarre behavior leads me to believe the solver is looking one level ABOVE the current working directory for recipes to resolve. In the case of the metapackage being at the same level as the rest of the recipes, the solver clearly misses them even though they're right there in the same directory.

So perhaps the solver prior to 4.0.0 started looking for recipes relative to the initial meta.yaml it was scanning. Perhaps something like recipes_live_here = join(abspath('meta.yaml'), abspath('..')). In the post-4.0.0 solver it feels like its searching relative to the recipe's directory instead: recipes_live_here = join(abspath('.'), abspath('..')). Substitute the '.' here for the path to the recipe being built.

Pre-4.0.0 resolver path: conda build metapackage1 == /path/to/top-level Post-4.0.0 resolver path: conda build metapackage1 == /path/to Post-4.0.0 resolver path, in subdir: conda build metapackages/metapackage1 == /path/to/top-level

Maybe I'm way off, but either way, I'd really like to see this start working again. You can easily test the desired behavior by installing conda v3.19.1 and conda-build v1.18.1 then building a metapackage with a bunch of build requirements defined. It will cycle through all of the unbuilt build dependencies, compile them, and produce a usable metapackage.

[jhunkeler]

1.20.1 seems to fix the dependency resolving issues, but now the shebang bug (#889) renders the latest release unusable. :(

[jakirkham]

...the shebang bug (#889) renders the latest release unusable.

That should be fixed in this PR ( https://github.com/conda/conda-build/pull/892 ), but I haven't had time to test. If you would like to give it a go, @jhunkeler, I would certainly appreciate your eyes on this.

[jhunkeler]

OK, I tested the current master branch and the shebang problem is fixed. I spoke too soon regarding the dependency resolver, but now things are a bit more clear.

Nesting Map:

---------------
| metapackage |
---------------
       |
 < package1 > - Requires package2, package3, package4
 < package2 > - Requires package3, package5
 < package3 > - Requires package4
 < package4 > - Requires package5
 < package5 > - No requirements

It should be doing this:

1. Parse metapackage build requirements
2. If build requirement is missing; begin building first unbuilt requirement
3. Parse build requirements of the current unbuilt requirement [loop step 2]
4. No more unbuilt requirements, continue building metapackage

What it does:

1. Parse metapackage build requirements
2. If build requirement is missing; begin building first unbuilt requirement
3. Parse build requirements of the currrent unbuilt requirement [loop step 2; stop on error]
4. No more unbuilt requirements, continue building metapackage [unlikely]

The problem here is that it will build unbuilt dependencies as long no other nested dependencies exist in the current unbuilt dependency. So if package1 requires package2, and package2 doesn't not require anything else (just python, for example) the dependency will be built and package1 will continue building. On the other hand, if package2 requires package3, and package3 requires package4, and package4 requires package5, then everything explodes.

The error output for this scenario might look like this:

% conda build --python=3.5 --numpy=1.10 drizzlepac
Removing old build environment
Removing old work directory
BUILD START: drizzlepac-2.1.3-np110py35_0
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
Solving package specifications: .
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency fitsblender, but found recipe directory, so building fitsblender first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency nictools, but found recipe directory, so building nictools first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stsci.convolve, but found recipe directory, so building stsci.convolve first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stsci.distutils, but found recipe directory, so building stsci.distutils first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stsci.image, but found recipe directory, so building stsci.image first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stsci.imagemanip, but found recipe directory, so building stsci.imagemanip first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stsci.imagestats, but found recipe directory, so building stsci.imagestats first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stsci.ndimage, but found recipe directory, so building stsci.ndimage first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stsci.skypac, but found recipe directory, so building stsci.skypac first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stsci.sphere, but found recipe directory, so building stsci.sphere first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stsci.stimage, but found recipe directory, so building stsci.stimage first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency stwcs, but found recipe directory, so building stwcs first
 Packages missing in current osx-64 channels:
  - fitsblender
  - nictools
  - stsci.convolve
  - stsci.distutils
  - stsci.image
  - stsci.imagemanip
  - stsci.imagestats
  - stsci.ndimage
  - stsci.skypac
  - stsci.sphere
  - stsci.stimage
  - stwcs
  - pyregion
Missing dependency pyregion, but found recipe directory, so building pyregion first
Removing old build environment
Removing old work directory
BUILD START: fitsblender-0.2.6-np110py35_0
Fetching package metadata: ......
Solving package specifications: .
 Packages missing in current osx-64 channels:
  - d2to1
  - stsci.distutils
  - stsci.tools
Missing dependency d2to1, but found recipe directory, so building d2to1 first
 Packages missing in current osx-64 channels:
  - d2to1
  - stsci.distutils
  - stsci.tools

So let's forget all of that crap for a second and look at what it's doing at the core:

% conda build --python=3.5 --numpy=1.10 drizzlepac | grep BUILD
Using Anaconda Cloud api site https://api.anaconda.org
BUILD START: drizzlepac-2.1.3-np110py35_0
BUILD START: fitsblender-0.2.6-np110py35_0
 Packages missing in current osx-64 channels:
  - d2to1
  - stsci.distutils
  - stsci.tools

Conda-build parses the recipe, starts building drizzlepac, but immediately realizes fitsblender doesn't exist (the first dependency listed under requirements/build), so it goes to build it. The fitsblender recipes too has its own build requirements that don't exist yet, d2to1, stsci.distutils, and stsci.tools.

What I'm expecting to see here is conda-build going: "d2to1 doesn't exist yet, let's build that." Instead conda-build is going, "Well, d2to1 doesn't -- what was I doing? Oh well."

Now if I manually build fitsblender I get this:

jhunk@makeitso:~/Downloads/astroconda-contrib% conda build --python=3.5 --numpy=1.10 fitsblender 2>err | grep BUILD
BUILD START: fitsblender-0.2.6-np110py35_0
BUILD START: d2to1-0.2.12-py35_0
BUILD END: d2to1-0.2.12-py35_0
BUILD START: stsci.distutils-0.3.8-np110py35_0
BUILD END: stsci.distutils-0.3.8-np110py35_0
BUILD START: stsci.tools-3.4.1-np110py35_0
BUILD START: pytools-2016.1-py35_0
BUILD START: appdirs-1.4.0-py35_0
BUILD END: appdirs-1.4.0-py35_0
BUILD START: pytools-2016.1-py35_0
BUILD END: pytools-2016.1-py35_0
BUILD START: stsci.tools-3.4.1-np110py35_0
BUILD END: stsci.tools-3.4.1-np110py35_0
BUILD START: fitsblender-0.2.6-np110py35_0
BUILD END: fitsblender-0.2.6-np110py35_0

Breaking it down:

• d2to1 doesn't have extra requirements
• stsci.distutils requires d2to1, nothing else
• stsci.tools requires pytools and appdirs (single depth - neither require anything extra, so both build)
• pytools doesn't have extra requirements
• appdirs doesn't have extra requirements

So what I'm seeing here is, if the depth of the unbuilt dependency is greater than one (maybe two?) the build stops.

Depth breakdown:

• 0: drizzlepac
• 1: fitsblender
• 2: d2to1 [ <<< oops]
• 2: stsci.distutils
• 2: stsci.tools
• 3: pytools
• 3: appdirs

For fitsblender it looks like:

• 0: fitsblender
• 1: d2to1
• 1: stsci.distutils
• 1: stsci.tools
• 2: pytools [<<< doesn't require anything else, so no oops here]
• 2: appdirs

I apologize for being so incredibly obnoxious about this, but the behavior doesn't seem right to me. I'm not sure if I'm going crazy here... Has no one ever built nested dependencies with conda-build before? This type of failure does not happen if the dependencies already exist locally (or remotely using -c [URL]), so vanilla folks building against Continuum's native packages won't see this, but for those of us that have pretty massive toolchains that don't exist yet in some upstream repository we're (or I am) totally dead in the water.

I'm tired of using conda-build v1.18.1... Its starting to show its age... I really want to use the latest and greatest release.

[jhunkeler]

Just to confirm, the master branch currently fixes #889

[jhunkeler]

Update: Issue persists into conda-4.0.6 and conda-build-1.20.3

[msarahan]

I think we have a test for this at https://github.com/conda/conda-build/tree/master/tests/test-recipes/metadata/recursive-build-two-layers and https://github.com/conda/conda-build/tree/master/tests/test-recipes/metadata/recursive-build-two-packages

Can you take a look at those tests and see if we can improve them to cover your case?

[pelson]

Sorry if you already know about it, but you may also be interested in looking at conda-build-all, a tool which I developed to handle recursive dependency building as well as automating the build matrix. conda-build-all is available from the conda-forge channel if you want to try it out (but I strongly encourage you to help conda-build test your usecase first).

[jhunkeler]

@msarahan If I modify those tests to be more like my situation it becomes clear that 4.0.0's solver can't handle circular dependencies. If you're not going to support circular dependencies at all, can the team at least implement better detection so the end-user can fix the problem?

FATAL: Circular dependency detected in [recipe]:
    [display dependency map with failure points highlighted]

Or something?

[msarahan]

@jhunkeler - are you talking about the exact 4.0.0 revision, or 4.0.x in general? If 4.0.x, @mcg1969 has indicated to me that it should not have problems with circular dependencies. If it does, this might be a conda bug, or a bug in how conda-build uses conda.

[mcg1969]

To the best of my knowledge, the mechanism conda-build uses to traverse the package build list is not the same as the mechanism conda uses to resolve dependencies. I'm happy to test any conda dependency problems we're seeing here, but my cursory understanding of how conda-build works suggests it's not the issue.

[msarahan]

Ah, I see, thanks @mcg1969 - the issue is not in getting build dependencies, but in ordering builds themselves.

Is there a way for us to use conda's solver for this job? Ryan used NetworkX to do something similar, I think, but it would be good to avoid a dependency.

[mcg1969]

I don't think the SAT solver can help, no. I'm really not sure what the best way is to handle circular dependencies in conda-build itself.

I'm not sure there should be a way. After all, if package A is required to build package B, and package B is required to build package A, and neither package yet has a pre-existing build, shouldn't that be intractable?

The only way to bootstrap this that I can see is:

• disallow circular dependencies in build requirements
• allow circular dependencies in run requirements; however
• delay the execution of run tests when building a cycle of packages, until all of the packages in the cycle are built.

[jhunkeler]

@msarahan Sorry I meant conda 4.0.x and conda-build >1.19.x in general.

@mcg1969 I'm all for disabling circular dependencies in builds requirements, so long as there's a clear error message to indicate why the build failed rather than an ambiguous, "Missing dep A found dep A building dep A -- Oops -- Missing dep B found dep B building dep B -- Oops" and so on.

From the sound of things I've been relying on a bug in 1.18.1 to build my packages. Is that safe to assume here?

[msarahan]

I think it's safe to assume you're correct here, but the outstanding issue is that conda-build needs to be better about build ordering. Rather than haphazardly trying one package at a time, it seems that it needs to compute an ordered build list more intelligently.

If you can pinpoint what changed between 1.18.1 and current behavior, I'll try to revert/restore it. Otherwise, I'll try to work on integrating the NetworkX work I mentioned before, present in https://github.com/ContinuumIO/ProtoCI/blob/master/protoci/build2.py

[jakirkham]

Maybe looking at conda-build-all is worthwhile.

[jhunkeler]

If/when conda-build is fixed I'll try conda-build-all again. The last time I tried conda-build-all the results were the same as conda-build.

[pelson]

Is there a way for us to use conda's solver for this job?

conda-build needs to be better about build ordering

Ugly but effective: https://github.com/SciTools/conda-build-all/blob/master/conda_build_all/order_deps.py#L4

This is what powers conda-forge/staged-recipes (directly), and conda-smithy (indirectly).

If/when conda-build is fixed I'll try conda-build-all again. The last time I tried conda-build-all the results were the same as conda-build.

I'm afraid you may be mistaken - conda-build-all should always have been able to deal with these dependencies, and doesn't go near the try..except that conda-build used to take to resolve build order. Could you double check conda-build-all? It should also raise an exception with circular dependencies. Feel free to raise an issue there if you encounter any resolution issues.

Thanks,

[jhunkeler]

Resolved after upgrading >=2.0.6 (maybe earlier)

I'm happy to finally be using the latest version. Closing this out.

[msarahan]

Excellent, glad things are finally getting worked out.

On Nov 1, 2016 16:40, "Joseph Hunkeler" notifications@github.com wrote:

Closed #849 https://github.com/conda/conda-build/issues/849.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/conda/conda-build/issues/849#event-844070564, or mute the thread https://github.com/notifications/unsubscribe-auth/AACV-dxK30zZe3InfHACKF6bA9NBaVAeks5q57HUgaJpZM4H494p .

[github-actions[bot]]

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!