search

conda-tools / conda-build-all

BSD 3-Clause "New" or "Revised" License
30 stars 24 forks source link

Fails with conda-build 3.0.3 #89

Closed rainwoodman closed 6 years ago

rainwoodman commented 7 years ago

conda-build-all version 1.0.4 fails with conda-build 3.0.3

conda build-all recipes --matrix-max-n-minor-versions 3 --matrix-conditions "python $PYTHON_BUILD_RESTRICTIONS" "$NUMPY_BUILD_RESTRICTION" 
Fetching package metadata .........
Traceback (most recent call last):
  File "/home/yfeng1/anaconda3/install/bin/conda-build-all", line 6, in <module>
    sys.exit(conda_build_all.cli.main())
  File "/home/yfeng1/anaconda3/install/lib/python3.5/site-packages/conda_build_all/cli.py", line 90, in main
    b.main()
  File "/home/yfeng1/anaconda3/install/lib/python3.5/site-packages/conda_build_all/builder.py", line 249, in main
    if build_config.CONDA_NPY is None:
AttributeError: 'Config' object has no attribute 'CONDA_NPY'
jakirkham commented 7 years ago

Thanks for the info. Though TBH am not surprised that conda-build 3 is incompatible with conda-build-all. Would recommend sticking to conda-build 2 when using conda-build-all.

At this time, it is unclear to me to what extent conda-build 3 has replaced conda-build-all and whether it makes sense to update conda-build-all to be compatible. Would be interested to hear other people's thoughts on this though.

rainwoodman commented 7 years ago

Oops. Conda-build 3 has a lot of new stuff.

The only documentation that is closest to a migration guide is the blog on continuum.io, at https://www.continuum.io/blog/developer-blog/package-better-conda-build-3

The essence of conda-build-all is to incrementally build/rebuild a single recipe template for a list of requirement-hashes. The 'variants' feature in conda-build might be useful for simplifying conda-build-all's code base:

jakirkham commented 7 years ago

Meant to include you on this some time ago, @msarahan, and must have forgotten. Reminded by our discussion today about conda-build 3. Admittedly we are doing stuff in conda-build-all that uses non-API stuff from conda-build. So no surprises things don't work without some changes. It will be up to us maintaining/using conda-build-all to fix this in some sensible way obviously. Though any tips you might have would certainly be appreciated.

msarahan commented 7 years ago

everything becomes a variant in cb3. Although you can pass in environment variables like you could for cb2, you can't access then in the same way. All the loaded variants from any source are available as the metadata_object.config.variant dictionary, which is one element of the expanded matrix present in metadata_object.variants

so here rather than config.CONDA_NPY, you'd have metadata_object.config.variant['numpy']

and note that the config is not global - it is a per-metadata object thing.

msarahan commented 7 years ago

It's very easy to add properties to a given config instance with CONDA_NPY just returning the value of its self.variant.get('numpy'). I'll try to do that, but PRs also welcome. That might be a good way to not require conda-build-all modification for at least a little while longer.

rainwoodman commented 7 years ago

I'd mention that we do want to build packages pinning to a version of mpich/openmpi with conda-build-all. The use case is to have dummy packages for cray-mpi and sgi-mpt, thus ship pre-compiled binaries for cray/sgi computers together with regular mpich / openmpi based binaries. Does this require cb3?

msarahan commented 7 years ago

I don't think it's ever accurate to say that something requires cb3. If you don't use it, though, you'll need to have many more recipes, or some alternate scheme for handling the variants. It's not that cb3 enables unique package builds so much as it makes them much easier.

rainwoodman commented 7 years ago

With the new cross-compilation system it becomes urgent for us to either find a way to get conda-build-all working with conda-build, or come up with some bash voodoo to achieve the same functionality. Which routine shall I take?

msarahan commented 7 years ago

Why don't we get cb3 to recognize the CONDA_* env vars, so that hopefully cb3 "just works" with c-b-a? If you have time for a PR, that would be great. Otherwise I can try to get to it soon.

rainwoodman commented 7 years ago

It's the best if you can do a PR -- such that I follow and have a better grab of where things are.

On Tue, Oct 17, 2017 at 6:26 AM, Mike Sarahan notifications@github.com wrote:

Why don't we get cb3 to recognize the CONDA_* env vars, so that hopefully cb3 "just works" with c-b-a? If you have time for a PR, that would be great. Otherwise I can try to get to it soon.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/conda-tools/conda-build-all/issues/89#issuecomment-337230577, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIbTOalyGm-MjNlLkEKbMEIb22JWwJ5ks5stKsHgaJpZM4ORpW3 .