conda / conda-build

Commands and tools for building conda packages
https://docs.conda.io/projects/conda-build/
Other
384 stars 424 forks source link

ValueError: Incompatible component merge: mpi_openmpi_* #4498

Open pawelqs opened 2 years ago

pawelqs commented 2 years ago

Actual Behavior

Hi, I'm trying to build R package which I would like to upload to anaconda. I am following the instructions from here: https://stackoverflow.com/questions/52061664/install-r-package-from-github-using-conda

conda skeleton cran https://github.com/pawel125/clonalityParsers works fine, although I have to correct the names of two packages: r-genomeinfodb -> bioconductor-genomeinfodb and r-plyranges -> bioconductor-plyranges

Unfortunatelly conda build --R=4.2.0 r-clonalityparsers results in the following error:

conda build --R=4.2.0 r-clonalityparsers                                                         ─╯
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from config.variant
INFO:conda_build.variants:Adding in variants from config.variant
Attempting to finalize metadata for r-clonalityparsers
INFO:conda_build.metadata:Attempting to finalize metadata for r-clonalityparsers
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): repo.anaconda.com:443
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/linux-64/repodata.json HTTP/1.1" 304 0
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/noarch/repodata.json HTTP/1.1" 304 0
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/r/linux-64/repodata.json HTTP/1.1" 304 0
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/r/noarch/repodata.json HTTP/1.1" 304 0
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): conda.anaconda.org:80
DEBUG:urllib3.connectionpool:http://conda.anaconda.org:80 "GET /dranew/linux-64/repodata.json HTTP/1.1" 301 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): conda.anaconda.org:443
DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /dranew/linux-64/repodata.json HTTP/1.1" 304 0
DEBUG:urllib3.connectionpool:http://conda.anaconda.org:80 "GET /dranew/noarch/repodata.json HTTP/1.1" 301 None
DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /dranew/noarch/repodata.json HTTP/1.1" 304 0
DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /t/kp-055f5a19-484f-4c40-8c02-5c8796609036/conda-forge/linux-64 HTTP/1.1" 302 None
DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/linux-64/ HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/r/channeldata.json HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/channeldata.json HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:http://conda.anaconda.org:80 "GET /dranew/channeldata.json HTTP/1.1" 301 None
DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /dranew/channeldata.json HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /t/kp-055f5a19-484f-4c40-8c02-5c8796609036/bioconda/linux-64 HTTP/1.1" 302 None
DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /bioconda/linux-64/ HTTP/1.1" 200 None
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
Traceback (most recent call last):
  File "/home/pkus/programs/mambaforge/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/cli/main_build.py", line 488, in main
    execute(sys.argv[1:])
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/cli/main_build.py", line 477, in execute
    outputs = api.build(args.recipe, post=args.post, test_run_post=args.test_run_post,
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/api.py", line 186, in build
    return build_tree(
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/build.py", line 3088, in build_tree
    packages_from_this = build(metadata, stats,
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/build.py", line 2051, in build
    output_metas = expand_outputs([(m, need_source_download, need_reparse_in_env)])
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/render.py", line 788, in expand_outputs
    for (output_dict, m) in deepcopy(_m).get_output_metadata_set(permit_unsatisfiable_variants=False):
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/metadata.py", line 2121, in get_output_metadata_set
    conda_packages = finalize_outputs_pass(ref_metadata, conda_packages, pass_no=0,
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/metadata.py", line 782, in finalize_outputs_pass
    fm = finalize_metadata(om, parent_metadata=parent_metadata,
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/render.py", line 546, in finalize_metadata
    build_unsat, host_unsat = add_upstream_pins(m,
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/render.py", line 408, in add_upstream_pins
    host_deps, host_unsat, extra_run_specs_from_host = _read_upstream_pin_files(m, 'host',
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/render.py", line 373, in _read_upstream_pin_files
    deps, actions, unsat = get_env_dependencies(m, env, m.config.variant,
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/render.py", line 138, in get_env_dependencies
    actions = environ.get_install_actions(tmpdir, tuple(dependencies), env,
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda_build/environ.py", line 799, in get_install_actions
    actions = install_actions(prefix, index, specs, force=True)
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/common/io.py", line 88, in decorated
    return f(*args, **kwds)
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/plan.py", line 474, in install_actions
    txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned)
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/core/solve.py", line 152, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/core/solve.py", line 195, in solve_for_diff
    final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/core/solve.py", line 313, in solve_final_state
    ssc = self._add_specs(ssc)
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/core/solve.py", line 707, in _add_specs
    conflicts = ssc.r.get_conflicting_specs(tuple(MatchSpec(_)
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/resolve.py", line 1109, in get_conflicting_specs
    C = r2.gen_clauses()
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/common/io.py", line 88, in decorated
    return f(*args, **kwds)
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/resolve.py", line 912, in gen_clauses
    for ms in self.ms_depends(prec):
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/resolve.py", line 796, in ms_depends
    deps = [MatchSpec(d) for d in prec.combined_depends]
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/models/records.py", line 340, in combined_depends
    result = {ms.name: ms for ms in MatchSpec.merge(self.depends)}
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/models/match_spec.py", line 478, in merge
    reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/models/match_spec.py", line 478, in <lambda>
    reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/models/match_spec.py", line 509, in _merge
    final = this_component.merge(that_component)
  File "/home/pkus/programs/mambaforge/lib/python3.9/site-packages/conda/models/match_spec.py", line 760, in merge
    raise ValueError("Incompatible component merge:\n  - %r\n  - %r"
ValueError: Incompatible component merge:
  - 'mpi_openmpi_*'
  - 'mpi_openmpi_tempest*'

Could you help me to solve this issue? Thanks!

Expected Behavior

Conda builds the package.

Steps to Reproduce

conda skeleton cran https://github.com/pawel125/clonalityParsers

Replace package names in meta.yaml: r-genomeinfodb -> bioconductor-genomeinfodb r-plyranges -> bioconductor-plyranges

conda build --R=4.2.0 r-clonalityparsers
Output of conda info

     active environment : base
    active env location : /home/pkus/programs/mambaforge
            shell level : 1
       user config file : /home/pkus/.condarc
 populated config files : /home/pkus/programs/mambaforge/.condarc
                          /home/pkus/.condarc
          conda version : 4.12.0
    conda-build version : 3.21.9
         python version : 3.9.13.final.0
       virtual packages : __linux=5.13.0=0
                          __glibc=2.34=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/pkus/programs/mambaforge  (writable)
      conda av data dir : /home/pkus/programs/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          http://conda.anaconda.org/dranew/linux-64
                          http://conda.anaconda.org/dranew/noarch
          package cache : /home/pkus/programs/mambaforge/pkgs
                          /home/pkus/.conda/pkgs
       envs directories : /home/pkus/programs/mambaforge/envs
                          /home/pkus/.conda/envs
               platform : linux-64
             user-agent : conda/4.12.0 requests/2.27.1 CPython/3.9.13 Linux/5.13.0-41-generic ubuntu/21.10 glibc/2.34
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

I am using conda version 4.12.0

mguijarr commented 2 years ago

I had the same problem with conda build. After days of struggle, I switched to boa : problem solved using the conda mambabuild command instead of conda build. Works like a charm.

pawelqs commented 2 years ago

@mguijarr It's working! Thanks a lot! So many snakes to use instead of conda :)

mguijarr commented 2 years ago

I don't think this should be closed, since it is not solved with conda itself.

pawelqs commented 2 years ago

Reopened ;)

stefdoerr commented 2 years ago

I was able to fix it by forcing higher numpy version (>=1.17) in my meta.yaml file. boa actually highlighted the issue to me more clearly but after increasing numpy version I don't need to use boa anymore.

jehturner commented 1 year ago

See a possible workaround in https://github.com/conda/conda/issues/11442#issuecomment-1416186371.

jehturner commented 1 year ago

I was able to fix it by forcing higher numpy version (>=1.17) in my meta.yaml file. boa actually highlighted the issue to me more clearly but after increasing numpy version I don't need to use boa anymore.

I accidentally rediscovered this solution after running into a case that was not solved by the workaround I posted in the last comment. As of writing, it looks like building any package that has conda-forge python and numpy as build/host requirements does not work with conda-build --python=3.10 on Linux, but if the constraint is changed to numpy >=1.17 in meta.yaml it works fine. I think this bug must occur during the process of comparing available package versions, rather than due to a real dependency conflict. Specifying --numpy=<version> for conda-build makes no difference (there aren't even any numpy builds earlier than 1.21 for Python 3.10).

This issue can be reproduced by trying to build the following trivial meta.yaml with or without the numpy version uncommented:

package:
    name: test
    version: 0.1

build:
    number: '0'
    script: touch ${PREFIX}/a_file

requirements:
    build:
    - python
    - numpy # >=1.17
    run:
    - python
    - numpy # >=1.17

It seems that some change within the conda-forge channel (probably the addition of certain builds?) has broken recipes that were previously working. I think this is really a conda problem (in fact I see @minrk says so here), but it mostly shows up when using conda-build and I'm just trying to document the manifestation a bit better. This has nothing explicitly to do with MPI.

chebee7i commented 1 year ago

Any progress on this? I'm hitting this regularly and have to use mambabuild instead.

ValueError: Incompatible component merge:
  - '*mpich*'
  - 'mpi_mpich_*'
$ cat meta.yaml
{% set name = "bitsandbytes" %}
{% set version = "0.39.0" %}

package:
  name: "{{ name|lower }}"
  version: "{{ version }}"

build:
  number: 0

requirements:
  host:
    - pip
    - python {{ PY_VER }}*,>=3.6
    - absl-py
    - nltk
    - numpy
    - six>=1.14
  run:
    - python {{ PY_VER }}*,>=3.6
    - absl-py
    - nltk
    - numpy
    - six>=1.14
test:
  imports:
    - bitsandbytes

$ cat build.sh
pip install bitsandbytes==0.39.0
tazimmerman commented 1 year ago

@chebee7i Did you try using numpy >=1.17 in your recipe? I have a similar issue and that minor change actually worked for me with plain conda-build.

chebee7i commented 1 year ago

That does indeed seem to fix the issue. Very weird that we need it, since it should be picking up something much more recent than 1.17 even.

mooreniemi commented 1 year ago

Restricting to >=1.19 did not work for me.

 14 requirements:
 15   build:
 16     - python
 17     - numpy>=1.19
 18     - setuptools
 19     - pip
 20   run:
 21     - python
 22     - numpy>=1.19

Switching to mambabuild did work for me.

rphel commented 1 year ago

Still hitting this issue with numpy 1.22.

github-actions[bot] commented 2 months ago

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

  1. Verify that you can still reproduce the issue at hand
  2. Comment that the issue is still reproducible and include:
    • What OS and version you reproduced the issue on
    • What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

minrk commented 2 months ago

I'd appreciate it if stale bots didn't create work for people.

Still an issue, fix is known: https://github.com/conda/conda/pull/11612

> command conda create -n test --dry-run 'petsc=*=real*' 'petsc=*=real_*'
Retrieving notices: ...working... done

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/exception_handler.py", line 18, in __call__
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/cli/main.py", line 84, in main_subshell
        exit_code = do_call(args, parser)
                    ^^^^^^^^^^^^^^^^^^^^^
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/cli/conda_argparse.py", line 200, in do_call
        result = getattr(module, func_name)(args, parser)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/notices/core.py", line 127, in wrapper
        return_value = func(*args, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/cli/main_create.py", line 148, in execute
        return install(args, parser, "create")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/cli/install.py", line 368, in install
        solver = solver_backend(
                 ^^^^^^^^^^^^^^^
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda_libmamba_solver/solver.py", line 87, in __init__
        super().__init__(
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/core/solve.py", line 96, in __init__
        self.specs_to_add = frozenset(MatchSpec.merge(s for s in specs_to_add))
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/models/match_spec.py", line 504, in merge
        reduce(lambda x, y: x._merge(y, union), group)
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/models/match_spec.py", line 504, in <lambda>
        reduce(lambda x, y: x._merge(y, union), group)
                            ^^^^^^^^^^^^^^^^^^
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/models/match_spec.py", line 536, in _merge
        final = this_component.merge(that_component)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/minrk/conda/lib/python3.11/site-packages/conda/models/match_spec.py", line 828, in merge
        raise ValueError(
    ValueError: Incompatible component merge:
      - 'real*'
      - 'real_*'

`$ /Users/minrk/conda/condabin/conda create -n test --dry-run petsc=*=real* petsc=*=real_*`

  environment variables:
      AUTOJUMP_ERROR_PATH=/Users/minrk/Library/autojump/errors.log
                 CIO_TEST=<not set>
    CONDA_ALLOW_SOFTLINKS=false
                CONDA_EXE=/Users/minrk/conda/bin/conda
         CONDA_PYTHON_EXE=/Users/minrk/conda/bin/python
               CONDA_ROOT=/Users/minrk/conda
              CONDA_SHLVL=0
           CURL_CA_BUNDLE=<not set>
                 INFOPATH=/opt/homebrew/share/info:
               LD_PRELOAD=<not set>
                     PATH=/Users/minrk/.local/bin:/Users/minrk/dev/mine/git-stuff/bin:/Users/min
                          rk/dev/mine/dotfiles/bin:/Users/minrk/conda/condabin:/Users/minrk/.kre
                          w/bin:/Users/minrk/conda/bin:/opt/homebrew/bin:/opt/homebrew/Caskroom/
                          google-cloud-sdk/latest/google-cloud-sdk/bin:/opt/homebrew/sbin:/usr/l
                          ocal/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/
                          var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/b
                          in:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin
                          :/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/apple
                          internal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/usr/local/MacGPG2/bi
                          n:/Library/TeX/texbin:/Applications/Wireshark.app/Contents/MacOS:/Libr
                          ary/Frameworks/Mono.framework/Versions/Current/Commands
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : None
            shell level : 0
       user config file : /Users/minrk/.condarc
 populated config files : /Users/minrk/conda/.condarc
                          /Users/minrk/.condarc
          conda version : 24.7.1
    conda-build version : 24.5.2.dev60
         python version : 3.11.9.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=m1
                          __conda=24.7.1=0
                          __osx=14.6.1=0
                          __unix=0=0
       base environment : /Users/minrk/conda  (writable)
      conda av data dir : /Users/minrk/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/foo/osx-arm64
                          https://conda.anaconda.org/foo/noarch
                          https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/minrk/conda/pkgs
                          /Users/minrk/.conda/pkgs
       envs directories : /Users/minrk/conda/envs
                          /Users/minrk/.conda/envs
               platform : osx-arm64
             user-agent : conda/24.7.1 requests/2.31.0 CPython/3.11.9 Darwin/23.6.0 OSX/14.6.1 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
                UID:GID : 501:20
             netrc file : /Users/minrk/.netrc
           offline mode : False

An unexpected error has occurred. Conda has prepared the above report.
If you suspect this error is being caused by a malfunctioning plugin,
consider using the --no-plugins option to turn off plugins.

Example: conda --no-plugins install <package>

Alternatively, you can set the CONDA_NO_PLUGINS environment variable on
the command line to run the command without plugins enabled.

Example: CONDA_NO_PLUGINS=true conda install <package>

Interestingly, mamba doesn't appear to have this problem.