search

atmtools / typhon

Tools for atmospheric research
http://www.radiativetransfer.org/
MIT License
58 stars 33 forks source link

Cannot conda install typhon for Python 3.8 #350

Closed gerritholl closed 4 years ago

gerritholl commented 4 years ago

Bug report

Bug summary

Using conda, I cannot install typhon for Python 3.8. In fact, conda throws an exception. Probably a bug in conda, but possibly triggered by something in typhon as it's not happening with Python 3.7 and I've not seen this bug with any other package installed via conda. The failure appears similar to https://github.com/conda/conda/issues/8811 except that it's supposed to be fixed in conda 4.8.2.

Code for reproduction

conda create -n test -c rttools -c conda-forge --strict-channel-priority python=3.8 typhon

Actual outcome

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... 
Building graph of deps:   0%|          | 0/4 [00:00<?, ?it/s]
Examining @/linux-64::__glibc==2.26=0:   0%|          | 0/4 [00:00<?, ?it/s]
Examining typhon:  25%|██▌       | 1/4 [00:00<00:00, 7724.32it/s]           
Examining python=3.8:  50%|█████     | 2/4 [00:02<00:02,  1.40s/it]
Examining python=3.8:  75%|███████▌  | 3/4 [00:02<00:00,  1.07it/s]
Examining @/linux-64::__cuda==10.0=0:  75%|███████▌  | 3/4 [00:02<00:00,  1.07it/s]

Determining conflicts:   0%|          | 0/4 [00:00<?, ?it/s]
Examining conflict for typhon python:   0%|          | 0/4 [00:00<?, ?it/s]
Examining conflict for __cuda:  25%|██▌       | 1/4 [00:00<00:02,  1.17it/s]
Examining conflict for __cuda:  50%|█████     | 2/4 [00:00<00:00,  2.34it/s]

Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/main.py", line 84, in _main
        exit_code = do_call(args, p)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 82, in do_call
        return getattr(module, func_name)(args, parser)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/main_create.py", line 37, in execute
        install(args, parser, 'create')
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/install.py", line 261, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 114, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 157, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 281, in solve_final_state
        ssc = self._run_sat(ssc)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 805, in _run_sat
        ssc.solution_precs = ssc.r.solve(tuple(final_environment_specs),
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 1318, in solve
        self.find_conflicts(specs, specs_to_add, history_specs)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 347, in find_conflicts
        bad_deps = self.build_conflict_map(specs, specs_to_add, history_specs)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 506, in build_conflict_map
        c = self.breadth_first_search_for_dep_graph(
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 369, in breadth_first_search_for_dep_graph
        last_spec = MatchSpec.union((path[-1], target_paths[-1][-1]))[0]
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 481, in union
        return cls.merge(match_specs, union=True)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 475, in merge
        reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 475, in <lambda>
        reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 502, in _merge
        final = this_component.union(that_component)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 764, in union
        return '|'.join(options)
    TypeError: sequence item 0: expected str instance, Channel found

`$ /data/gholl/miniconda3/bin/conda create -n test -c rttools -c conda-forge --strict-channel-priority python=3.8 typhon`

  environment variables:
         ALSA_CONFIG_PATH=/etc/alsa-pulse.conf
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=/media/nas/x21324/miniconda3
                CONDA_EXE=/data/gholl/miniconda3/bin/conda
             CONDA_PREFIX=/media/nas/x21324/miniconda3
    CONDA_PROMPT_MODIFIER=(/media/nas/x21324/miniconda3)
         CONDA_PYTHON_EXE=/data/gholl/miniconda3/bin/python
               CONDA_ROOT=/data/gholl/miniconda3
              CONDA_SHLVL=1
                  MANPATH=/media/nas/x21308/inst/texlive/2019/texmf-
                          dist/doc/man:/media/nas/x21308/inst/texlive/2019/texmf-
                          dist/doc/man:/media/nas/x21308/inst/texlive/2019/texmf-
                          dist/doc/man:/usr/local/man:/usr/share/man
                 NO_PROXY=<set>
                     PATH=/media/nas/x21308/inst/texlive/2019/bin/x86_64-linux:/media/nas/x21308
                          /inst/texlive/2019/bin/x86_64-linux:/media/nas/x21308/inst/texlive/201
                          9/bin/x86_64-linux:/media/nas/x21324/miniconda3/bin:/media/nas/x21324/
                          miniconda3/condabin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/loc
                          al/bin:/usr/bin:/bin:/opt/bin:/usr/lib/mit/sbin:/opt/perforce/bin:/opt
                          /hdf-java/bin:/usr/lib/mit/sbin:/opt/perforce/bin
            PYTHONSTARTUP=/home/gholl/.pythonrc.py
       REQUESTS_CA_BUNDLE=<not set>
              SOCKS_PROXY=<set>
            SSL_CERT_FILE=<not set>
               WINDOWPATH=7
                 XNLSPATH=/usr/share/X11/nls
                ftp_proxy=<set>
             gopher_proxy=<set>
               http_proxy=<set>
              https_proxy=<set>
                 no_proxy=<set>
              socks_proxy=<set>

     active environment : /media/nas/x21324/miniconda3
    active env location : /media/nas/x21324/miniconda3
            shell level : 1
       user config file : /home/gholl/.condarc
 populated config files : /home/gholl/.condarc
          conda version : 4.8.2
    conda-build version : not installed
         python version : 3.8.1.final.0
       virtual packages : __cuda=10.0
                          __glibc=2.26
       base environment : /data/gholl/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/rttools/linux-64
                          https://conda.anaconda.org/rttools/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /data/gholl/miniconda3/pkgs
                          /home/gholl/.conda/pkgs
       envs directories : /data/gholl/miniconda3/envs
                          /home/gholl/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.8.1 Linux/4.12.14-lp150.12.82-default opensuse-leap/15.0 glibc/2.26
                UID:GID : 16209:16217
             netrc file : None
           offline mode : False

An unexpected error has occurred. Conda has prepared the above report.

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?

[y/N]: 
Timeout reached. No report sent.

Expected outcome

I expected a working conda environment with the latest released typhon and Python 3.8.1.

Version information

gerritholl commented 4 years ago

Sometimes conda doesn't throw an exception, but instead outputs:

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... 
Building graph of deps:   0%|          | 0/4 [00:00<?, ?it/s]
Examining @/linux-64::__cuda==10.0=0:   0%|          | 0/4 [00:00<?, ?it/s]
Examining typhon:  25%|██▌       | 1/4 [00:00<00:00, 4315.13it/s]          
Examining python=3.8:  50%|█████     | 2/4 [00:02<00:02,  1.36s/it]
Examining python=3.8:  75%|███████▌  | 3/4 [00:02<00:00,  1.10it/s]
Examining @/linux-64::__glibc==2.26=0:  75%|███████▌  | 3/4 [00:02<00:00,  1.10it/s]

Determining conflicts:   0%|          | 0/4 [00:00<?, ?it/s]
Examining conflict for __cuda:   0%|          | 0/4 [00:00<?, ?it/s]
Examining conflict for typhon python:  25%|██▌       | 1/4 [00:00<00:00, 10699.76it/s]

Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
typhon -> python[version='3.5.*|3.6.*|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
typhon -> docutils -> python[version='2.7.*|>=2.7,<2.8.0a0|>=3.8,<3.9.0a0|3.4.*|>=3.6|>=3.5.3|>=3.5|>=3.8.0a,<3.9.0a0']
python=3.8The following specifications were found to be incompatible with your CUDA driver:

  - feature:/linux-64::__cuda==10.0=0

Your installed CUDA driver is: 10.0

Note that strict channel priority may have removed packages required for satisfiability.
gerritholl commented 4 years ago

Also reported at https://github.com/conda/conda/issues/9681

gerritholl commented 4 years ago

If I install it into an existing environment I get:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: /
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - typhon -> python[version='3.5.*|3.6.*|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']

Your python: python=3.8

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with your CUDA driver:

  - feature:/linux-64::__cuda==10.0=0

Your installed CUDA driver is: 10.0

Note that strict channel priority may have removed packages required for satisfiability.
olemke commented 4 years ago

We don't provide any Python 3.8 Anaconda packages, since there is no stable Anaconda Distribution with Python 3.8 available yet. When you're trying to install the current py37 package in a Python 3.8 environment, it most likely triggers a downgrade to Python 3.7 and everything goes haywire in Anaconda's dependency resolution. This is an upstream issue.

gerritholl commented 4 years ago

Why do you need a stable anaconda distribution to provide packages when there is a python-3.8 package on conda-forge, which most people use? Typhon appears quite alone in this decision, there are python 3.8 packages for scipy, numpy, pandas, numba, xarray, numexpr, satpy, matplotlib, scikit-learn, imageio, bokeh, and many others. I'm not aware of other packages that wait for the full Anaconda Distribution before providing builds for 3.8. What is the rationale here?

olemke commented 4 years ago

We agreed upon that typhon only depends on anaconda main and not on conda-forge. Currently it is not possible to build a python 3.8 package without using conda-forge because netcdf4-py38 is still missing in main:

conda create -n test python=3.8 python
[snip]

conda install netcdf4     
[snip]
UnsatisfiableError: The following specifications were found
[snip]
  - netcdf4 -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0']

Your python: python=3.8

As soon as this dependency is satisfied by anaconda/main, I'll happily add a 3.8 package to our builds.

gerritholl commented 4 years ago

There's still a difference between not requiring conda-forge and not supporting conda-forge — adding a 3.8 package would not affect people trying to install typhon for python 3.7 without using conda-forge. But it's not important, I can easily install typhon in my python 3.8 environment with pip.