Closed dhirschfeld closed 4 years ago
I instrumented that line:
pin_names = ['kubernetes', 'blas', 'numpy', 'mkl', 'mkl-service', 'mkl_fft', 'mkl_random', 'scipy', 'scikit-learn', 'tensorflow', 'numexpr', 'qt']
type(key) = <class 'str'>
key = 'blas'
type(pins) = <class 'tuple'>
pins = (
MatchSpec("kubernetes=1.17.5", optional=True),
MatchSpec("pkgs/main::blas[build=*mkl]", optional=True),
MatchSpec("pkgs/main::numpy", optional=True),
MatchSpec("pkgs/main::mkl", optional=True),
MatchSpec("pkgs/main::mkl-service", optional=True),
MatchSpec("pkgs/main::mkl_fft", optional=True),
MatchSpec("pkgs/main::mkl_random", optional=True),
MatchSpec("pkgs/main::scipy", optional=True),
MatchSpec("pkgs/main::scikit-learn", optional=True),
MatchSpec("pkgs/main::tensorflow", optional=True),
MatchSpec("pkgs/main::numexpr", optional=True),
MatchSpec("conda-forge::qt[version='>=5.9.7']", optional=True)
)
Looking for: ['abseil-cpp', 'aiofiles', 'aiohttp', 'alabaster', 'algopy', 'altair', 'anaconda-client', 'aniso8601', 'anyio', 'anyjson', 'appdirs', 'argh', 'argon2_cffi', 'arrow', 'arrow-cpp', 'arviz', 'asciitree', 'asgiref', 'asks', 'astroid', 'astropy', 'async-timeout', 'asyncpg', 'async_generator', 'atomicwrites', 'attrs', 'autograd', 'autopep8', 'aws-sdk-cpp', 'azure-devops', 'babel', 'backcall', 'backports', 'backports.csv', 'backports.shutil_get_terminal_size', 'bat', 'bcrypt', 'beautifulsoup4', 'binaryornot', 'bitarray', 'black', 'pkgs/main::blas[build=*mkl]', 'bleach', 'blinker', 'blosc', 'bokeh', 'boost-cpp', 'boto', 'bottleneck', 'brotli', 'brotlipy', 'bzip2', 'c-ares', 'ca-certificates', 'cachetools', 'catalogue', 'catboost', 'certifi', 'cffi', 'cftime', 'chainer', 'chardet', 'chart-studio', 'cheroot', 'cherrypy', 'chest', 'click', 'cli_helpers', 'cloudpickle', 'clyent', 'colorama', 'colorcet', 'colorlover', 'colour', 'comtypes', 'conda', 'conda-build', 'conda-package-handling', 'configobj', 'console_shortcut', 'contextlib2', 'cookiecutter', 'coverage', 'cryptography', 'curio', 'curl', 'cx_oracle', 'cycler', 'cymem', 'cython', 'cython-blis', 'cytoolz', 'dask', 'dask-core', 'dask-glm', 'dask-ml', 'datashader', 'datashape', 'deap', 'decorator', 'defusedxml', 'diagrams', 'diff-match-patch', 'dill', 'distributed', 'docopt', 'docutils', 'entrypoints', 'et_xmlfile', 'factory_boy', 'faker', 'fastapi', 'fastavro', 'fastcache', 'fasteners', 'filelock', 'flake8', 'flask', 'flask-cors', 'fletcher', 'freetds', 'freetype', 'fs', 'fsspec', 'furl', 'future', 'get_terminal_size', 'gevent', 'gflags', 'gidgethub', 'gitdb', 'github3.py', 'gitpython', 'glob2', 'glog', 'graphene', 'graphql-core', 'graphql-relay', 'graphviz', 'greenlet', 'grpc-cpp', 'h11', 'h2', 'h5py', 'hdf4', 'hdf5', 'heapdict', 'holoviews', 'hpack', 'hstspreload', 'httpcore', 'httpx', 'humanize', 'hvplot', 'hypercorn', 'hyperframe', 'hypothesis', 'ibis-framework', 'icc_rt', 'icu', 'idna', 'imagecodecs-lite', 'imageio', 'imagesize', 'importlib-metadata', 'importlib_metadata', 'impyla', 'infinity', 'inflect', 'inflection', 'intel-openmp', 'intervals', 'intervaltree', 'invoke', 'ipykernel', 'ipympl', 'ipyparallel', 'ipyscales', 'ipython', 'ipython_genutils', 'ipywidgets', 'isodate', 'isort', 'itsdangerous', 'jaraco.classes', 'jaraco.collections', 'jaraco.functools', 'jaraco.text', 'jdcal', 'jedi', 'jinja2', 'jinja2-time', 'joblib', 'jpeg', 'json5', 'jsonschema', 'jupyter', 'jupyter-server-proxy', 'jupyterlab', 'jupyterlab_server', 'jupyter_client', 'jupyter_console', 'jupyter_core', 'jwcrypto', 'kiwisolver', 'krb5', 'lazy-object-proxy', 'libarchive', 'libblas', 'libcblas', 'libclang', 'libcurl', 'libgpuarray', 'libiconv', 'liblapack', 'liblapacke', 'liblief', 'libmklml', 'libnetcdf', 'libopencv', 'libpng', 'libpq', 'libprotobuf', 'libsodium', 'libssh2', 'libtiff', 'libwebp', 'libxgboost', 'libxml2', 'libxslt', 'lightgbm', 'line_profiler', 'llvmlite', 'locket', 'lxml', 'lz4', 'lz4-c', 'm2-msys2-runtime', 'm2-patch', 'm2w64-gcc-libgfortran', 'm2w64-gcc-libs', 'm2w64-gcc-libs-core', 'm2w64-gmp', 'm2w64-libwinpthread-git', 'mako', 'markdown', 'markupsafe', 'matplotlib', 'matplotlib-base', 'mccabe', 'mechanicalsoup', 'memory_profiler', 'menuinst', 'mistune', 'pkgs/main::mkl', 'mkl-include', 'pkgs/main::mkl-service', 'mklml', 'pkgs/main::mkl_fft', 'pkgs/main::mkl_random', 'mlxtend', 'mock', 'monotonic', 'more-itertools', 'mpmath', 'msgpack-python', 'msrest', 'msys2-conda-epoch', 'multidict', 'multipledispatch', 'murmurhash', 'mypy', 'mypy_extensions', 'nbconvert', 'nbdime', 'nbformat', 'ndg-httpsclient', 'netcdf4', 'networkx', 'ninja', 'nltk', 'nose', 'notebook', 'numba', 'numcodecs', 'numdifftools', 'pkgs/main::numexpr', 'pkgs/main::numpy', 'numpydoc', 'oauthlib', 'olefile', 'opencv', 'openpyxl', 'openssl', 'opt_einsum', 'orderedmultidict', 'outcome', 'packaging', 'pandas', 'pandoc', 'pandocfilters', 'panel', 'param', 'parquet-cpp', 'parso', 'partd', 'passlib', 'path', 'path.py', 'pathlib2', 'pathspec', 'pathtools', 'patsy', 'pcre', 'pep8', 'pexpect', 'pgcli', 'pgspecial', 'phantomjs', 'phonenumbers', 'pickleshare', 'pillow', 'pip', 'pkginfo', 'plac', 'plotly', 'pluggy', 'ply', 'pomegranate', 'portend', 'poyo', 'preshed', 'priority', 'prometheus_client', 'promise', 'prompt-toolkit', 'prompt_toolkit', 'protobuf', 'psutil', 'psycopg2', 'py', 'py-lief', 'py-opencv', 'py-xgboost', 'pyaml', 'pyarrow', 'pyasn1', 'pycodestyle', 'pycosat', 'pycparser', 'pycrypto', 'pyct', 'pyct-core', 'pycurl', 'pydantic', 'pydocstyle', 'pydot', 'pyflakes', 'pygments', 'pygpu', 'pyjwt', 'pylint', 'pymc3', 'pympler', 'pymysql', 'pyodbc', 'pyopenssl', 'pyparsing', 'pyqt', 'pyreadline', 'pyrsistent', 'pysocks', 'pytables', 'pytest', 'pytest-cov', 'python-blosc', 'python-cufflinks', 'python-dateutil', 'python-graphviz', 'python-hdfs', 'python-jsonrpc-server', 'python-libarchive-c', 'python-multipart', 'python-slugify', 'pythonnet', 'python_abi', 'pytorch-cpu', 'pytz', 'pyviz_comms', 'pywavelets', 'pywin32', 'pywinpty', 'pyyaml', 'pyzmq', 'qdarkstyle', "conda-forge::qt[version='>=5.9.7']", 'qtawesome', 'qtconsole', 'qtpy', 'quandl', 'quart', 'quart-trio', 're2', 'regex', 'repoze.lru', 'requests', 'requests-kerberos', 'requests-negotiate-sspi', 'requests-oauthlib', 'retrying', 'rfc3986', 'ripgrep', 'rise', 'rope', 'routes', 'ruamel_yaml', 'rx', 'scikit-image', 'pkgs/main::scikit-learn', 'scikit-optimize', 'pkgs/main::scipy', 'seaborn', 'selenium', 'send2trash', 'setproctitle', 'setuptools', 'simpervisor', 'simplegeneric', 'simplejson', 'singledispatch', 'sip', 'six', 'skorch', 'smmap', 'snakeviz', 'snappy', 'sniffio', 'snowballstemmer', 'sockjs-tornado', 'sortedcontainers', 'soupsieve', 'spacy', 'sphinx', 'sphinxcontrib-applehelp', 'sphinxcontrib-devhelp', 'sphinxcontrib-htmlhelp', 'sphinxcontrib-jsmath', 'sphinxcontrib-qthelp', 'sphinxcontrib-serializinghtml', 'sqlalchemy', 'sqlalchemy-utils', 'sqlite', 'sqlparse', 'srsly', 'starlette', 'statsmodels', 'stopit', 'sympy', 'tabulate', 'tbb', 'tblib', 'tempora', 'termcolor', 'terminado', 'terminaltables', 'testpath', 'text-unidecode', 'theano', 'thinc', 'threadpoolctl', 'thrift-cpp', 'thriftpy', 'thriftpy2', 'tifffile', 'tk', 'toml', 'toolz', 'torchvision-cpu', 'tornado', 'tpot', 'tqdm', 'traitlets', 'trio', 'turbodbc', 'typed-ast', 'typing_extensions', 'tzlocal', 'ujson', 'umap-learn', 'unicodecsv', 'unidecode', 'update_checker', 'uritemplate', 'uritemplate.py', 'urllib3', 'vc', 'vs2015_runtime', 'vs2015_win-64', 'wasabi', 'watchdog', 'wcwidth', 'webencodings', 'werkzeug', 'wheel', 'whichcraft', 'widgetsnbextension', 'wincertstore', 'winkerberos', 'winpty', 'win_inet_pton', 'win_unicode_console', 'wrapt', 'wsaccel', 'wsproto', 'xarray', 'xlrd', 'xlsxwriter', 'xlwings', 'xlwt', 'xz', 'yaml', 'yapf', 'yarl', 'zarr', 'zc.lockfile', 'zeromq', 'zict', 'zipp', 'zlib', 'zope.event', 'zope.interface', 'zstd', '_py-xgboost-mutex']
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "C:\Miniconda3\lib\site-packages\conda\exceptions.py", line 1079, in __call__
return func(*args, **kwargs)
File "C:\Miniconda3\lib\site-packages\mamba\mamba.py", line 872, in exception_converter
raise e
File "C:\Miniconda3\lib\site-packages\mamba\mamba.py", line 866, in exception_converter
exit_code = _wrapped_main(*args, **kwargs)
File "C:\Miniconda3\lib\site-packages\mamba\mamba.py", line 832, in _wrapped_main
exit_code = do_call(args, p)
File "C:\Miniconda3\lib\site-packages\mamba\mamba.py", line 732, in do_call
exit_code = update(args, parser)
File "C:\Miniconda3\lib\site-packages\mamba\mamba.py", line 659, in update
install(args, parser, 'update')
File "C:\Miniconda3\lib\site-packages\mamba\mamba.py", line 587, in install
solver.add_jobs(mamba_solve_specs, solver_task)
RuntimeError: Cannot erase a channel-specific package. (pkgs/main::blas[build=*mkl])
`$ C:\Miniconda3\Scripts\mamba update --all`
I think my kitchen-sink environment is giving this feature a pretty tough test! 😬
I might have to leave it for now but let me know how I could instrument that last error to get you more useful info...
@dhirschfeld would you mind sharing your pins
file here?
Thanks for testing mamba so thoroughly :)
btw I think we need to re-engineer slightly how pins work in mamba:
We need to use SOLVER_LOCK for all packages that don't match the pin so that they will not be in the set of packages that might be installed. This is how we need to implement this to work fine: https://github.com/openSUSE/libsolv/issues/388
I understand that your pins are optional (i.e. you don't necessarily have these packages installed in your environment?).
also right now, pins are only applied with update_all
which isn't correct ;)
I pin blas to MKL because we use Intel processors and I pin numpy
to defaults because IIUC they patch in some MKL functionality to get better performance. Although it may not be necessary I also pin some other MKL using packages.
.condarc
pinned_packages:
- kubernetes=1.17.5
- https://repo.anaconda.com/pkgs/main::blas[build=*mkl]
- https://repo.anaconda.com/pkgs/main::numpy
- https://repo.anaconda.com/pkgs/main::mkl
- https://repo.anaconda.com/pkgs/main::mkl-service
- https://repo.anaconda.com/pkgs/main::mkl_fft
- https://repo.anaconda.com/pkgs/main::mkl_random
- https://repo.anaconda.com/pkgs/main::scipy
- https://repo.anaconda.com/pkgs/main::scikit-learn
- https://repo.anaconda.com/pkgs/main::tensorflow
- https://repo.anaconda.com/pkgs/main::numexpr
- https://conda.anaconda.org/conda-forge::qt >=5.9.7
thanks for the info, this will give me a good test case to make sure pinnings work as expected
Unless it has been addressed elsewhere https://github.com/QuantStack/mamba/issues/328#issuecomment-641648880 is still a problem.
I'll wait for the next release, test and if it's still there will open a new issue (since I can't seem to re-open this one)
Nope, it's not 100% fixed. I just merged your PR for now.
Just posting, as a data-point, the different solutions for a conda
/mamba
update --all
for my kitchen sink analytics environment.
(Haven't had time to look yet myself - just wanted to post the data so it wasn't lost)
I'll note, initially mamba
wouldn't let me update until I'd removed inconsistent packages (xref: #412)
Your pinning does not match what's currently installed. Please remove the pin and fix your installation
Pin: pkgs/main::numpy
Currently installed: https://conda.anaconda.org/conda-forge/win-64::numpy==1.18.4=py37hae9e721_0
I like being strict but maybe mamba could just enforce consistency after the install rather than requiring the environment to be consistent first - otherwise it makes it harder to get to a consistent state if something is mucked up.
Sorry for necro-posting - in a rush. Can open new issues if you think it's worthwhile...
Will also xref my issue #297 as the output above is hard to grok when copy/pasted with no colours.
Hey @dhirschfeld thanks a lot for your testing! I'll reopen the issue just so that I don't forget to take a look at your new comments.
I checked your output and I agree that mamba needs some love with regards to updates. The channel stuff is not consistently handled and we need some sort of policy if noarch packages should be preferred or not. But I think the main issue in this thread is fixed (with the pinnings). Do you want to create 1 or 2 new issues? one for the way you wish pinnings would work, and one for update love?
Yep, I'll circle back and see what (if anything) needs to be done. Prob won't get to that until the weekend though!