conda / conda-build

Commands and tools for building conda packages
https://docs.conda.io/projects/conda-build/
Other
381 stars 423 forks source link

building package with scikit-learn dependency fails on Windows #3582

Closed newville closed 1 year ago

newville commented 5 years ago

First, I am not 100% sure whether this is a problem with conda-build or with a particular package (scikit-learn). I suspect it is a problem with conda-build (the package installs and runs OK), but my apologies if this issue should be raised elsewhere -- please let me know if and where it should be raised.

Any suggestions for a workaround or different approach would be greatly appreciated, Thanks!

Actual Behavior

With Anaconda3-2019-3 installed and updated on Windows-10 64-bit I cannot build packages that depend on scikit-learn. These all give errors similar to:

(t1) C:\Users\newville.CARS_NT_DOMAIN\Documents\GitHub\conda_recipes>conda build lmfit
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for lmfit
INFO:conda_build.metadata:Attempting to finalize metadata for lmfit
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
BUILD START: ['lmfit-0.9.13-py37_0.tar.bz2']
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\conda-bld\lmfit_1561737929650\_h_env

The following NEW packages will be INSTALLED:

    asteval:         0.9.13-py37_0          gsecars
    blas:            1.0-mkl
    ca-certificates: 2019.5.15-0
    certifi:         2019.6.16-py37_0
    icc_rt:          2019.0.0-h0cc432a_1
    intel-openmp:    2019.4-245
    joblib:          0.13.2-py37_0
    mkl:             2019.4-245
    mkl-service:     2.0.2-py37he774522_0
    mkl_fft:         1.0.12-py37h14836fe_0
    mkl_random:      1.0.2-py37h343c172_0
    numpy:           1.16.4-py37h19fb1c0_0
    numpy-base:      1.16.4-py37hc3f5095_0
    openssl:         1.1.1c-he774522_1
    pip:             19.1.1-py37_0
    python:          3.7.3-h8c8aaf0_1
    scikit-learn:    0.21.2-py37h6288b17_0
    scipy:           1.2.1-py37h29ff71c_0
    setuptools:      41.0.1-py37_0
    six:             1.12.0-py37_0
    sqlite:          3.28.0-he774522_0
    uncertainties:   3.0.2-py37_0           gsecars
    vc:              14.1-h0510ff6_4
    vs2015_runtime:  14.15.26706-h3a45250_4
    wheel:           0.33.4-py37_0
    wincertstore:    0.2-py37_0

Preparing transaction: ...working... done
Verifying transaction: ...working... failed
Traceback (most recent call last):
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\Scripts\conda-build-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda_build\cli\main_build.py", line 445, in main
    execute(sys.argv[1:])
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda_build\cli\main_build.py", line 436, in execute
    verify=args.verify, variants=args.variants)
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda_build\api.py", line 209, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda_build\build.py", line 2313, in build_tree
    notest=notest,
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda_build\build.py", line 1377, in build
    create_build_envs(top_level_pkg, notest)
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda_build\build.py", line 1283, in create_build_envs
    is_cross=m.is_cross, is_conda=m.name() == 'conda')
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda_build\environ.py", line 868, in create_env
    execute_actions(actions, index)
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda\common\io.py", line 88, in decorated
    return f(*args, **kwds)
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda\plan.py", line 321, in execute_actions
    execute_instructions(plan, index, verbose)
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda\plan.py", line 533, in execute_instructions
    cmd(state, arg)
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda\instructions.py", line 73, in UNLINKLINKTRANSACTION_CMD
    unlink_link_transaction.execute()
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda\core\link.py", line 240, in execute
    self.verify()
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda\common\io.py", line 88, in decorated
    return f(*args, **kwds)
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda\core\link.py", line 230, in verify
    maybe_raise(CondaMultiError(exceptions), context)
  File "C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\lib\site-packages\conda\exceptions.py", line 983, in maybe_raise
    raise error
conda.CondaMultiError: The package for scikit-learn located at C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\pkgs\scikit-learn-0.21.2-py37h6288b17_0
appears to be corrupted. The path 'Lib/site-packages/sklearn/datasets/tests/data/openml/292/api-v1-json-data-list-data_name-australian-limit-2-data_version-1-status-deactivated.json.gz'
specified in the package manifest cannot be found.

The package for scikit-learn located at C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\pkgs\scikit-learn-0.21.2-py37h6288b17_0
appears to be corrupted. The path 'Lib/site-packages/sklearn/datasets/tests/data/openml/40675/api-v1-json-data-list-data_name-glass2-limit-2-data_version-1-status-deactivated.json.gz'
specified in the package manifest cannot be found.

which indicates that the scikit-learn package is corrupted. However, the file (here at `C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\pkgs\scikit-learn-0.21.2-py37h6288b17_0\Lib\site-packages\sklearn\datasets\tests\data\openml\40675\api-v1-json-data-list-data_name-glass2-limit-2-data_version-1-status-deactivated.json.gz'

does actually exist. I do not know why that file cannot be found. The name is pretty long (>256 characters, which might be suspicious) but AFAICT contains no spaces or non-Windows-friendly characters.

Expected Behavior

the package would build. To be clear, it all works on Linux and MacOS.

Steps to Reproduce

Set up the environment:

conda create --name t1 python=3.7 scipy=1.2 matplotlib
conda activate t1
conda install scikit-learn conda-build
conda install -c gsecars lmfit

(to be clear, lmfit is pure python and its dependencies are also pure python). Then build try to build a package with recipe in lmfit\meta.yaml:

## lmfit conda recipe
{% set version = "0.9.13" %}

package:
  name: lmfit
  version: {{ version }}

source:
  git_rev: {{ version }}
  git_url: https://github.com/lmfit/lmfit-py

build:
  number: 0
  script: python setup.py install

requirements:
  build:
    - python>=3.7
    - setuptools
    - numpy>=1.15
    - scipy>=1.2
    - scikit-learn>=0.21
    - six>=1.10
    - uncertainties>=3.0
    - asteval>=0.9.12

  run:
    - python>=3.7
    - setuptools
    - numpy>=1.15
    - scipy>=0.2
    - scikit-learn>=0.21
    - six>=1.10
    - uncertainties>=3.0
    - asteval>=0.9.12

test:
  # Python imports
  imports:
    - lmfit

about:
  home: http://lmfit.github.io/lmfit-py/
  license: BSD
  summary: >
      Non-Linear Least Squares Minimization, with flexible Parameter
      settings, based on scipy.optimize.leastsq, and with many additional
      classes and methods for curve fitting http:/lmfit.github.io/lmfit-py/

With this, running conda build lmfit gives the error above about not all of the test data files for scikit-learn being found. I believe the problem will happen with any package that includes scikit-learn as part of the dependency.

Output of conda info

(t1) C:\Users\newville.CARS_NT_DOMAIN\Documents\GitHub\conda_recipes>conda info

     active environment : base
    active env location : C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1
            shell level : 2
       user config file : C:\Users\newville.CARS_NT_DOMAIN\.condarc
 populated config files : C:\Users\newville.CARS_NT_DOMAIN\.condarc
          conda version : 4.7.5
    conda-build version : 3.18.5
         python version : 3.7.3.final.0
       virtual packages :
       base environment : C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
                          https://conda.anaconda.org/gsecars/win-64
                          https://conda.anaconda.org/gsecars/noarch
          package cache : C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\pkgs
                          C:\Users\newville.CARS_NT_DOMAIN\.conda\pkgs
                          C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1\envs
                          C:\Users\newville.CARS_NT_DOMAIN\.conda\envs
                          C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.7.5 requests/2.22.0 CPython/3.7.3 Windows/10 Windows/10.0.17134
          administrator : False
             netrc file : None
           offline mode : False

conda list gives:

(t1) C:\Users\newville.CARS_NT_DOMAIN\Documents\GitHub\conda_recipes>conda list
# packages in environment at C:\Users\newville.CARS_NT_DOMAIN\AppData\Local\Continuum\anaconda3\envs\t1:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0
asteval                   0.9.13                   py37_0    gsecars
beautifulsoup4            4.7.1                    py37_1
blas                      1.0                         mkl
bzip2                     1.0.6                hfa6e2cd_5
ca-certificates           2019.5.15                     0
certifi                   2019.6.16                py37_0
cffi                      1.12.3           py37h7a1dbc1_0
chardet                   3.0.4                    py37_1
conda                     4.7.5                    py37_0
conda-build               3.18.5                   py37_0
conda-package-handling    1.3.10                   py37_0
cryptography              2.7              py37h7a1dbc1_0
cycler                    0.10.0                   py37_0
filelock                  3.0.12                     py_0
freetype                  2.9.1                ha9979f8_1
glob2                     0.6                      py37_1
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha66f8fd_1
idna                      2.8                      py37_0
intel-openmp              2019.4                      245
jinja2                    2.10.1                   py37_0
joblib                    0.13.2                   py37_0
jpeg                      9b                   hb83a4c4_2
kiwisolver                1.1.0            py37ha925a31_0
libarchive                3.3.3                h0643e63_5
libiconv                  1.15                 h1df5818_7
liblief                   0.9.0                ha925a31_2
libpng                    1.6.37               h2a8f88b_0
libxml2                   2.9.9                h464c3ec_0
lmfit                     0.9.13                   py37_0    gsecars
lz4-c                     1.8.1.2              h2fa13f4_0
lzo                       2.10                 h6df0209_2
markupsafe                1.1.1            py37he774522_0
matplotlib                3.1.0            py37hc8f65d3_0
menuinst                  1.4.16           py37he774522_0
mkl                       2019.4                      245
mkl-service               2.0.2            py37he774522_0
mkl_fft                   1.0.12           py37h14836fe_0
mkl_random                1.0.2            py37h343c172_0
numpy                     1.16.4           py37h19fb1c0_0
numpy-base                1.16.4           py37hc3f5095_0
openssl                   1.1.1c               he774522_1
pip                       19.1.1                   py37_0
pkginfo                   1.5.0.1                  py37_0
psutil                    5.6.2            py37he774522_0
py-lief                   0.9.0            py37ha925a31_2
pycosat                   0.6.3            py37hfa6e2cd_0
pycparser                 2.19                     py37_0
pyopenssl                 19.0.0                   py37_0
pyparsing                 2.4.0                      py_0
pyqt                      5.9.2            py37h6538335_2
pysocks                   1.7.0                    py37_0
python                    3.7.3                h8c8aaf0_1
python-dateutil           2.8.0                    py37_0
python-libarchive-c       2.8                      py37_6
pytz                      2019.1                     py_0
pywin32                   223              py37hfa6e2cd_1
pyyaml                    5.1              py37he774522_0
qt                        5.9.7            vc14h73c81de_0
requests                  2.22.0                   py37_0
ruamel_yaml               0.15.46          py37hfa6e2cd_0
scikit-learn              0.21.2           py37h6288b17_0
scipy                     1.2.1            py37h29ff71c_0
setuptools                41.0.1                   py37_0
sip                       4.19.8           py37h6538335_0
six                       1.12.0                   py37_0
soupsieve                 1.8                      py37_0
sqlite                    3.28.0               he774522_0
tornado                   6.0.2            py37he774522_0
tqdm                      4.32.1                     py_0
uncertainties             3.0.2                    py37_0    gsecars
urllib3                   1.24.2                   py37_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_4
wheel                     0.33.4                   py37_0
win_inet_pton             1.1.0                    py37_0
wincertstore              0.2                      py37_0
xz                        5.2.4                h2fa13f4_4
yaml                      0.1.7                hc54c509_2
zlib                      1.2.11               h62dcd97_3
zstd                      1.3.7                h508b16e_0
angelo-peronio commented 5 years ago

Same issue here, using scikit-learn-0.21.2-py36h7208079_1 from conda-forge. @newville , have you meanwhile found a workaround?

conda build gives:

[...]
The package for scikit-learn located at C:\Users\angelo.peronio\AppData\Local\Continuum\miniconda3\envs\packaging\pkgs\scikit-learn-0.21.2-py36h7208079_1
appears to be corrupted. The path 'Lib/site-packages/sklearn/datasets/tests/data/openml/40675/api-v1-json-data-list-data_name-glass2-limit-2-data_version-1-status-deactivated.json.gz'
specified in the package manifest cannot be found.

conda info gives:

     active environment : packaging
    active env location : C:\Users\angelo.peronio\AppData\Local\Continuum\miniconda3\envs\packaging
            shell level : 1
       user config file : C:\Users\angelo.peronio\.condarc
 populated config files : C:\Users\angelo.peronio\.condarc
          conda version : 4.7.8
    conda-build version : not installed
         python version : 3.7.3.final.0
       virtual packages : __cuda=10.2
       base environment : C:\Users\angelo.peronio\AppData\Local\Continuum\miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\angelo.peronio\AppData\Local\Continuum\miniconda3\pkgs
                          C:\Users\angelo.peronio\.conda\pkgs
                          C:\Users\angelo.peronio\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\angelo.peronio\AppData\Local\Continuum\miniconda3\envs
                          C:\Users\angelo.peronio\.conda\envs
                          C:\Users\angelo.peronio\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.7.8 requests/2.22.0 CPython/3.7.3 Windows/10 Windows/10.0.17763
          administrator : False
             netrc file : None
           offline mode : False
angelo-peronio commented 5 years ago

Solved by enabling paths longer as 260 characters in Win10, as described at https://superuser.com/q/1119883

angelo-peronio commented 5 years ago

@newville Should this be closed?

newville commented 5 years ago

@angelo-peronio I am not entirely sure this issue should be closed. If an OS-level workaround is needed to build with long path names, is it also possible that would also be needed on the installing machine? It seems to me like a problem that conda-build gets itself into by having (probably at least somewhat unavoidable or course) by using long path names, but that does sort of imply that packages (like sklean) cannot also have long/deep path names.

It seems like conda-build should be acknowledging and warning about this limitation. In one respect, it is reporting an error when building a package building when the error is really with a dependency -- that dependency (here, sklearn) is kind of broken in that it cannot be used as a dependency (kind of the point of packages, no?).

github-actions[bot] commented 1 year ago

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

  1. Verify that you can still reproduce the issue at hand
  2. Comment that the issue is still reproducible and include:
    • What OS and version you reproduced the issue on
    • What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!