conda / constructor

tool for creating installers from conda packages
https://conda.github.io/constructor/
Other
452 stars 166 forks source link

constructor 3.4-3.7: installer fails looking for file included in the bash script #770

Closed briantoby closed 2 months ago

briantoby commented 3 months ago

Checklist

What happened?

I have have been using conda constructor for a while to build a package that includes a conda package that is included from local storage rather than downloaded (see construct.yaml, below). The process has been working for me for a number of years, but recently I rebuilt the installer (for MacOS) using command CONDA_SOLVER=classic constructor g2full and got an installer (bash script) that fails, looking for that file under its original location:

tobyvm@macos-14 ~ % bash /Volumes/pi/misc/gsas2full-5773-MacOSX-x86_64.sh 

Welcome to gsas2full 5773

In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
...
[/Users/tobyvm/gsas2full] >>> /tmp/g2a
PREFIX=/tmp/g2a
Unpacking payload ...

Installing base environment...

Downloading and Extracting Packages:

PathNotFoundError: /Users/toby/build/osx-64/gsas2complete-5773-0.tar.bz2

Here is the message again, but from CONDA_VERBOSITY=2 bash /Volumes/SharedFolders/pi/misc/gsas2full-5773-MacOSX-x86_64.sh

Installing base environment...

Traceback (most recent call last):
  File "conda/exception_handler.py", line 17, in __call__
  File "conda/cli/main.py", line 83, in main_subshell
  File "conda/cli/conda_argparse.py", line 196, in do_call
  File "conda/notices/core.py", line 124, in wrapper
  File "conda/cli/main_install.py", line 140, in execute
  File "conda/cli/install.py", line 265, in install
  File "conda/misc.py", line 90, in explicit
  File "conda/core/package_cache_data.py", line 886, in execute
conda.CondaMultiError: /Users/toby/build/osx-64/gsas2complete-5773-0.tar.bz2
/Users/toby/build/osx-64/gsas2complete-5773-0.tar.bz2

If I revert back to constructor=3.3, the created installer does work OK (but not when I tried constructor 3.6, 3.5 or 3.4 first; N.B. with 3.3 I can use command constructor g2full). Once I have reverted, upgrading constructor back to 3.7 seems to leave things working. I need to do a fresh miniforge install to get back to the buggy constructor. Very perplexing.

N.B. The package I am building is rather big and messy, but the conda build & constructor information is all in GitHub and I am happy to provide more details. I'm not so sure if I can come up with a smaller & cleaner example that fails.

# g2full/construct.yaml
name: gsas2full
version: 5773

channels:
  - https://conda.anaconda.org/conda-forge

conda_default_channels:
  - https://conda.anaconda.org/conda-forge

specs:
    - python=3.11
    - numpy=1.26
    - matplotlib=3.8
    - wxpython=4.2.1
    - pillow
    - pyopengl
    - scipy
    - conda
    - requests
    - hdf5
    - h5py
    - imageio
    - git
    - gitpython
    - pywin32                              [win]
    - C:\tmp\builds\win-64\gsas2complete-5773-0.tar.bz2 [win]
    - ~/build/osx-64/gsas2complete-5773-0.tar.bz2 [not win]

post_install: g2postinstall.sh           [unix]
post_install: g2postinstall.bat           [win]

license_file: EULA.txt

welcome_image: gsas2-welcome.png
icon_image: gsas2.png

keep_pkgs: False
register_python_default: False
initialize_by_default: False
write_condarc: True

Conda Info

(bldpy311) toby@BHT20 /tmp % conda info                                

     active environment : bldpy311
    active env location : /private/tmp/conda311/envs/bldpy311
            shell level : 1
       user config file : /Users/toby/.condarc
 populated config files : /private/tmp/conda311/.condarc
          conda version : 24.1.2
    conda-build version : not installed
         python version : 3.10.14.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=cannonlake
                          __conda=24.1.2=0
                          __osx=13.6.6=0
                          __unix=0=0
       base environment : /private/tmp/conda311  (writable)
      conda av data dir : /private/tmp/conda311/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /private/tmp/conda311/pkgs
                          /Users/toby/.conda/pkgs
       envs directories : /private/tmp/conda311/envs
                          /Users/toby/.conda/envs
               platform : osx-64
             user-agent : conda/24.1.2 requests/2.31.0 CPython/3.10.14 Darwin/22.6.0 OSX/13.6.6 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.7
                UID:GID : 3328:131
             netrc file : None
           offline mode : False

Conda Config

(bldpy311) toby@BHT20 /tmp % conda config --show-sources
==> /private/tmp/conda311/.condarc <==
channels:
  - conda-forge

Conda list

(bldpy311) toby@BHT20 /tmp % conda list --show-channel-urls
# packages in environment at /private/tmp/conda311/envs/bldpy311:
#
# Name                    Version                   Build  Channel
anaconda-client           1.12.3             pyhd8ed1ab_0    conda-forge
anaconda-project          0.11.1             pyhd8ed1ab_0    conda-forge
archspec                  0.2.3              pyhd8ed1ab_0    conda-forge
attrs                     23.2.0             pyh71513ae_0    conda-forge
beautifulsoup4            4.12.3             pyha770c72_0    conda-forge
boltons                   24.0.0             pyhd8ed1ab_0    conda-forge
brotli                    1.1.0                h0dc2134_1    conda-forge
brotli-bin                1.1.0                h0dc2134_1    conda-forge
brotli-python             1.1.0           py311hdf8f085_1    conda-forge
bzip2                     1.0.8                h10d778d_5    conda-forge
c-ares                    1.28.1               h10d778d_0    conda-forge
ca-certificates           2024.2.2             h8857fd0_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cctools                   986                  hd3558d4_0    conda-forge
cctools_osx-64            986                  h58a35ae_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py311hc0b63fd_0    conda-forge
chardet                   5.2.0           py311h6eed73b_1    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     24.3.0          py311h6eed73b_0    conda-forge
conda-build               24.3.0          py311h6eed73b_1    conda-forge
conda-index               0.4.0              pyhd8ed1ab_0    conda-forge
conda-libmamba-solver     24.1.0             pyhd8ed1ab_0    conda-forge
conda-pack                0.7.1              pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
conda-standalone          24.1.2               h694c41f_0    conda-forge
constructor               3.7.0              pyh55f8243_0    conda-forge
contourpy                 1.2.0           py311h7bea37d_0    conda-forge
curl                      8.7.1                h726d00d_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distro                    1.9.0              pyhd8ed1ab_0    conda-forge
expat                     2.6.2                h73e2aa4_0    conda-forge
filelock                  3.13.3             pyhd8ed1ab_0    conda-forge
fmt                       10.2.1               h7728843_0    conda-forge
fonttools                 4.50.0          py311he705e18_0    conda-forge
freetype                  2.12.1               h60636b9_2    conda-forge
gettext                   0.21.1               h8a4c099_0    conda-forge
git                       2.44.0          pl5321h8687b19_0    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.43             pyhd8ed1ab_0    conda-forge
h5py                      3.10.0          nompi_py311hf7b785c_101    conda-forge
hdf5                      1.14.3          nompi_h691f4bf_100    conda-forge
icu                       73.2                 hf5e326d_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
imageio                   2.34.0             pyh4b66e23_0    conda-forge
importlib_resources       6.4.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py311h6eed73b_3    conda-forge
jsonschema                4.21.1             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.2           py311h6eed73b_0    conda-forge
kiwisolver                1.4.5           py311h5fe6e05_1    conda-forge
krb5                      1.21.2               hb884880_0    conda-forge
lcms2                     2.16                 ha2f27b4_0    conda-forge
ld64                      711                  h4e51db5_0    conda-forge
ld64_osx-64               711                  had5d0d3_0    conda-forge
lerc                      4.0.0                hb486fe8_0    conda-forge
libaec                    1.1.3                h73e2aa4_0    conda-forge
libarchive                3.7.2                hd35d340_1    conda-forge
libblas                   3.9.0           21_osx64_openblas    conda-forge
libbrotlicommon           1.1.0                h0dc2134_1    conda-forge
libbrotlidec              1.1.0                h0dc2134_1    conda-forge
libbrotlienc              1.1.0                h0dc2134_1    conda-forge
libcblas                  3.9.0           21_osx64_openblas    conda-forge
libcurl                   8.7.1                h726d00d_0    conda-forge
libcxx                    16.0.6               hd57cbcb_0    conda-forge
libdeflate                1.20                 h49d49c5_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 h10d778d_2    conda-forge
libexpat                  2.6.2                h73e2aa4_0    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgfortran               5.0.0           13_2_0_h97931a8_3    conda-forge
libgfortran5              13.2.0               h2873a65_3    conda-forge
libiconv                  1.17                 hd75f5a5_2    conda-forge
libjpeg-turbo             3.0.0                h0dc2134_1    conda-forge
liblapack                 3.9.0           21_osx64_openblas    conda-forge
liblief                   0.14.1               ha0df490_1    conda-forge
libllvm18                 18.1.2               hbcf5fad_0    conda-forge
libmamba                  1.5.8                ha449628_0    conda-forge
libmambapy                1.5.8           py311h6c5c7ae_0    conda-forge
libnghttp2                1.58.0               h64cf6d3_1    conda-forge
libopenblas               0.3.26          openmp_hfef2a42_0    conda-forge
libpng                    1.6.43               h92b6c6a_0    conda-forge
libsolv                   0.7.28               h2d185b6_2    conda-forge
libsqlite                 3.45.2               h92b6c6a_0    conda-forge
libssh2                   1.11.0               hd019ec5_0    conda-forge
libtiff                   4.6.0                h129831d_3    conda-forge
libwebp-base              1.3.2                h0dc2134_0    conda-forge
libxcb                    1.15                 hb7f2c08_0    conda-forge
libxml2                   2.12.6               hc0ae0f7_1    conda-forge
libzlib                   1.2.13               h8a1eda9_5    conda-forge
llvm-openmp               18.1.2               hb6ac08f_0    conda-forge
lz4-c                     1.9.4                hf0c8a7f_0    conda-forge
lzo                       2.10              haf1e3a3_1000    conda-forge
markupsafe                2.1.5           py311he705e18_0    conda-forge
matplotlib                3.8.3           py311h6eed73b_0    conda-forge
matplotlib-base           3.8.3           py311h6ff1f5f_0    conda-forge
menuinst                  2.0.2           py311h6eed73b_0    conda-forge
more-itertools            10.2.0             pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
nbformat                  5.10.3             pyhd8ed1ab_0    conda-forge
ncurses                   6.4.20240210         h73e2aa4_0    conda-forge
numpy                     1.26.4          py311hc43a94b_0    conda-forge
openjpeg                  2.5.2                h7310d3a_0    conda-forge
openssl                   3.2.1                hd75f5a5_1    conda-forge
packaging                 24.0               pyhd8ed1ab_0    conda-forge
patch                     2.7.6             hbcf498f_1002    conda-forge
pathlib2                  2.3.7.post1     py311h6eed73b_3    conda-forge
pcre2                     10.43                h0ad2156_0    conda-forge
perl                      5.32.1          7_h10d778d_perl5    conda-forge
pillow                    10.2.0          py311hea5c87a_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pkginfo                   1.10.0             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.4.0              pyhd8ed1ab_0    conda-forge
psutil                    5.9.8           py311he705e18_0    conda-forge
pthread-stubs             0.4               hc929b4f_1001    conda-forge
py-lief                   0.14.1          py311h8fdbec6_1    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py311h2725bcf_0    conda-forge
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pyopengl                  3.1.6              pyhd8ed1ab_1    conda-forge
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
pypubsub                  4.0.3                      py_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.8          h9f0c242_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.19.1             pyhd8ed1ab_0    conda-forge
python-libarchive-c       5.1             py311h6eed73b_0    conda-forge
python.app                1.4             py311h2725bcf_3    conda-forge
python_abi                3.11                    4_cp311    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1           py311h2725bcf_1    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
referencing               0.34.0             pyhd8ed1ab_0    conda-forge
reproc                    14.2.4.post0         h10d778d_1    conda-forge
reproc-cpp                14.2.4.post0         h93d8f39_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-toolbelt         1.0.0              pyhd8ed1ab_0    conda-forge
ripgrep                   14.1.0               h11a7dfb_0    conda-forge
rpds-py                   0.18.0          py311hd64b9fd_0    conda-forge
ruamel.yaml               0.18.6          py311he705e18_0    conda-forge
ruamel.yaml.clib          0.2.8           py311he705e18_0    conda-forge
ruamel_yaml               0.15.80         py311h2725bcf_1009    conda-forge
scipy                     1.12.0          py311h86d0cd9_2    conda-forge
scons                     4.7.0           py311h6eed73b_0    conda-forge
setuptools                69.2.0             pyhd8ed1ab_0    conda-forge
sigtool                   0.1.3                h88f4db0_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
tapi                      1100.0.11            h9ce4665_0    conda-forge
tk                        8.6.13               h1abcd95_1    conda-forge
tornado                   6.4             py311he705e18_0    conda-forge
tqdm                      4.66.2             pyhd8ed1ab_0    conda-forge
traitlets                 5.14.2             pyhd8ed1ab_0    conda-forge
truststore                0.8.0              pyhd8ed1ab_0    conda-forge
typing_extensions         4.10.0             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
urllib3                   2.2.1              pyhd8ed1ab_0    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
wxpython                  4.2.1           py311ha82924a_4    conda-forge
xorg-libxau               1.0.11               h0dc2134_0    conda-forge
xorg-libxdmcp             1.1.3                h35c211d_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
yaml                      0.2.5                h0d85af4_2    conda-forge
yaml-cpp                  0.8.0                he965462_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h8a1eda9_5    conda-forge
zstandard                 0.22.0          py311hed14148_0    conda-forge
zstd                      1.5.5                h829000d_0    conda-forge

Additional Context

No response

briantoby commented 3 months ago

Problem is not Mac specific. Confirmed that I see the same bug on Linux.

jaimergp commented 2 months ago

Hm, I didn't know we could use abs paths there. I would approach this via:

IOW, something like this:

name: gsas2full
version: 5773

channels:
  - /Users/user/builds  # [osx]
  - /home/user/builds  # [linux]
  - C:\\tmp\\builds  # [win]
  - https://conda.anaconda.org/conda-forge

conda_default_channels:
  - https://conda.anaconda.org/conda-forge

specs:
    - python=3.11
    - numpy=1.26
    - matplotlib=3.8
    - wxpython=4.2.1
    - pillow
    - pyopengl
    - scipy
    - conda
    - requests
    - hdf5
    - h5py
    - imageio
    - git
    - gitpython
    - pywin32                              [win]
    - gsas2complete=5773

post_install: g2postinstall.sh           [unix]
post_install: g2postinstall.bat           [win]

license_file: EULA.txt

welcome_image: gsas2-welcome.png
icon_image: gsas2.png

keep_pkgs: False
register_python_default: False
initialize_by_default: False
write_condarc: True
briantoby commented 2 months ago

I was not aware that one could designate a local directory as a channel. This makes for a nice simplification in my construct.yaml. (It would be even nicer if I could set that channel as an argument to the constructor command). As far as I can tell, this change to the .yaml addresses the problem I was seeing before.