Anaconda environment issues: erratic manual installs vs yml vs folder copies

[evanseitz]

I've been having considerable trouble installing an environment on numerous machines / OS (libraries listed below). On some systems, everything works fine (such as openSUSE 15.1, macOS mojave, and Ubuntu 16) while on Ubuntu 18 I get back an error at the end telling me there's "no module named numpy"... however conda list shows numpy there.

Here's the exact procedure:

conda create --no-default-packages -n ManifoldEM python=2.7
conda install -c anaconda vtk=6.3.0
conda install -c anaconda mayavi=4.5.0
conda install -c conda-forge matplotlib
conda install -c anaconda scipy
conda install -c conda-forge mrcfile
conda install -c anaconda scikit-learn
conda install h5py
conda install -c anaconda pillow
conda install -c satra vtk
conda install pandas
conda install -c conda-forge opencv
conda install -c menpo imageio
conda install -c anaconda psutil
conda install -c anaconda numpy
conda install -c anaconda pango
conda install -c anaconda mpi4py

I also tried installing an old .yml I was using on this same machine (after scrapping the above env) and it installed successfully, but then I got a different error on running python script: undefined symbol: g_log_structured_standard

I then scrapped that environment and pasted the anaconda folder from a previous machine into this OS... and everything worked perfectly.

All in all, something has definitely changed in the backend conda libraries (for instance, I used to use conda install vtk=6.3 which no longer works)... and I'm pretty desperate to get a reliable environment for an upcoming release.

thank you so much for your time.

[msarahan]

Your procedure unfortunately is very loose. That makes it fragile over time. I can't fix it all for you, but here's some advice:

1. do not use -c anaconda. This has the same content as the defaults channel, but loses priority within that channel.
2. try to stick to one channel as much as possible. Here your best bet is probably conda-forge.
3. Install things from your non-primary channel with channel specifiers, not with the channel argument
4. do things as one step

conda create --no-default-packages -c conda-forge -n ManifoldEM python=2.7 mayavi matplotlib scipy mrcfile scikit-learn h5py pillow satra::vtk pandas opencv menpo::imageio psutil numpy pango mpi4py

when you get this working, you should create "lock files" to use next time you just want to install the same environment. https://github.com/mariusvniekerk/conda-lock/blob/master/conda_lock.py Re-lock with your original list of deps if you want to update the lock file.

[msarahan]

Note: I have not tested the above at all. It should be a step in the right direction, but it may not work without a bit more work.

[evanseitz]

Thank you @msarahan for this very clear picture. I'll move forward with your advice in mind.

[msarahan]

When you get it working, you should consider making a conda package for your software and adding your recipe to conda forge. That will make maintenance much easier for you in the future. The info on https://conda-forge.org/ should help you get started.

[evanseitz]

Hi @msarahan - I've made several attempts to create the lock methods using the repository you've so kindly recommended. However, I must be making a mistake somewhere in my logic; I'm doing the following on Linux:

1. placing the conda_lock.py file on my local system
2. adding the path to my python interpreter at the top of this script via #! /Users/.../bin/python (taken from which python)
3. making the script executable via chmod 755 conda_lock.py
4. activating the environment that I'm interested in locking via conda activate FOO and creating a .yml file for it via conda env export > environment.yml (which now resides in the same directory as conda_lock.py... then deactivating environment
5. pip install pathlib on the conda base (the script will fuss if not)
6. in this directory, using ./conda-lock.py -f environment.yml -p osx-64 -p linux-64 as directed in the readme

At this point, I'm getting the error:

TypeError: expected str, bytes or os.PathLike object, not dict
ees2192@midway:~/conda-lock-master> ./conda_lock.py -f environment.yml
generating lockfile for osx-64
Traceback (most recent call last):
  File "./conda_lock.py", line 107, in <module>
    platforms=args.platform or DEFAULT_PLATFORMS,
  File "./conda_lock.py", line 64, in make_lock_files
    platform=platform, channels=channels, specs=specs
  File "./conda_lock.py", line 37, in solve_specs_for_arch
    json_output = subprocess.check_output(args, encoding="utf-8", env=env)
  File "/home/ees2192/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/ees2192/anaconda3/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/ees2192/anaconda3/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/home/ees2192/anaconda3/lib/python3.6/subprocess.py", line 1275, in _execute_child
    restore_signals, start_new_session, preexec_fn)
TypeError: expected str, bytes or os.PathLike object, not dict

I should note that this happens in different flavors when I remove one of the -p arguments or all of them from ./conda-lock.py -f environment.yml -p osx-64 -p linux-64.