msarahan
commented
6 years ago Try updating conda-build. The cran skeleton has been hugely rewritten very recently.
Closed nick-youngblut closed 6 years ago
msarahan
commented
6 years ago Try updating conda-build. The cran skeleton has been hugely rewritten very recently.
nick-youngblut
commented
6 years ago conda is trying to downgrade conda when I try to upgrade conda-build:
$ conda update -n base conda-build
Solving environment: done
## Package Plan ##
environment location: /ebio/abt3_projects/software/dev/miniconda3_dev
added / updated specs:
- conda-build
The following packages will be downloaded:
package | build
---------------------------|-----------------
psutil-5.4.3 | py36_0 329 KB conda-forge
conda-build-3.7.1 | py36_0 423 KB conda-forge
conda-4.3.34 | py36_0 514 KB conda-forge
------------------------------------------------------------
Total: 1.2 MB
The following NEW packages will be INSTALLED:
psutil: 5.4.3-py36_0 conda-forge
The following packages will be UPDATED:
conda-build: 3.2.1-py36_0 anaconda --> 3.7.1-py36_0 conda-forge
The following packages will be DOWNGRADED:
conda: 4.4.11-py36_0 --> 4.3.34-py36_0 conda-forge
Proceed ([y]/n)? nI've tried using defaults:
$ conda install -n base defaults::conda-build
Solving environment: done
## Package Plan ##
environment location: /ebio/abt3_projects/software/dev/miniconda3_dev
added / updated specs:
- defaults::conda-build
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-build-3.7.1 | py36_0 423 KB
conda-4.3.34 | py36_0 514 KB conda-forge
psutil-5.4.3 | py36_0 329 KB conda-forge
------------------------------------------------------------
Total: 1.2 MB
The following NEW packages will be INSTALLED:
psutil: 5.4.3-py36_0 conda-forge
The following packages will be UPDATED:
conda-build: 3.2.1-py36_0 anaconda --> 3.7.1-py36_0
The following packages will be DOWNGRADED:
conda: 4.4.11-py36_0 --> 4.3.34-py36_0 conda-forge
Proceed ([y]/n)? n...but the same thing is happening. Which channel should I be using?
mingwandroid
commented
6 years ago This is actually an issue due to a change in sed's behaviour AFAIR. Thing is we don't even need to do this sed stuff!
Conda skeleton cran was added to a lot recently but really the whole thing should be rewritten!
It's not that it was badly designed to begin with it's just that it needs to do a lot more now than the original design allowed for.
But since it works mostly ok any rewrite is not something I anticipate I'll get round to anytime soon.
nick-youngblut
commented
6 years ago Hi @mingwandroid. Any idea on what I should do about my conda downgrade issue?
mingwandroid
commented
6 years ago Yes, stop using the conda-forge channel in your base/root environment.
nick-youngblut
commented
6 years ago I didn't know that I could set channels at a per-environment level (eg., base versus other environment). I thought channel priority applied to all environments
nick-youngblut
commented
6 years ago I don't see how to drop the conda-forge from just the base environment. @mingwandroid, are you suggesting to completely remove the conda-forge channel? If yes, then this no-conda-forge setup would conflict with the bioconda-defined channel priority:
conda config --add channels defaults
conda config --add channels conda-forge
conda config --add channels biocondaTherefore, I'd rather not remove conda-forge.
If there is a way to specify channel priority on a per-environment basis, can you point me to the documentation?
mingwandroid
commented
6 years ago You do not need to remove conda-forge altogether. .condarc files are searched for in the environment before your $HOME directory (edit: they are combined according to some rules actually). This is how to create an env called conda-forge that only gets software from conda-forge:
/opt/conda/bin/conda create -n conda-forge
echo "channels:" > /opt/conda/envs/conda-forge/.condarc
echo " conda-forge" >> /opt/conda/envs/conda-forge/.condarcFrom the base/root env, conda info shows:
..
user config file : /Users/rdonnelly/.condarc
..
channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/osx-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/osx-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/osx-64
https://repo.anaconda.com/pkgs/pro/noarchIf we activate your new conda-forge channel and run conda info, we should see:
..
user config file : /Users/rdonnelly/.condarc
populated config files : /opt/conda/envs/cf/.condarc
..
channel URLs : https://conda.anaconda.org/conda-forge/osx-64
https://conda.anaconda.org/conda-forge/noarchJust to be clear, for the base conda environment, I would have to create a new .condarc file in /opt/conda/envs/ (using your example), correct?
mingwandroid
commented
6 years ago mingwandroid
commented
6 years ago I think the base environment's .condarc should live in /opt/conda, but due to (probably bad) habit, I just use ~/.condarc for my base environment. @kalefranz can you advise more on this topic?
nick-youngblut
commented
6 years ago I just tried the both options and neither worked. The .condarc file that I used consists of:
channels:
- bioconda
- defaults
anaconda_upload: falseTo make the .condarc, I just removed the conda-forge channel from my ~/.condarc file.
nick-youngblut
commented
6 years ago To clarify, by "neither worked", I mean that trying to upgrade conda-build generates the following:
$ conda update -n base conda-build
Solving environment: done
## Package Plan ##
environment location: /ebio/abt3_projects/software/dev/miniconda3_dev
added / updated specs:
- conda-build
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-build-3.7.1 | py36_0 423 KB conda-forge
conda-4.3.34 | py36_0 514 KB conda-forge
psutil-5.4.3 | py36_0 329 KB conda-forge
------------------------------------------------------------
Total: 1.2 MB
The following NEW packages will be INSTALLED:
psutil: 5.4.3-py36_0 conda-forge
The following packages will be UPDATED:
conda-build: 3.2.1-py36_0 anaconda --> 3.7.1-py36_0 conda-forge
The following packages will be DOWNGRADED:
conda: 4.4.11-py36_0 --> 4.3.34-py36_0 conda-forge
Proceed ([y]/n)? nPlease use conda info to investigate what channels conda is pulling software from.
I do not recommend having defaults and bioconda in your base env, you should have only defaults in that to ensure that this critical software does not come from anywhere but us.
I am recommending you keep your base env as small as possible, containing only conda, conda-build and their dependencies.
Then you should create another env to do your work in and that env can have whatever channels you need.
nick-youngblut
commented
6 years ago hmm... don't the bioconda docs instruct the user to install bioconda, conda-forge, and defaults into the user's ~/.condarc? Maybe I'm misinterpreting their instructions, or are you suggesting that the bioconda documentation shouldn't be followed?
mingwandroid
commented
6 years ago Yes, I would recommend not following those instructions if you want to have guaranteed access to the latest conda and conda-build versions. I think @mbargull is a bioconda developer and I know he's very active with conda too, so maybe he could give us his opinion on this?
nick-youngblut
commented
6 years ago For now, I just updated conda-build by removing conda-forge from my ~/.condarc file, upgrading conda-build, then replacing conda-forge.
After updating conda-build (conda-build 3.7.1 py36_0), I'm getting the following error when using conda skeleton cran:
$ conda skeleton cran fd
Traceback (most recent call last):
File "/ebio/abt3_projects/software/dev/miniconda3_dev/bin/conda-skeleton", line 11, in <module>
sys.exit(main())
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/cli/main_skeleton.py", line 66, in main
return execute(sys.argv[1:])
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/cli/main_skeleton.py", line 62, in execute
version=args.version, config=config)
AttributeError: 'Namespace' object has no attribute 'version'It seems that --version must be provided, so I tried that:
$ conda skeleton cran fd --version 1.0-12
Tip: install CacheControl and lockfile (conda packages) to cache the CRAN metadata
Fetching metadata from https://cran.r-project.org
Parsing input package fd:
.. name: fd location: None new_location: /ebio/abt3_projects/software/dev/conda_packages/pez/r-fd
Making/refreshing recipe for fd
Tip: install CacheControl and lockfile (conda packages) to cache the CRAN metadata
Leaving build/test directories:
Work: /ebio/abt3_projects/software/dev/miniconda3_dev/conda-bld/skeleton_1520796943854/work
Test: /ebio/abt3_projects/software/dev/miniconda3_dev/conda-bld/skeleton_1520796943854/test_tmp
Leaving build/test environments:
Test: source activate /ebio/abt3_projects/software/dev/miniconda3_dev/conda-bld/skeleton_1520796943854/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac
Build: source activate /ebio/abt3_projects/software/dev/miniconda3_dev/conda-bld/skeleton_1520796943854/_build_env
Traceback (most recent call last):
File "/ebio/abt3_projects/software/dev/miniconda3_dev/bin/conda-skeleton", line 11, in <module>
sys.exit(main())
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/cli/main_skeleton.py", line 66, in main
return execute(sys.argv[1:])
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/cli/main_skeleton.py", line 62, in execute
version=args.version, config=config)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/api.py", line 280, in skeletonize
recursive=recursive, config=config, **kwargs)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/skeletons/cran.py", line 790, in skeletonize
raise NotImplementedError("Package versions from CRAN are not yet implemented")
NotImplementedError: Package versions from CRAN are not yet implementedI can create a separate issue for this if needed.
mingwandroid
commented
6 years ago 
mbargull
commented
6 years ago Yes, I would recommend not following those instructions if you want to have guaranteed access to the latest
condaandconda-buildversions. I think @mbargull is abiocondadeveloper and I know he's very active withcondatoo, so maybe he could give us his opinion on this?
Those instructions are there to make it easier for Bioconda users to just conda create -n env some-bioinformatics-tool without the need to set up those channels for every new environment. But due to the reasons you already stated, I admit it's not optimal in the current situation.
I am recommending you keep your base env as small as possible, containing only conda, conda-build and their dependencies.
I very much agree to that. (E.g., even for anaconda users, I think a Miniconda3 install followed by a conda create -n anaconda3 anaconda is the superior option.)
There are a couple of things that are going to improve but are a bit amiss just now, e.g.,
conda still isn't isolated in a separate _conda_ environment (a feature that is currently put on hold),conda-forge's version of conda is still <4.4, but I expect that to change soon.But even though things ought to improve, there are workarounds you can use now if you want to follow @mingwandroid's advice on using conda and other packages from defaults. E.g.
conda >=4.4 pin packages via history to the defaults channel, e.g.,
conda install -nbase defaults::conda defaults::conda-build defaults::pythonconda config --system --add pinned_packages defaults::conda --add pinned_packages defaults::conda-build
(you don't want to pin defaults::python here as that would also apply to non-base environments; also better use conda >=4.4 with this too)-c defaults in conjunction with -n base, e.g.,
conda update -c defaults -n base conda, conda update -c defaults -n base --allAlso, you don't have to manually write those .condarc files. With conda config you can manipulate those and also specify which file. Excerpt from the help text:
$ docker run --rm -it continuumio/miniconda3 conda config --help
usage: conda config [-h] [--json] [--debug] [--verbose]
[--system | --env | --file FILE]
[...]
Modify configuration values in .condarc. This is modeled after the git
config command. Writes to the user .condarc file (/root/.condarc) by default.
[...]
--system Write to the system .condarc file
(/opt/conda/.condarc). Otherwise writes to the user
config file (/root/.condarc).
--env Write to the active conda environment .condarc file
(<no active environment>). If no environment is
active, write to the user config file
(/root/.condarc).
--file FILE Write to the given file. Otherwise writes to the user
config file (/root/.condarc) or the file path given by
the 'CONDARC' environment variable, if it is set
(default: /root/.condarc).
[...]Hence you can do something like
# create an empty env and populate after channel configuration
conda create -yn new-env
conda activate new-env
conda config --env --add channels defaults --add channels conda-forge --add channels bioconda
conda install some-bioinformatics-toolI think the base environment's .condarc should live in /opt/conda, but due to (probably bad) habit, I just use ~/.condarc for my base environment. @kalefranz can you advise more on this topic?
It depends on how you want to use the installation, e.g., multi- vs. single-user, etc. For most users, I assume the way you use it to be common. conda config even defaults to writing to ~/.condarc.
(For more exotic uses, e.g., if, for whatever reason, you have multiple separate conda installations, you should be aware that ~/.condarc (and other files) apply to all those installations. For further information, take a look at @kalefranz detailed explanations in
https://www.anaconda.com/blog/developer-blog/conda-configuration-engine-power-users/.)
nick-youngblut
commented
6 years ago Thanks for all of the information. My research lab does have multiple conda installations, for a few reasons. This lack of complete isolation between conda installs (eg., the ~/.condarc file) has caused multiple problems for us. Thank you for pointing out https://www.anaconda.com/blog/developer-blog/conda-configuration-engine-power-users/.
As I mentioned above, I did try creating .condarc files in both locations that might be "base-specific", but neither worked. Is there a simple way of having a base-env-specific .condarc, or do I have to use something more complicated as in the conda-configuration-engine-power-users?
There doesn't appear to be any mention of this whole "must use defaults" topic in the documentation for conda-build. Am I just not seeing it?
mbargull
commented
6 years ago My research lab does have multiple conda installations, for a few reasons. This lack of complete isolation between conda installs (eg., the
~/.condarcfile) has caused multiple problems for us.
If you want to prevent ~/.condarc from ever being used, you can do something like
touch ~/.condarc && chmod -w ~/.condarc, i.e., make ~/.condarc write protected. (That's how I use it since I usually also want to isolate all the different installations I have on my machine, e.g., during development of conda itself.)
Is there a simple way of having a base-env-specific .condarc
Sadly, I don't think so. base is a special case since its environment config file is the same as the installation base config file, i.e., $CONDA_PREFIX/.condarc == $CONDA_ROOT/.condarc.
(It's just one more thing to add to the list of what will improve when we move conda into its own non-base environment at some time.)
I think for now you should use one of the workarounds I provided above if you want defaults' packages in base.
There doesn't appear to be any mention of this whole "must use defaults" topic in the documentation for conda-build. Am I just not seeing it?
There isn't necessarily a need to "must use defaults" . It's just that currently things are in a state of flux and thus conda-forge lacks a bit behind on the conda releases. This should resolve itself in near future.
Just keep in mind that defaults is the official channel whereas conda-forge is a community effort. In regards to bioconda, you want to prefer conda-forge though, at least in your non-base workflow/application environments, due to compatibility issues -- but even that is going to improve in future as there are great people at Anaconda, Inc. as well as conda-forge, which thankfully work closely together.
nick-youngblut
commented
6 years ago @mingwandroid:
No need, #2751
It's been many days, so I hoped that the bug fix has been integrated into the newest conda-build. So, I tried updating conda-build with conda update -n root conda-build, which only updated conda:
The following packages will be UPDATED:
conda: 4.4.11-py36_0 --> 4.5.0-py36_0I'm still getting the arg parse error:
$ conda skeleton cran pez --recursive --update-policy overwrite
Traceback (most recent call last):
File "/ebio/abt3_projects/software/dev/miniconda3_dev/bin/conda-skeleton", line 11, in <module>
sys.exit(main())
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/cli/main_skeleton.py", line 66, in main
return execute(sys.argv[1:])
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/cli/main_skeleton.py", line 62, in execute
version=args.version, config=config)
AttributeError: 'Namespace' object has no attribute 'version'Do I need to change my channel priorities or something (currently defaults is top priority)?
$ conda info
active environment : R_pkg_build
active env location : /ebio/abt3_projects/software/dev/miniconda3_dev/envs/R_pkg_build
shell level : 1
user config file : /ebio/abt3/nyoungblut/.condarc
populated config files : /ebio/abt3_projects/software/dev/miniconda3_dev/.condarc
/ebio/abt3/nyoungblut/.condarc
conda version : 4.5.0
conda-build version : 3.7.1
python version : 3.6.4.final.0
base environment : /ebio/abt3_projects/software/dev/miniconda3_dev (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
https://conda.anaconda.org/r/linux-64
https://conda.anaconda.org/r/noarch
https://conda.anaconda.org/qiime2/linux-64
https://conda.anaconda.org/qiime2/noarch
https://conda.anaconda.org/bioconda/linux-64
https://conda.anaconda.org/bioconda/noarch
package cache : /ebio/abt3_projects/software/dev/miniconda3_dev/pkgs
/ebio/abt3/nyoungblut/.conda/pkgs
envs directories : /ebio/abt3_projects/software/dev/miniconda3_dev/envs
/ebio/abt3/nyoungblut/.conda/envs
platform : linux-64
user-agent : conda/4.5.0 requests/2.18.4 CPython/3.6.4 Linux/4.4.67 ubuntu/16.04 glibc/2.23
UID:GID : 6354:350
netrc file : None
offline mode : FalseJust apply the patch yourself:
pushd $CONDA_PREFIX/lib/python3.6/site-packages
wget -c https://patch-diff.githubusercontent.com/raw/conda/conda-build/pull/2751.patch
patch -p1 < https://patch-diff.githubusercontent.com/raw/conda/conda-build/pull/2751.patchWe will be releasing a new conda-build soon. I have been sick, and we're backlogged.
msarahan
commented
6 years ago conda-build 3.7.2 is tagged. Packages are being uploaded now and should be available shortly.
nick-youngblut
commented
6 years ago Great! Thanks for letting me know!
nick-youngblut
commented
6 years ago I'm still getting the same DESCRIPTION error for the fd package with conda-build v3.7.2.
The error:
+GFORTRAN=/ebio/abt3_projects/software/dev/miniconda3_dev/conda-bld/r-fd_1522309056239/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
Fatal error: cannot open file 'CMD': No such file or directory
Traceback (most recent call last):
File "/ebio/abt3_projects/software/dev/miniconda3_dev/bin/conda-build", line 11, in <module>
sys.exit(main())
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 420, in main
execute(sys.argv[1:])
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 411, in execute
verify=args.verify)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/api.py", line 199, in build
notest=notest, need_source_download=need_source_download, variants=variants)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/build.py", line 2097, in build_tree
notest=notest,
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/build.py", line 1370, in build
utils.check_call_env(cmd, env=env, cwd=src_dir, stats=build_stats)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/utils.py", line 301, in check_call_env
return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/lib/python3.6/site-packages/conda_build/utils.py", line 280, in _func_defaulting_env_to_os_environ
raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/ebio/abt3_projects/software/dev/miniconda3_dev/conda-bld/r-fd_1522309056239/work/conda_build.sh']' returned non-zero exit status 2The problematic DESCRIPTION file:
Package: FD
Type: Package
Title: Measuring functional diversity (FD) from multiple traits, and
other tools for functional ecology
Version: 1.0-12
Date: 2014-19-08
Binary file DESCRIPTION.old matchesConda info:
$ conda info
active environment : R_pkg_build
active env location : /ebio/abt3_projects/software/dev/miniconda3_dev/envs/R_pkg_build
shell level : 1
user config file : /ebio/abt3/nyoungblut/.condarc
populated config files : /ebio/abt3_projects/software/dev/miniconda3_dev/.condarc
/ebio/abt3/nyoungblut/.condarc
conda version : 4.5.0
conda-build version : 3.7.2
python version : 3.6.4.final.0
base environment : /ebio/abt3_projects/software/dev/miniconda3_dev (writable)
channel URLs : https://conda.anaconda.org/bioconda/linux-64
https://conda.anaconda.org/bioconda/noarch
https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
https://conda.anaconda.org/r/linux-64
https://conda.anaconda.org/r/noarch
https://conda.anaconda.org/qiime2/linux-64
https://conda.anaconda.org/qiime2/noarch
package cache : /ebio/abt3_projects/software/dev/miniconda3_dev/pkgs
/ebio/abt3/nyoungblut/.conda/pkgs
envs directories : /ebio/abt3_projects/software/dev/miniconda3_dev/envs
/ebio/abt3/nyoungblut/.conda/envs
platform : linux-64
user-agent : conda/4.5.0 requests/2.18.4 CPython/3.6.4 Linux/4.4.67 ubuntu/16.04 glibc/2.23
UID:GID : 6354:350
netrc file : None
offline mode : FalseAny idea what's wrong with the description file? I'm already changing many of the build.sh files for the dependency R packages in order for conda-build to work (changing $R to R).
github-actions[bot]
commented
2 years ago Hi there, thank you for your contribution!
This issue has been automatically locked because it has not had recent activity after being closed.
Please open a new issue if needed.
Thanks!
I'm trying to create a conda package for the
FDR package. I used the following:I got the following error:
The DESCRIPTION file (
/ebio/abt3_projects/software/dev/miniconda3_dev/conda-bld/r-fd_1520781448380/work/DESCRIPTION) contains the following:The meta.yaml file for the r-fd/ seems to be OK:
I don't see what's causing the oddly-formatted DESCRIPTION file, and thus the error.