search

WatsonLab / MAGpy

Snakemake pipeline for downstream analysis of metagenome-assembled genomes (MAGs) (pronounced mag-pie)
62 stars 23 forks source link

miniconda permission/version error #7

Closed GeoMicroSoares closed 4 years ago

GeoMicroSoares commented 5 years ago

Hi there,

Managed to follow the 10min install and complete it but get to test the install with snakemake --use-conda -s MAGpy test and get the following:

$ snakemake --use-conda -s MAGpy test
Building DAG of jobs...
Creating conda environment envs/ete3.yaml...
Downloading remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /media/andre/B2F8C9A0F8C962E9/SGG_metagenomes/data/MAGpy/envs/ete3.yaml:
Fetching package metadata ..........An unexpected error has occurred.
Please consider posting the following information to the
conda GitHub issue tracker at:

    https://github.com/conda/conda/issues

Current conda install:

               platform : linux-64
          conda version : 4.3.30
       conda is private : False
      conda-env version : 4.3.30
    conda-build version : not installed
         python version : 2.7.13.final.0
       requests version : 2.12.4
       root environment : /home/andre/miniconda2  (read only)
    default environment : /home/andre/miniconda3/envs/magpy_install
       envs directories : /home/andre/.conda/envs
                          /home/andre/miniconda2/envs
          package cache : /home/andre/miniconda2/pkgs
                          /home/andre/.conda/pkgs
           channel URLs : https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : None
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.30 requests/2.12.4 CPython/2.7.13 Linux/4.4.6-040406-generic debian/jessie/sid glibc/2.19    
                UID:GID : 1000:1000

`$ /home/andre/miniconda2/bin/conda-env create --file /media/andre/B2F8C9A0F8C962E9/SGG_metagenomes/data/MAGpy/.snakemake/conda/d1cdcc3a.yaml --prefix /media/andre/B2F8C9A0F8C962E9/SGG_metagenomes/data/MAGpy/.snakemake/conda/d1cdcc3a`

    Traceback (most recent call last):
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda/exceptions.py", line 640, in conda_exception_handler
        return_value = func(*args, **kwargs)
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda_env/cli/main_create.py", line 108, in execute
        installer.install(prefix, pkg_specs, args, env)
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda_env/installers/conda.py", line 29, in install
        prefix=prefix)
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda/core/index.py", line 101, in get_index
        index = fetch_index(channel_priority_map, use_cache=use_cache)
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda/core/index.py", line 120, in fetch_index
        repodatas = collect_all_repodata(use_cache, tasks)
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda/core/repodata.py", line 75, in collect_all_repodata
        repodatas = _collect_repodatas_serial(use_cache, tasks)
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda/core/repodata.py", line 485, in _collect_repodatas_serial
        for url, schan, pri in tasks]
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda/core/repodata.py", line 115, in func
        res = f(*args, **kwargs)
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda/core/repodata.py", line 467, in fetch_repodata
        touch(cache_path)
      File "/home/andre/miniconda2/lib/python2.7/site-packages/conda/gateways/disk/update.py", line 64, in touch
        utime(path, None)
    OSError: [Errno 13] Permission denied: '/home/andre/miniconda2/pkgs/cache/9ca791dd.json'

So not only is it unable to get permission to certain folders, it's also weirdly calling my old miniconda2 install during this. Any help?

mw55309 commented 5 years ago

Hey

So this sounds like a snakemake/conda issue rather than a MAGpy one, but I can try and help out

Are you running snakemake from within an env? If you type "which conda" into the terminal does it still point to your old miniconda? Can you create the ete3 environment yourself? i.e.

conda env create -f MAGpy/envs/ete3.yaml

Unfortunately conda has become quite unstable recently

GeoMicroSoares commented 5 years ago

Hey there @mw55309 , thanks for the help.

Running snakemake from the magpy_install environment created with the 10min install yep.

All you asked for seems to make sense:

$which conda
/home/andre/miniconda3/envs/magpy_install/bin/conda
$ conda env create -f MAGpy/envs/ete3.yaml
Collecting package metadata: done
Solving environment: done

Downloading and Extracting Packages
numpy-1.16.2         | 4.3 MB    | ########################################################################################## | 100% 
setuptools-40.8.0    | 625 KB    | ########################################################################################## | 100% 
certifi-2019.3.9     | 149 KB    | ########################################################################################## | 100% 
six-1.12.0           | 22 KB     | ########################################################################################## | 100% 
gobject-introspectio | 1.2 MB    | ########################################################################################## | 100% 
pip-19.0.3           | 1.8 MB    | ########################################################################################## | 100% 
lxml-4.3.2           | 1.5 MB    | ########################################################################################## | 100% 
sip-4.18.1           | 277 KB    | ########################################################################################## | 100% 
pyqt-5.6.0           | 5.4 MB    | ########################################################################################## | 100% 
wheel-0.33.1         | 34 KB     | ########################################################################################## | 100% 
scipy-1.2.1          | 17.7 MB   | ########################################################################################## | 100% 
python-3.7.1         | 36.4 MB   | ########################################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate ete3
#
# To deactivate an active environment, use
#
#     $ conda deactivate
mw55309 commented 5 years ago

I suspect the failure might have been a one-off conda gremlin - might be worth trying again?

How is your $HOME/.condarc? I am not sure where Snakemake is picking up your old conda environment from, but it might worth removing any reference to your miniconda2 installation in the .condarc

GeoMicroSoares commented 5 years ago

Tried it again with the same result... Weirdly enough, no .condarc around here! Should I create one?

mw55309 commented 5 years ago

There is definitely something weird going on in your set up in that both the envs and packages directories are pointing to your miniconda2 environment:

       __root environment : /home/andre/miniconda2  (read only)__
    default environment : /home/andre/miniconda3/envs/magpy_install
       envs directories : /home/andre/.conda/envs
                          __/home/andre/miniconda2/envs__
          package cache : __/home/andre/miniconda2/pkgs__
                          /home/andre/.conda/pkgs

I don't really know what to do here as this isn't a MAGpy issue, it is some weird Snakemake/Conda issue

mw55309 commented 5 years ago

condarc is supposed to tell conda where to store envs, where to download packages, what channels to use etc

https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html

The issue is if you begin f*cking with conda, all your other envs might get messed up

Can you run MAGpy as a different user? (with a clean conda install?)

GeoMicroSoares commented 5 years ago

Yeah, proper weird stuff. I'll try and get some help from them somehow. Looking forward to solving this and trying out MAGpy though. Thought of creating something like a docker image to run MAGpy? Might be good to avoid all the conda-related problems for example. All the best!

*Edit, didn't see your second reply!

I'll try giving the docs a look and running it as a different user yep - thanks!