keenajm1 commented 1 year ago

Hello,

I'm sorry for posting yet another installation troubleshooting request, but I've tried browsing the solved and unsolved issue boards and still just can't find a solution.

My Mac has an M2 Max Chip and I therefore tried to install the biostars software with the newest version of x86_64 miniconda3, which I installed via the script. (I should note here that I have tried to install the software like 3 different times using Anaconda and miniconda, both ARM and x86_64). It would seem that regardless of what iteration of conda I choose, I can't install the biostars software.

I believe I have been going through the prerequisite steps appropriately, but when I use the installation script, I get arrested at step 4/10, where the terminal outputs the error I have attached below:

I have also tried downloading both samtools and htslib directly via conda install samtools/htslib. It doesn't seem to do much, as I can see that they are both downloaded, but I still get the same error. I have also tried downloading the most recent versions of both that are mentioned in the prompt (1.17), but it leads to further version conflicts with libdeflate, libzlib, and lib.

I do suffer from some computer science illiteracy and the fact that I've done this several times has muddled things together, but I do know for certain that no matter what I've done, I've been stopped at step 4 with the claim of these packages being an issue.

On a side note, on my most recent installation using miniconda, I'm struggling with permission issues. I got a permission issue in step 3, and I can't activate the "bioconda" environment. I was able to in past tries. This permission issue is a real annoyance, as I can't download base r from the console anymore. I have had to use sudo conda install on my most recent attempt to install samtools and htslib, which doesn't seem proper.

Anyways, I apologize for piling on with another troubleshooting issue. If the answer is I simply need to be patient since these M2 chips are very new, I understand. I want to dive into the biostars material, but I can't even get myself to square one. Any help is greatly appreciated and thank you for taking the time!

ialbert commented 1 year ago

The ever-changing Mac platforms are a problem.

I have recently (as in a few days ago) rewritten the installation instructions to use micromamba installer.

Micromamba is a standalone variant of conda/mamba that is more robust.

Give micromamba a go but always use the Intel x86_64 version of it. Bioinformatics software is only compiled for Intel and always needs to be run in emulation.

This week a cohort of new students will go through the book and as such the various pitfalls will come out and hopefully get correct.

Meshesha78 commented 10 months ago

Hi, same problem here with a similar error message. Fortunately, I have a Windows PC and working ok with Ubuntu. Let me know if you solve it.

Thanks,

Mes

ialbert commented 10 months ago

We went through the installation with a cohort of 30 students. Half had fairly new Apple laptops.

In all cases where we had troubles (and that was about 3 people), the problem was always that installation instructions were not followed strictly or that the student already had another version of conda (sometimes anaconda or miniconda) already installed.

Make sure to activate bash, install the dependencies on MacOS, and so on.

Installing micromamba is not bioinformatics-related task and as far as I know, tens of thousands of people use it successfully on M2. Basically what I am saying here is that to best of my knowledge it works well.

Meshesha78 commented 10 months ago

Thanks a lot, I see the problem. I have anaconda3 already installed which may be creating the problem. Will remove it and try again. Thanks again

Mes

ialbert commented 10 months ago

if you have anaconda you might want to try using that instead of micromamba it might work,

or start invoking micromamba directly to bypass the existing setup, invoke it as:

~/bin/micromamba

these tools are very similar but can compete with one another

Meshesha78 commented 10 months ago

Thank you very much I did try with anaconda and still did not work. I have pasted the entire process below, I am sorry for the mess, please see if there is anything i did wrong. I appreciate it, thank you.

Last login: Wed Jan 3 14:03:54 on ttys000

@.*** ~

$ mkdir -p ~/bin

@.*** ~

$ curl -Ls https://micromamba.snakepit.net/api/micromamba/osx-64/latest | tar -xj bin/micromamba

@.*** ~

$

@.*** ~

$ ./bin/micromamba shell init -s bash -p ~/micromamba -q

@.*** ~

$ curl -s http://data.biostarhandbook.com/install/biostar.sh > ~/.biostar.sh

@.*** ~

$ echo "source ~/.biostar.sh" >> ~/.bashrc

@.*** ~

$ echo "source ~/.bashrc" >> ~/.bash_profile

@.*** ~

$ conda config prepend channels conda-forge

@.*** ~

$ conda config append channels bioconda

@.*** ~

$ conda config set channel_priority strict

@.*** ~

$ conda create -n bioinfo -y python=3.8

conda-forge/osx-64 Using cache

conda-forge/noarch Using cache

bioconda/osx-64 Using cache

bioconda/noarch Using cache

pkgs/main/osx-64 No change

pkgs/r/osx-64 No change

pkgs/main/noarch No change

pkgs/r/noarch No change

Transaction

Prefix: /Users/mesfinmeshesha/micromamba/envs/bioinfo

Updating specs:

python=3.8

Package Version Build Channel Size

─────────────────────────────────────────────────────────────────────────────

Install:

─────────────────────────────────────────────────────────────────────────────

xz 5.2.6 h775f41a_0 conda-forge Cached
ncurses 6.4 h93d8f39_2 conda-forge Cached
bzip2 1.0.8 h10d778d_5 conda-forge Cached
libffi 3.4.2 h0d85af4_5 conda-forge Cached
libzlib 1.2.13 h8a1eda9_5 conda-forge Cached
ca-certificates 2023.11.17 h8857fd0_0 conda-forge Cached
readline 8.2 h9e318b2_1 conda-forge Cached
tk 8.6.13 h1abcd95_1 conda-forge Cached
libsqlite 3.44.2 h92b6c6a_0 conda-forge Cached
openssl 3.2.0 hd75f5a5_1 conda-forge Cached
python 3.8.18 h5ba8234_1_cpython conda-forge Cached
wheel 0.42.0 pyhd8ed1ab_0 conda-forge Cached
setuptools 69.0.3 pyhd8ed1ab_0 conda-forge Cached
pip 23.3.2 pyhd8ed1ab_0 conda-forge Cached

Summary:

Install: 14 packages

Total download: 0 B

─────────────────────────────────────────────────────────────────────────────

Transaction starting

Linking xz-5.2.6-h775f41a_0

Linking ncurses-6.4-h93d8f39_2

Linking bzip2-1.0.8-h10d778d_5

Linking libffi-3.4.2-h0d85af4_5

Linking libzlib-1.2.13-h8a1eda9_5

Linking ca-certificates-2023.11.17-h8857fd0_0

Linking readline-8.2-h9e318b2_1

Linking tk-8.6.13-h1abcd95_1

Linking libsqlite-3.44.2-h92b6c6a_0

Linking openssl-3.2.0-hd75f5a5_1

Linking python-3.8.18-h5ba8234_1_cpython

Linking wheel-0.42.0-pyhd8ed1ab_0

Linking setuptools-69.0.3-pyhd8ed1ab_0

Linking pip-23.3.2-pyhd8ed1ab_0

Transaction finished

To activate this environment, use:

micromamba activate bioinfo

Or to execute a single command in this environment, use:

micromamba run -n bioinfo mycommand

@.*** ~

$ conda activate bioinfo

(bioinfo)

@.*** ~

$ curl -s http://data.biostarhandbook.com/install/conda.txt | xargs conda install -q -y

Retrieving notices: ...working... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): repo.anaconda.com:443

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): repo.anaconda.com:443

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): conda.anaconda.org:443

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/notices.json HTTP/1.1" 404 None

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/r/notices.json HTTP/1.1" 404 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/notices.json HTTP/1.1" 404 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /bioconda/notices.json HTTP/1.1" 404 None

done

Collecting package metadata (current_repodata.json): ...working... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): conda.anaconda.org:443

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): conda.anaconda.org:443

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): repo.anaconda.com:443

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): conda.anaconda.org:443

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): repo.anaconda.com:443

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/noarch/current_repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/r/osx-arm64/current_repodata.json HTTP/1.1" 304 0

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/r/noarch/current_repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/osx-arm64/current_repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /bioconda/osx-arm64/current_repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/osx-arm64/current_repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/noarch/current_repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /bioconda/noarch/current_repodata.json HTTP/1.1" 200 None

done

Solving environment: ...working... unsuccessful initial attempt using frozen solve. Retrying with flexible solve.

Collecting package metadata (repodata.json): ...working... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): conda.anaconda.org:443

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): conda.anaconda.org:443

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): repo.anaconda.com:443

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/noarch/repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/r/osx-arm64/repodata.json HTTP/1.1" 304 0

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/osx-arm64/repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/r/noarch/repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /bioconda/osx-arm64/repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/noarch/repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/osx-arm64/repodata.json HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /bioconda/noarch/repodata.json HTTP/1.1" 200 None

done

Solving environment: ...working... unsuccessful initial attempt using frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

blast
bcftools
bioawk
csvtk
aria2
samtools[version='>=1.14']
freebayes
minimap2
bowtie2
fastp
emboss
datamash
hisat2
bwa
ucsc-bedgraphtobigwig
mafft
seqtk
bedtools
seqkit
subread

Current channels:

To search for alternate channels that may provide the conda package you're

looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

(bioinfo)

@.*** ~

$ pip install bio --upgrade

Collecting bio

Using cached bio-1.6.0-py3-none-any.whl.metadata (5.1 kB)

Collecting biopython>=1.80 (from bio)

Downloading biopython-1.82-cp38-cp38-macosx_10_9_x86_64.whl.metadata (13 kB)

Collecting requests (from bio)

Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)

Collecting tqdm (from bio)

Using cached tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)

Collecting mygene (from bio)

Using cached mygene-3.2.2-py2.py3-none-any.whl (5.4 kB)

Collecting pandas (from bio)

Using cached pandas-2.0.3-cp38-cp38-macosx_10_9_x86_64.whl.metadata (18 kB)

Collecting pooch (from bio)

Using cached pooch-1.8.0-py3-none-any.whl.metadata (9.9 kB)

Collecting gprofiler-official (from bio)

Using cached gprofiler_official-1.0.0-py3-none-any.whl (9.3 kB)

Collecting numpy (from biopython>=1.80->bio)

Using cached numpy-1.24.4-cp38-cp38-macosx_10_9_x86_64.whl.metadata (5.6 kB)

Collecting biothings-client>=0.2.6 (from mygene->bio)

Using cached biothings_client-0.3.1-py2.py3-none-any.whl.metadata (9.8 kB)

Collecting python-dateutil>=2.8.2 (from pandas->bio)

Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)

Collecting pytz>=2020.1 (from pandas->bio)

Using cached pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB)

Collecting tzdata>=2022.1 (from pandas->bio)

Downloading tzdata-2023.4-py2.py3-none-any.whl.metadata (1.4 kB)

Collecting platformdirs>=2.5.0 (from pooch->bio)

Using cached platformdirs-4.1.0-py3-none-any.whl.metadata (11 kB)

Collecting packaging>=20.0 (from pooch->bio)

Using cached packaging-23.2-py3-none-any.whl.metadata (3.2 kB)

Collecting charset-normalizer<4,>=2 (from requests->bio)

Using cached charset_normalizer-3.3.2-cp38-cp38-macosx_10_9_x86_64.whl.metadata (33 kB)

Collecting idna<4,>=2.5 (from requests->bio)

Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)

Collecting urllib3<3,>=1.21.1 (from requests->bio)

Using cached urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)

Collecting certifi>=2017.4.17 (from requests->bio)

Using cached certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)

Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas->bio)

Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)

Using cached bio-1.6.0-py3-none-any.whl (279 kB)

Downloading biopython-1.82-cp38-cp38-macosx_10_9_x86_64.whl (2.7 MB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 5.9 MB/s eta 0:00:00

Using cached pandas-2.0.3-cp38-cp38-macosx_10_9_x86_64.whl (11.7 MB)

Using cached pooch-1.8.0-py3-none-any.whl (62 kB)

Using cached requests-2.31.0-py3-none-any.whl (62 kB)

Using cached tqdm-4.66.1-py3-none-any.whl (78 kB)

Using cached biothings_client-0.3.1-py2.py3-none-any.whl (29 kB)

Using cached certifi-2023.11.17-py3-none-any.whl (162 kB)

Using cached charset_normalizer-3.3.2-cp38-cp38-macosx_10_9_x86_64.whl (121 kB)

Using cached idna-3.6-py3-none-any.whl (61 kB)

Using cached numpy-1.24.4-cp38-cp38-macosx_10_9_x86_64.whl (19.8 MB)

Using cached packaging-23.2-py3-none-any.whl (53 kB)

Using cached platformdirs-4.1.0-py3-none-any.whl (17 kB)

Using cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)

Downloading tzdata-2023.4-py2.py3-none-any.whl (346 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 346.6/346.6 kB 8.0 MB/s eta 0:00:00

Using cached urllib3-2.1.0-py3-none-any.whl (104 kB)

Installing collected packages: pytz, urllib3, tzdata, tqdm, six, platformdirs, packaging, numpy, idna, charset-normalizer, certifi, requests, python-dateutil, biopython, pooch, pandas, gprofiler-official, biothings-client, mygene, bio

Successfully installed bio-1.6.0 biopython-1.82 biothings-client-0.3.1 certifi-2023.11.17 charset-normalizer-3.3.2 gprofiler-official-1.0.0 idna-3.6 mygene-3.2.2 numpy-1.24.4 packaging-23.2 pandas-2.0.3 platformdirs-4.1.0 pooch-1.8.0 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.31.0 six-1.16.0 tqdm-4.66.1 tzdata-2023.4 urllib3-2.1.0

(bioinfo)

@.*** ~

$ yes no | sh -c "$(curl -fsSL ftp://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/install-edirect.sh)" > /dev/null

(bioinfo)

@.*** ~

$ curl -s http://data.biostarhandbook.com/install/sratools.sh | bash

(bioinfo)

@.*** ~

$ mkdir -p ~/bin

(bioinfo)

@.*** ~

$ curl -s http://data.biostarhandbook.com/install/doctor.py > ~/bin/doctor.py

(bioinfo)

@.*** ~

$ chmod +x ~/bin/doctor.py

(bioinfo)

@.*** ~

$ ~/bin/doctor.py

Doctor! Doctor! Give me the news.

Checking symptoms ...

bwa --> ERROR

datamash --> ERROR

fastqc -h --> ERROR

hisat2 --> ERROR

seqret -h --> ERROR

featureCounts --> ERROR

efetch ... OK

esearch ... OK

samtools --> ERROR

fastq-dump --> ERROR

bowtie2 --> ERROR

bcftools --> ERROR

seqtk --> ERROR

seqkit --> ERROR

bio ... OK

fastq-dump -X 1 -Z SRR1553591 --> ERROR

#

Found some problems

#

Unable to run: bwa, datamash, fastqc -h, hisat2, seqret -h,

featureCounts, samtools, fastq-dump, bowtie2, bcftools, seqtk, seqkit, fastq-dump -X 1 -Z SRR1553591

#

Check the Biostar Handbook for installation help.

#

(bioinfo)

@.*** ~

$

On Wed, Jan 3, 2024 at 12:44 PM Istvan Albert @.***> wrote:

if you have anaconda you might want to try using that instead of micromamba it might work,

or start invoking micromamba directly to bypass the existing setup, invoke it as:

~/bin/micromamba

these tools are very similar but can compete with one another

— Reply to this email directly, view it on GitHub https://github.com/biostars/biostar-handbook/issues/282#issuecomment-1875739640, or unsubscribe https://github.com/notifications/unsubscribe-auth/AX6LDMLQEYQ7I7F7CTLCU4DYMWKJBAVCNFSM6AAAAAA3XQ7LKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVG4ZTSNRUGA . You are receiving this because you commented.Message ID: @.***>

-- __//__

Mesfin Meshesha (Ph.D)

Senior Lab Scientist

Cell: 443 648 6609

*mail: @. @.>*

@. @.> *

ialbert commented 10 months ago

You have installed the incorrect version of micromamba, make sure to follow the manual.

You have to install the Intel based micromamba even on the M1 processor.

It will work seamlessly and just fine. Bioinformatics software is available only for Intel based Macs and will run in emulation mode on M1.

You will need to uninstall then follow the book closely.

curl http://data.biostarhandbook.com/uninstall.sh | bash

then do the installation again

Meshesha78 commented 10 months ago

Thank you, it works now!!

Mes

On Wed, Jan 3, 2024 at 3:24 PM Istvan Albert @.***> wrote:

You have installed the incorrect version of micromamba, make sure to follow the manual.

You have to install the Intel based micromamba even on the M1 processor.

It will work seamlessly and just fine. Bioinformatics software is available only for Intel based Macs and will run in emulation mode on M1.

You will need to uninstall the follow the book:

curl http://data.biostarhandbook.com/uninstall.sh | bash

then do the installation again

— Reply to this email directly, view it on GitHub https://github.com/biostars/biostar-handbook/issues/282#issuecomment-1875925778, or unsubscribe https://github.com/notifications/unsubscribe-auth/AX6LDMJXBH6IR2SDMEVQG23YMW467AVCNFSM6AAAAAA3XQ7LKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVHEZDKNZXHA . You are receiving this because you commented.Message ID: @.***>

-- __//__

Mesfin Meshesha (Ph.D)

Senior Lab Scientist

Cell: 443 648 6609

*mail: @. @.>*

@. @.> *

biostars / biostar-handbook

M2 MacOS Installation Troubles #282

Doctor! Doctor! Give me the news.

Checking symptoms ...

bwa --> ERROR

datamash --> ERROR

fastqc -h --> ERROR

hisat2 --> ERROR

seqret -h --> ERROR

featureCounts --> ERROR

efetch ... OK

esearch ... OK

samtools --> ERROR

fastq-dump --> ERROR

bowtie2 --> ERROR

bcftools --> ERROR

seqtk --> ERROR

seqkit --> ERROR

bio ... OK

fastq-dump -X 1 -Z SRR1553591 --> ERROR

Found some problems

Unable to run: bwa, datamash, fastqc -h, hisat2, seqret -h,

Check the Biostar Handbook for installation help.