Lecture 1-Biostar Handbook- Doctor.py issue

[LPain]

Hello,

I am reading the lecture 1 until running the doctor.py srcipt (Windows 10), which doesn't work :/ Here is what my terminal shows:

"bash: /root/bin/doctor.py: /usr/bin/python: bad interpreter: No such file or directory"

So I did: "ls" which gave "Miniconda3-latest-Linux-x86_64.sh bash_profile bin miniconda3"

When I choose bin ("cd bin" then "ls"), it gave me: "doctor.py"

...

Any idea of what it could be happening with the script doctor?

Many thanks in advance!

[ialbert]

First question. Have you followed the instructions for Windows:

https://www.biostarhandbook.com/setup/setup.html

If so what gets printed when you type:

python --version

and

which python

These instructions will tell you that python has been installed correctly.

In addition, I can see that you are not running the scripts from the correct location. When you set up Windows Bash it will create a home directory for you and that should not be /root/.

It asks you a username when it installs itself.

[LPain]

Thank you for answering,

Yes I have followed the instructions for Windows via https://www.biostarhandbook.com/setup/setup.html

When I type : python --version "Python 3.6.1 :: Continuum Analytics, Inc." and which python /root/miniconda3/bin/python

Moreover, when i installed Windows Bash, i was typing the username I choose but 2 sec after, I didn't get the possibility to type a passworld since it was not asked and "root@Inspiron3521" appeared at the top of my terminal...

[ialbert]

This looks like it has installed your user as a root environment. I am not quite certain why that happened - it is not a good habit to have it that way since one can accidentally overwrite essential files that otherwise would be protected.

I have added an update to the doctor.py that I believe makes it work under a root environment as well. Reinstall doctor.py it with:

mkdir -p ~/bin
curl http://data.biostarhandbook.com/install/doctor.py > ~/bin/doctor.py
chmod +x ~/bin/doctor.py

And let me know whether it works.

I believe that the software will and ought to work this way as well.

That being said I even if this works I would recommend reinstalling everything (there are instructions on reinstalling Windows Bash). Of course reinstalling can be annoying. But in general it is not a good habit to run tools as root - one mitigating factor is that at least in Windows Bash you can't really mess up Windows itself since it is isolated from it.

https://www.biostarhandbook.com/setup/setup-windows.html

[LPain]

Thank you very much for these explanations, I will write a post to say if everythings works properly

[LPain]

Hello,

I re-install everything until conda and the line: "conda create -y --name bioinfo python=3" At this line, I have an error message: "Fetching package metadata ... CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://conda.anaconda.org/biostar/linux-64/repodata.json Elapsed: -

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError (MaxRetryError('HTTPSConnextionPool(host=\'conda.anaconda.org\', port 443):Max retries exceeded with yrl: /bioconda/linux-64/repodata.json(Caused by SSLError (''Can\'t connect to HTTPS URL because the SSL module is not available. '',))',),)

I tried 3 times but it still have this message. I searched for solutions for similar situations but did not find any solutions on github...

Any idea of how to fix this situation?

Many thanks in advance,

[ialbert]

You seem to have closed this issue. Was the problem solved?

[LPain]

No, it was not since i couldn't execute "conda create -y --name bioinfo python=3" At this line, I have an error message: "Fetching package metadata ... CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://conda.anaconda.org/biostar/linux-64/repodata.json Elapsed: -

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError (MaxRetryError('HTTPSConnextionPool(host='conda.anaconda.org', port 443):Max retries exceeded with yrl: /bioconda/linux-64/repodata.json(Caused by SSLError (''Can't connect to HTTPS URL because the SSL module is not available. '',))',),)

I tried 3 times but it still have this message. I searched for solutions for similar situations but did not find any solutions on github...

[ialbert]

This is a difficult issue to diagnose. I have had someone else have the same problem - I am not sure exactly what causes it. I suspect that the Linux setup commands have not all been executed correctly. Can you ensure that everything here has been executed

https://www.biostarhandbook.com/setup/setup-linux.html

Then I would delete my miniconda3 directory.

rm -rf miniconda3

Then reinstall conda as instructed here:

https://www.biostarhandbook.com/setup/setup-conda.html

[LPain]

I reinstalled the Linux set up commands as adviced Then I deleted my miniconda3 directory and reinstall conda. When I typed: "conda config --add channels r" I got: "Warning: 'r' already in 'channels' list, moving to the top"

"conda config --add channels conda-forge" I got: "Warning: 'conda-forge' already in 'channels' list, moving to the top"

conda config --add channels bioconda I got "Warning: 'bioconda' already in 'channels' list, moving to the top"

I tried: source deactivate conda update conda I got "Fetching package metadata ... CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/biostar/linux-64/repodata.json Elapsed: -

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError (MaxRetryError('HTTPSConnextionPool(host='conda.anaconda.org', port 443):Max retries exceeded with yrl: /bioconda/linux-64/repodata.json(Caused by SSLError (''Can't connect to HTTPS URL because the SSL module is not available. '',))',),)

[LPain]

I typed the Doctor.py script and you can see below the answer: $ ~/bin/doctor.py

Doctor! Doctor! Give me the news.

Checking 15 symptoms...

Missing programm: bwa

Missing programm: datamash

Missing programm: fastqc

Missing programm: hisat2

Missing programm: seqret

Missing programm: subread-align

Missing programm: featureCounts

Missing programm: R

Missing programm: efetch

Missing programm: esearch

Missing programm: samtools

Missing programm: fastq-dump

Optional program not found: global-align.sh

Optional program not found: local-align.sh

Your system shows 12 errors.

See also: doctor.py --fixme

I don't know if it can help

[ialbert]

Somehow your python installation became completely broken on your system.

The problem is not about bioinformatics tools. Conda itself does not work at all because the Python that you have installed is missing a library.

This is very difficult to troubleshoot I am not sure what happened there. You need to remove and reinstall python or remove an reinstall everything from scratch but this time paying attention on what happens.

I think when you installed into the root directory things may have went bad.

I would remove the Unix Bash as shown here

https://www.howtogeek.com/261188/how-to-uninstall-or-reinstall-windows-10s-ubuntu-bash-shell/

then start new with Linux installation but make sure to pick a simple username when you install Linix the first time around.

[HolyRomulus]

Based on the dates listed in the Miniconda Archive, it looks like conda was recently updated. Could that be the source of the issue? I recently started my Ubuntu installation from scratch after getting the same doctor.py results and a error message after attempting to run fastqc with the text "CondaError: OSError(22, invalid argument)" and a file path referring to my Python location. Now I am also getting the same "CondaHTTP" error as the original poster here.

python --version output: Python 3.6.2 :: Anaconda, Inc.
which python output: /home/unix/miniconda3/bin/python

Based on this I had a few questions for clarification:

1). Where should the the linked "unix" directory that is visible to both Linux and Windows be located? My default directory in Ubuntu is /home/username. When following the course instructions the folder is created at /home/username/unix, but the text on page said it would be in our home directory so I made it at /home/unix using a sudo command. Should the file be located at /home/username/unix instead?

2). Irrespective of 1), should our bash_profile, bashrc, & conda files be nested within the linked unix folder? i.e. should I append the bash_profile.txt to /home/unix/.bash_profile (or /home/username/unix/.bash_profile if my assumption in 1) was incorrect).

[ialbert]

The role of the linked directory it to have a location is visible in Windows no other purpose.

The book does not instruct you to create a directory with sudo. If you did need sudo then we would make that very clear since sudo and running commands as root have the potential to mess up your system for good. So I cannot comment on creating a new home directory called unix since that is not what we ask people to do.

I do see a source of potential confusion that the instructions should clarify. There is a unix username and a windows username.

In the command

 mkdir -p '/mnt/c/Users/ialbert/Desktop/unix'

the ialbert is my Windows Username. If you visit your Computer from Windows Explorer (even before you created the Bash) you could see the path to your Desktop from Explorer.

So a path that in Windows for me shows up as:

C:\Users\ialbert\Desktop

will be visible in Bash as:

/mnt/c/Users/ialbert/Desktop/

Note that this path is a valid Windows location that is not Bash specific. It existed before installing Bash. It is visible in bash under a different name but that's about it.

You should NOT use this directory for anything else other than the stated purpose of creating files that you might want to have visible from both Unix and Windows. In those directorie we are mixing two different operating systems there and that can cause problems.

Now to the conda error.

I still believe that if conda does not work at all it has to do with the way Windows Bash was set up. As I mentioned before conda is not a bioinformatics tool - it is a generic tool used by thousands of people in various situations. It is hard to believe (though not impossible as I learned) that there is something wrong with conda. My first suggestion is to wipe everything clean, reinstall bash. Follow the Linux setup very closely, then install conda.

The conda error clearly states that there is a missing Python SSL library. So it is not even a conda error, it comes from Python programming language. And that happens when SSL was not installed properly. Now since the entire internet runs on HTTPS (and SSL is part of that) it is very hard to conceive that when an installation is performed correctly the SSL is missing - nothing internet related would work on that system - as indicated by your error.

Troubleshooting this is very difficult since it basically goes to the "core" of the system. If it is set up right conda works as it should so it is impossible to reproduce the error. Wiping a system means losing all existing files and start with an empty system, it is very tedious to get one's computer wiped totally to reproduce an error that might not be a real error.

But you do make a good point that there is a new conda installer - and that could explain some problems. I will see if I can manage to do a full wipe this week and try to reproduce the problem.

[ialbert]

I have now investigated with a fresh install and I can confirm that the problem exists and is valid. It is also reported at, and it is indeed a problem with laters conda and the Windows Subsystem

https://github.com/conda/conda/issues/6064 https://github.com/ContinuumIO/anaconda-issues/issues/6424

The following two commands (as shown on the page above, helped me get through it):

sudo apt install execstack
execstack -c ~/miniconda3/lib/libcrypto.so.1.0.0

after which the conda create command worked:

conda create -y --name bioinfo python=3 

The book will be updated to represent this.

[ialbert]

I moved the SSL error into its own issue:

https://github.com/biostars/biostar-handbook-issues/issues/24

I will close this issue. Please open new and separate issues if these still persist. It is too complicated to answer multiple issues in one topic.

[ialbert]

I just got a note and confirmation that the latest conda does not work on windows.

See the conda installation on Windows (I have just updated it) make sure to reload your page.

https://www.biostarhandbook.com/setup/setup-conda.html

In a nutshell, you will need to remove your existing conda package with

rm -rf miniconda3

then install a previous version of it like so:

wget https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh
bash Miniconda3-4.3.21-Linux-x86_64.sh

Now if conda is not visible in your terminal it means you have not installed it properly.

[davichen1]

Has this issue been fixed in the updated version of the Handbook? I used the most updated version of Handbook and it doesn't seem resolved?

I followed all the instructions but get the same thing as LPain. I had 12 errors, identical to LPain, but when I went through the chronic entrez I got it reduced to 10. I tried to install fastqc manually using conda install -y fastqc but got error, CondaError: OSError(22, 'Invalid Argument') three times.

I can't seem to go further in the handbook without fixing this issue.

I've already installed and uninstalled the whole thing multiple times, following the instructions. I remember when installing, it seemed like there are 2 python versions installed? I have Python version 3.6.1 but I think when it tries to install, it tries to do python 2.7.14-0 when I do the python fix me | bash

[ialbert]

The problem itself has been fixed - as an entire class has been taught with it in classroom settings. Everything did work well as long as a few weeks ago.

On the other hand, it is always possible that the conda developers gave us a "Christmas Surprise" - I have read about of conda version 4.0 as being released at some point.

I am away on vacation for the next two weeks and would not be able to investigate until I am back.

[davichen1]

I don't know what I did differently but maybe I fixed it? I might have run

curl http://data.biostarhandbook.com/install/bash_profile.txt >> ~/.bash_profile curl http://data.biostarhandbook.com/install/bashrc.txt >> ~/.bashrc

this time, can't recall exactly what I did. I just kept trying over and over again. My system only shows 1 error now, not 12 errors. Says I'm missing R program, although I already have Rstudio and R installed from before.

So I think I'm okay to go?

[ialbert]

R is required only for RNA-Seq data analysis - so it is optional. See the chapter on R installation in the section titled:

RNA-Seq Statistical Analysis

Other than that it seems everything should be working. So I will close this issue.

Good job fixing problems - that is one of the first things we have to learn in this field - most of the time we can track down and fix installation problems by repeating the process.