search

faircloth-lab / phyluce

software for UCE (and general) phylogenomics
http://phyluce.readthedocs.org/
Other
80 stars 49 forks source link

Illumiprocessor wants to use Java 1.6 or 1.7 #100

Closed ericcrandall closed 5 years ago

ericcrandall commented 6 years ago

When I run illumiprocessor I get the following error/traceback:

Traceback (most recent call last):
  File "/anaconda2/envs/phyluce/bin/illumiprocessor", line 17, in <module>
    sys.exit(main())
  File "/anaconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/cli/main.py", line 131, in main
    main(args)
  File "/anaconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/main.py", line 22, in main
    core.check_dependencies(args)
  File "/anaconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/core.py", line 258, in check_dependencies
    java.test_version()
  File "/anaconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/core.py", line 238, in test_version
    "Java does not appear to be 1.6 or 1.7"
AssertionError: Java does not appear to be 1.6 or 1.7

However from reading other issues here, it seems like the program should run with any version of Java. When I run:

java -jar /anaconda2/envs/phyluce/jar/trimmomatic.jar 
Usage: 
       PE [-threads <threads>] [-phred33|-phred64] [-trimlog <trimLogFile>] [-basein <inputBase> | <inputFile1> <inputFile2>] [-baseout <outputBase> | <outputFile1P> <outputFile1U> <outputFile2P> <outputFile2U>] <trimmer1>...
   or: 
       SE [-threads <threads>] [-phred33|-phred64] [-trimlog <trimLogFile>] <inputFile> <outputFile> <trimmer1>...

It seems like trimmomatic is working. I traced the error messages back to illumiprocessor core.py and tried commenting out lines dealing with this error, but no luck.

Any ideas?

brantfaircloth commented 6 years ago

Can you let me know what version of illumiprocessor you have?

conda list illumiprocessor

ericcrandall commented 6 years ago 
# packages in environment at /anaconda2/envs/phyluce:
#
# Name                    Version                   Build  Channel
illumiprocessor           2.0.7                    py27_1    faircloth-lab
brantfaircloth commented 6 years ago

can you:

conda update illumiprocessor

ericcrandall commented 6 years ago

Thanks for your quick reply:

(phyluce) M053-303-26932:UCE_Fish_2 cran5048$ conda update illumiprocessor
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda

# All requested packages already installed.
brantfaircloth commented 6 years ago

There is a newer version of illlumiprocessor that should fix this issue. You should be able to install this using conda update illumiprocessor. Alternatively, you can just clone the repository from github and run python /path/to/illumiprocessor/bin/illumiprocessor.

ericcrandall commented 6 years ago

condo update illumiprocessor doesn't work, but condo remove illumiprocessor --force followed by conda install illumiprocessor upgrades it from 2.0.7 to 2.0.8.

I also had Java 10 installed and illumiprocessor wanted Java 8. So I figured out this set of commands for Mac with homebrew installed (just for anyone's future reference):

#  install Java 8 (I have 10 installed)
brew cask install caskroom/versions/java8
#now install jenv to manage java versions, add init line to .bash_profile as suggested here https://gist.github.com/tomysmile/a9a7aee85ff73454bd57e198ad90e614
brew install jenv
#find paths to java installations
/usr/libexec/java_home -V
# add java8 to jenv by using the paths I found in the last step
jenv add /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home
jenv versions
#set java version to 8
jenv local 1.8.0.162

However, now illumiprocessor can't find trimmomatic 

(phyluce) M053-303-26932:~ cran5048$ illumiprocessor --input demultiplexed --output uce-clean --config illumiprocessor_UCE2_config.txt --cores 4
Traceback (most recent call last):
  File "/Users/cran5048/anaconda2/envs/phyluce/bin/illumiprocessor", line 17, in <module>
    sys.exit(main())
  File "/Users/cran5048/anaconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/cli/main.py", line 131, in main
    main(args)
  File "/Users/cran5048/anaconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/main.py", line 22, in main
    core.check_dependencies(args)
  File "/Users/cran5048/anaconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/core.py", line 258, in check_dependencies
    java.test_trimmomatic(args.trimmomatic)
  File "/Users/cran5048/anaconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/core.py", line 251, in test_trimmomatic
    "Trimmomatic does not appear to be installed"
AssertionError: Trimmomatic does not appear to be installed

So I fixed the lines 28 and 33 in ~/anaconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/cli/main.py

to add "anaconda2" in the paths there (which originally just had "anaconda")

This seems to have fixed that issue.

brantfaircloth commented 6 years ago

You can also pass the path to trimmomatic to the script via the --trimmomatic option. The reason that the path is changing is because anaconda can be installed in a number of locations, and I need to make some changes to the code to deal with different conda environments, etc. (these paths also keep changing in conda, which makes things difficult).

drwhitehouse commented 6 years ago

I've tried to replicate ericrandall's fix above (although on CentOS). This doesn't appear to work with openjdk and I suspect the problem may be this test:

    def test_install(self):
        assert self.result_split[0].startswith("java version"), \
            "Java does not appear to be installed"

openjdk version prints:

Singularity phyluce.simg:/tmp/test_phyluce/uce-tutorial> /bin/java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)

I'm going to try Oracle Java and see what happens, but it might be worth looking at this.

brantfaircloth commented 6 years ago

This should be fixed in the current version of illumiprocessor on github. There is also a build of illumiprocessor with the fix available from bioconda. I have not edited the documentation to reflect this yet because of the dependencies between phyluce and illumiprocessor.

brantfaircloth commented 6 years ago

The illumiprocessor build is available from bioconda and the illumiprocessor documentation has been updated to reflect how to install this package (from bioconda).

ghost commented 6 years ago

The bioconda channel is missing 'noarch/repodata.json' and assocaited 'noarch/repodata.json.bz2' file. Can you put 2.09 onto Anaconda Cloud as well?

brantfaircloth commented 6 years ago

Not sure what the problem could be, conda create -n test_illumi python=2.7 illumiprocessor installs just fine on my end. To put into Anaconda Cloud, I have to rejigger part of the code to deal with java/trimmomatic issues outside of bioconda...

ghost commented 6 years ago

It turns out I had a typo in the channel which caused this problem, got it updated now, thank you!

brantfaircloth commented 6 years ago

Right on - glad it's working!

ghost commented 6 years ago

Hi Dr. Faircloth,

I've been following your phyluce tutorial, and I copied the illumiprocessor config file but somehow I am getting this error:

Administrators-MacBook-Pro:uce-tutorial admin$ illumiprocessor --input raw-fastq --output clean-fastq --config illumiprocessor.conf --cores 4 2018-08-17 12:37:26,286 - illumiprocessor - INFO - ==================== Starting illumiprocessor =================== 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Version: 2.0.9 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --config: illumiprocessor.conf 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --cores: 4 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --input: /Users/admin/Desktop/uce-tutorial/raw-fastq 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --log_path: None 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --min_len: 40 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --no_merge: False 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --output: /Users/admin/Desktop/uce-tutorial/clean-fastq 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --phred: phred33 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --r1_pattern: None 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --r2_pattern: None 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --se: False 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --trimmomatic: /Users/admin/miniconda2/bin/trimmomatic 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --verbosity: INFO Traceback (most recent call last): File "/Users/admin/miniconda2/bin/illumiprocessor", line 17, in sys.exit(main()) File "/Users/admin/miniconda2/lib/python2.7/site-packages/illumiprocessor-2.0.9-py2.7.egg/illumiprocessor/cli/main.py", line 121, in main main(args) File "/Users/admin/miniconda2/lib/python2.7/site-packages/illumiprocessor-2.0.9-py2.7.egg/illumiprocessor/main.py", line 33, in main for start_name, end_name in conf.items('names'): File "/Users/admin/miniconda2/lib/python2.7/ConfigParser.py", line 642, in items raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'names'

I have all the files named and arranged the same way as the tutorial.

Thanks!

Miles

liugang9988 commented 5 years ago

Hi Dr. Faircloth,

I've been following your phyluce tutorial, and I copied the illumiprocessor config file but somehow I am getting this error:

Administrators-MacBook-Pro:uce-tutorial admin$ illumiprocessor --input raw-fastq --output clean-fastq --config illumiprocessor.conf --cores 4 2018-08-17 12:37:26,286 - illumiprocessor - INFO - ==================== Starting illumiprocessor =================== 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Version: 2.0.9 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --config: illumiprocessor.conf 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --cores: 4 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --input: /Users/admin/Desktop/uce-tutorial/raw-fastq 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --log_path: None 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --min_len: 40 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --no_merge: False 2018-08-17 12:37:26,287 - illumiprocessor - INFO - Argument --output: /Users/admin/Desktop/uce-tutorial/clean-fastq 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --phred: phred33 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --r1_pattern: None 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --r2_pattern: None 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --se: False 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --trimmomatic: /Users/admin/miniconda2/bin/trimmomatic 2018-08-17 12:37:26,288 - illumiprocessor - INFO - Argument --verbosity: INFO Traceback (most recent call last): File "/Users/admin/miniconda2/bin/illumiprocessor", line 17, in sys.exit(main()) File "/Users/admin/miniconda2/lib/python2.7/site-packages/illumiprocessor-2.0.9-py2.7.egg/illumiprocessor/cli/main.py", line 121, in main main(args) File "/Users/admin/miniconda2/lib/python2.7/site-packages/illumiprocessor-2.0.9-py2.7.egg/illumiprocessor/main.py", line 33, in main for start_name, end_name in conf.items('names'): File "/Users/admin/miniconda2/lib/python2.7/ConfigParser.py", line 642, in items raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'names'

I have all the files named and arranged the same way as the tutorial.

Thanks!

Miles

Hello, Miles,

I am now facing the same problems like you. Did you solve this problem and how? Thank you. Best, Gang

kkmills commented 5 years ago

Hi Brant,

I am having the same issue as @ericcrandall, and updating illumiprocessor did not change anything.

(phyluce) kendall@DESKTOP-NOSAIQH:~/marmot-uces$ illumiprocessor --input raw-gz/ --output clean-uces/ --config illumiprocessor-config --cores 2
Traceback (most recent call last):
  File "/home/kendall/miniconda2/envs/phyluce/bin/illumiprocessor", line 17, in <module>
    sys.exit(main())
  File "/home/kendall/miniconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/cli/main.py", line 131, in main
    main(args)
  File "/home/kendall/miniconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/main.py", line 22, in main
    core.check_dependencies(args)
  File "/home/kendall/miniconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/core.py", line 256, in check_dependencies
    java.test_install()
  File "/home/kendall/miniconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/core.py", line 232, in test_install
    "Java does not appear to be installed"
AssertionError: Java does not appear to be installed

The version is

(phyluce) kendall@DESKTOP-NOSAIQH:~/marmot-uces$ java -version
openjdk version "1.8.0_152-release"
OpenJDK Runtime Environment (build 1.8.0_152-release-1056-b12)
OpenJDK 64-Bit Server VM (build 25.152-b12, mixed mode)

and

(phyluce) kendall@DESKTOP-NOSAIQH:~/marmot-uces$ conda list illumiprocessor
# packages in environment at /home/kendall/miniconda2/envs/phyluce:
# 
# Name                    Version                   Build  Channel
illumiprocessor           2.0.8            py27h448edb2_1    http://conda.binstar.org/faircloth-lab

I can see that java is sitting inside ~/miniconda2/envs/phyluce/bin, but it doesn't look like there is a flag available to point illumiprocessor to it.

Edit: I tried installing java myself and added ~/jre1.8.0_231/bin to the beginning of my PATH. When I tried to run illumiprocessor, I got the same error as above that trimmomatic was not installed. I added the flag --trimmomatic ~/miniconda2/envs/phyluce/jar/trimmomatic.jar, and now I have the same error message as the last two commenters about there being no [names] section in the config file:

(phyluce) kendall@DESKTOP-NOSAIQH:~/miniconda2/envs/phyluce$ illumiprocessor --input raw-gz/ --output clean-uces/ --config illumiprocessor-config --cores 2 --trimmomatic ~/miniconda2/envs/phyluce/jar/trimmomatic.jar
2019-11-19 23:36:53,058 - illumiprocessor - INFO - ==================== Starting illumiprocessor ===================
2019-11-19 23:36:53,082 - illumiprocessor - INFO - Version: 2.0.6
2019-11-19 23:36:53,083 - illumiprocessor - INFO - Argument --config: illumiprocessor-config
2019-11-19 23:36:53,084 - illumiprocessor - INFO - Argument --cores: 2
2019-11-19 23:36:53,085 - illumiprocessor - INFO - Argument --input: /home/kendall/miniconda2/envs/phyluce/raw-gz
2019-11-19 23:36:53,086 - illumiprocessor - INFO - Argument --log_path: None
2019-11-19 23:36:53,087 - illumiprocessor - INFO - Argument --min_len: 40
2019-11-19 23:36:53,088 - illumiprocessor - INFO - Argument --no_merge: False
2019-11-19 23:36:53,088 - illumiprocessor - INFO - Argument --output: /home/kendall/miniconda2/envs/phyluce/clean-uces
2019-11-19 23:36:53,089 - illumiprocessor - INFO - Argument --phred: phred33
2019-11-19 23:36:53,089 - illumiprocessor - INFO - Argument --r1_pattern: None
2019-11-19 23:36:53,090 - illumiprocessor - INFO - Argument --r2_pattern: None
2019-11-19 23:36:53,091 - illumiprocessor - INFO - Argument --se: False
2019-11-19 23:36:53,091 - illumiprocessor - INFO - Argument --trimmomatic: /home/kendall/miniconda2/envs/phyluce/jar/trimmomatic.jar
2019-11-19 23:36:53,092 - illumiprocessor - INFO - Argument --verbosity: INFO
Traceback (most recent call last):
  File "/home/kendall/miniconda2/envs/phyluce/bin/illumiprocessor", line 17, in <module>
    sys.exit(main())
  File "/home/kendall/miniconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/cli/main.py", line 131, in main
    main(args)
  File "/home/kendall/miniconda2/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/main.py", line 33, in main
    for start_name, end_name in conf.items('names'):
  File "/home/kendall/miniconda2/envs/phyluce/lib/python2.7/ConfigParser.py", line 642, in items
    raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'names'

But my config file has a [names] section at the end.

Edit: OK, turns out I just had a rogue space in one of my names. So the [names] error was not related to the java and trimmomatic errors I was getting before. I'm still not sure why java was not working originally, but I've at least got illumiprocessor to run now.

Danisystematic commented 4 years ago

Hello Dr. Faircloth and kkmills, After installing miniconda and using Trinity (all works perfectly fine), we have to use illumiprocessor on the same environment but we get the same message error as previously shown:

"Java does not appear to be installed" AssertionError: Java does not appear to be installed

We began by the tutorial test so the data and names are correct.

So we tried to update and check the version that seems to be fine:

version of illumiprocessor: 2.0.8

and java is installed in the repertory of phyluce is the same version as you kkmills.

openjdk version "1.8.0_152-release" OpenJDK Runtime Environment (build 1.8.0_152-release-1056-b12) OpenJDK 64-Bit Server VM (build 25.152-b12, mixed mode)

So the problem persists and seems to come with the path, the programm illumiprocessor cant find it. I have no idea how to precise this path on miniconda, is there a way? We have tried the solution proposed by kkmils in this post, but we are realy beginners in linux system and dont understand how he did it.

Thank you in advance for your answer and for all the super tools.

brantfaircloth commented 4 years ago

This sounds like an error that used to pop up in older versions of illumiprocessor. Can you try to install only illumiprocessor, then use that installation?

conda create -n test_illumiprocessor illumiprocessor=2.0.9
conda activate test_illumiprocessor
illumiprocessor <parameters...>
Danisystematic commented 4 years ago

Thanks for your fast reply. We tried what you have suggested and got the following error message:

(test_illumiprocessor) liocheles@liocheles-MacPro ~/Documents/UCE-test-daniel $ illumiprocessor --input raw-fastq/ --output clean-fastqc --config illumiprocessor.conf --cores 4 [WARNING] Output directory exists, REMOVE [Y/n]? Y Traceback (most recent call last): File "/home/liocheles/miniconda3/envs/test_illumiprocessor/bin/illumiprocessor", line 17, in sys.exit(main()) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/cli/main.py", line 121, in main main(args) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/main.py", line 22, in main core.check_dependencies(args) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/core.py", line 241, in check_dependencies java.test_trimmomatic(args.trimmomatic) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/core.py", line 229, in test_trimmomatic stderr=subprocess.PIPE File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/subprocess.py", line 394, in init errread, errwrite) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/subprocess.py", line 1047, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory

Different this time however, dont seems to come with java.

brantfaircloth commented 4 years ago

ok, try:

conda activate test_illumiprocessor
conda install openjdk=8.0.265
Danisystematic commented 4 years ago

Sorry to be an inconvenience, but doesnt work again. The environnement seems to work fine, we can open it and we have also reinstalled openjdk. After the command: conda install openjdk=8.0.265 we got the message that it was already installed and the problem persists with the same error message.

Traceback (most recent call last): File "/home/liocheles/miniconda3/envs/test_illumiprocessor/bin/illumiprocessor", line 17, in sys.exit(main()) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/cli/main.py", line 121, in main main(args) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/main.py", line 22, in main core.check_dependencies(args) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/core.py", line 241, in check_dependencies java.test_trimmomatic(args.trimmomatic) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/core.py", line 229, in test_trimmomatic stderr=subprocess.PIPE File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/subprocess.py", line 394, in init errread, errwrite) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/subprocess.py", line 1047, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory

Thank you again for your help.

brantfaircloth commented 4 years ago

I'm really not sure what the problem is (or could be - all tests on the software in Mac OS and Linux run just fine).

You might try to pass the path to trimmomatic to the software by adding the flag (also check to be sure trimmomatic exists at this location):

--trimmomatic /home/liocheles/miniconda3/envs/test_illumiprocessor/bin/trimmomatic
Danisystematic commented 4 years ago

The problem seems to be there, trimmomatic is not installed in the good repertory, but it exists in the folder pkgs of miniconda3 so when we ask to conda to install it or to check for it, it's recognized as already installed.

The command: conda install --name test_illumiprocessor trimmomatic return the message that All the requested packages are already installed.

We will work on this base more longer and try to find a way to precise the path for the installation.

Thank you again and good week.

Danisystematic commented 4 years ago

We tried a lot of things but the problem persists.

It seems impossible to install trimmomatic inside the good environnement (test-illumiprocessor) even with using the --force installing command--> #All packages are already installed.

So we tried to use a path from where it exists in miniconda pkgs for exemple or in the phyluce environment(potentialy in the wrong place because we cannot run illumiprocessor either in the phyluce environment). but now with specifying the path in both cases the error is that we don't have permission to do this:

(test_illumiprocessor) liocheles@liocheles-MacPro ~/Documents/UCE-test-daniel $ illumiprocessor --input raw-fastq/ --output clean-fastqc --config illumiprocessor.conf --cores 4 --trimmomatic /home/liocheles/miniconda3/envs/phyluce/jar/trimmomatic.jar [WARNING] Output directory exists, REMOVE [Y/n]? Y Traceback (most recent call last): File "/home/liocheles/miniconda3/envs/test_illumiprocessor/bin/illumiprocessor", line 17, in sys.exit(main()) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/cli/main.py", line 121, in main main(args) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/main.py", line 22, in main core.check_dependencies(args) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/core.py", line 241, in check_dependencies java.test_trimmomatic(args.trimmomatic) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/site-packages/illumiprocessor/core.py", line 229, in test_trimmomatic stderr=subprocess.PIPE File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/subprocess.py", line 394, in init errread, errwrite) File "/home/liocheles/miniconda3/envs/test_illumiprocessor/lib/python2.7/subprocess.py", line 1047, in _execute_child raise child_exception OSError: [Errno 13] Permission denied

We also tried to install trimmomatic2.0.9 with a zipfile provided on ghitub but we don't know exactly where to send and extract it in the environment and so far we haven't been successful.

Is there a way to precise the path for the instalation inside the environment(despite tha fact that it seems to be installed elsewhere)? And if so, where is the good place for it. Is it indeed in the environment/bin/trimmomatic folder?

brantfaircloth commented 4 years ago

If conda says that the package is installed in the test_illumiprocessor environment, then it should be installed where you need for it to be. can you:

conda activate test_illumiprocessor
# then run
trimmomatic

you should see output like:

> trimmomatic
Usage:
       PE [-version] [-threads <threads>] [-phred33|-phred64] [-trimlog <trimLogFile>] [-summary <statsSummaryFile>] [-quiet] [-validatePairs] [-basein <inputBase> | <inputFile1> <inputFile2>] [-baseout <outputBase> | <outputFile1P> <outputFile1U> <outputFile2P> <outputFile2U>] <trimmer1>...
   or:
       SE [-version] [-threads <threads>] [-phred33|-phred64] [-trimlog <trimLogFile>] [-summary <statsSummaryFile>] [-quiet] <inputFile> <outputFile> <trimmer1>...
   or:
       -version

also, when in that environment, what happens when you run:

java -version
Danisystematic commented 4 years ago

Sorry for my late answer, we did what you asked and it did not work for trimmomatic->command not found so we realized that all was goodly updated, except the version for trimmommatic (we got 0.35 and the last one is 0.39) after the command: conda install -c bioconda trimmomatic=0.39 The programm work finaly fine now.

So the error: "OSError: [Errno 2] No such file or directory",as you mentionned, was not because no trimmomatic was in good folder , but because it was not updated and did not be recognized.

We wanted to thank you very much, also wishing you a good continuation.

NataliaCD commented 4 years ago

Hello Dr. Faircloth, I am having a similar issue to the one exposed by Danisystematic, however none of the suggested solutions have worked for me. This is the error:

(phyluce) ncortes@vortex:~$ illumiprocessor --input /home/ncortes/Novogene_Data/raw_data_Artibeus_fraterculus --output /home/ncortes/Novogene_Data/clean_fastq/ --config /home/ncortes/Novogene_Data/Illumiprocessor_AF.conf --r1-pattern _1.fq.gz --r2-pattern _2.fq.gz --trimmomatic /home/ncortes/anaconda/envs/phyluce/share/trimmomatic-0.39-1/trimmomatic [WARNING] Output directory exists, REMOVE [Y/n]? Y 2020-12-02 13:14:30,431 - illumiprocessor - INFO - ==================== Starting illumiprocessor =================== 2020-12-02 13:14:30,431 - illumiprocessor - INFO - Version: 2.0.9 2020-12-02 13:14:30,431 - illumiprocessor - INFO - Argument --config: /home/ncortes/Novogene_Data/Illumiprocessor_AF.conf 2020-12-02 13:14:30,431 - illumiprocessor - INFO - Argument --cores: 1 2020-12-02 13:14:30,431 - illumiprocessor - INFO - Argument --input: /home/ncortes/Novogene_Data/raw_data_Artibeus_fraterculus 2020-12-02 13:14:30,431 - illumiprocessor - INFO - Argument --log_path: None 2020-12-02 13:14:30,431 - illumiprocessor - INFO - Argument --min_len: 40 2020-12-02 13:14:30,431 - illumiprocessor - INFO - Argument --no_merge: False 2020-12-02 13:14:30,432 - illumiprocessor - INFO - Argument --output: /home/ncortes/Novogene_Data/clean_fastq 2020-12-02 13:14:30,432 - illumiprocessor - INFO - Argument --phred: phred33 2020-12-02 13:14:30,432 - illumiprocessor - INFO - Argument --r1_pattern: _1.fq.gz 2020-12-02 13:14:30,432 - illumiprocessor - INFO - Argument --r2_pattern: _2.fq.gz 2020-12-02 13:14:30,432 - illumiprocessor - INFO - Argument --se: False 2020-12-02 13:14:30,432 - illumiprocessor - INFO - Argument --trimmomatic: /home/ncortes/anaconda/envs/phyluce/share/trimmomatic-0.39-1/trimmomatic 2020-12-02 13:14:30,432 - illumiprocessor - INFO - Argument --verbosity: INFO 2020-12-02 13:14:30,464 - illumiprocessor - INFO - Trimming samples with Trimmomatic Running...........................Traceback (most recent call last): File "/home/ncortes/anaconda/envs/phyluce/bin/illumiprocessor", line 17, in sys.exit(main()) File "/home/ncortes/anaconda/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/cli/main.py", line 121, in main main(args) File "/home/ncortes/anaconda/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/main.py", line 47, in main map(core.runner, work) File "/home/ncortes/anaconda/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/core.py", line 317, in runner trimmomatic_merger(sample) File "/home/ncortes/anaconda/envs/phyluce/lib/python2.7/site-packages/illumiprocessor/core.py", line 304, in trimmomatic_merger os.rename(singles[0], new_pth) OSError: [Errno 2] No such file or directory

As additional info, here are the versions of illimiprocessor, java and trimmomatic: illumiprocessor 2.0.9 py_2 bioconda openjdk version "1.8.0_265" OpenJDK Runtime Environment (Zulu 8.48.0.53-CA-linux64) (build 1.8.0_265-b11) OpenJDK 64-Bit Server VM (Zulu 8.48.0.53-CA-linux64) (build 25.265-b11, mixed mode) trimmomatic 0.39 1 bioconda

I'd really appreciate any suggestions. Thank you.

brantfaircloth commented 4 years ago

It looks like the specification of your --r1 and --r2 patterns is not correct, but I cannot be sure. What are some example names of files you are trying to process?

NataliaCD commented 4 years ago

Here are some examples: MZ168894_CKDL200165158-1B-AK7078-AK8586_HCJTTCCX2_L2_1.fq.gz
MZ168894_CKDL200165158-1B-AK7078-AK8586_HCJTTCCX2_L2_2.fq.gz
MZ168895_CKDL200165158-1B-AK7078-AK10616_HCJTTCCX2_L2_1.fq.gz
MZ168895_CKDL200165158-1B-AK7078-AK10616_HCJTTCCX2_L2_2.fq.gz

brantfaircloth commented 4 years ago

At the very least, you will need to use `--r1-pattern {}_1.fq.gz --r2-pattern {}_2.fq.gz``, but this also depends on how you have the config file setup.

NataliaCD commented 4 years ago

This is how my config file looks like:

[adapters] i7:GATCGGAAGAGCACACGTCTGAACTCCAGTCACATCTCGTATGCCGTCTTCTGCTTG i5:AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTGTGTAGATCTCGGTGGTCGCCGTATCATT

[tag sequences] i7_102_04:GATTCAGC i7_102_03:CGTTGCAA i7_102_03:CGTTGCAA i7_102_01:ATAAGGCG i7_102_02:CTTACCTG i7_102_02:CTTACCTG i7_102_04:GATTCAGC i7_102_02:CTTACCTG i7_102_03:CGTTGCAA i7_102_01:ATAAGGCG i7_102_02:CTTACCTG i7_102_02:CTTACCTG i7_102_04:GATTCAGC i7_102_01:ATAAGGCG i7_102_04:GATTCAGC i7_102_01:ATAAGGCG i7_102_04:GATTCAGC i7_102_01:ATAAGGCG i7_102_04:GATTCAGC i7_102_03:CGTTGCAA i7_102_02:CTTACCTG i7_102_04:GATTCAGC i7_102_03:CGTTGCAA i7_102_03:CGTTGCAA i7_102_01:ATAAGGCG i7_102_04:GATTCAGC i7_102_01:ATAAGGCG i7_102_02:CTTACCTG i7_102_02:CTTACCTG i7_102_01:ATAAGGCG i7_102_03:CGTTGCAA i7_102_03:CGTTGCAA i7_101_02:CTGTGTTG i7_101_02:CTGTGTTG i7_101_02:CTGTGTTG i7_101_02:CTGTGTTG i7_101_01:ACGTTACC i7_101_02:CTGTGTTG i7_101_01:ACGTTACC i7_101_01:ACGTTACC i5_01_C:CACAGACT i5_01_C:CACAGACT i5_01_B:AGTGGCAA i5_01_D:CGACACTT i5_01_H:TAGCTGAG i5_01_F:GTGAGACT i5_01_E:GACTTGTG i5_01_A:ACCGACAA i5_01_E:GACTTGTG i5_01_A:ACCGACAA i5_01_E:GACTTGTG i5_01_G:GTTCCATG i5_01_H:TAGCTGAG i5_01_E:GACTTGTG i5_01_A:ACCGACAA i5_01_G:GTTCCATG i5_01_B:AGTGGCAA i5_01_C:CACAGACT i5_01_D:CGACACTT i5_01_F:GTGAGACT i5_01_B:AGTGGCAA i5_01_G:GTTCCATG i5_01_A:ACCGACAA i5_01_D:CGACACTT i5_01_B:AGTGGCAA i5_01_F:GTGAGACT i5_01_H:TAGCTGAG i5_01_C:CACAGACT i5_01_D:CGACACTT i5_01_F:GTGAGACT i5_01_H:TAGCTGAG i5_01_G:GTTCCATG i5_02_G:TGGATGGT i5_02_E:CAACTCCA i5_02_F:AACACGCT i5_02_D:AGACGCTA i5_01_C:CACAGACT i5_02_H:TTCGAAGC i5_01_B:AGTGGCAA i5_01_D:CGACACTT

[tag map] VPT634_CKDL200165158-1B-AK9143-AK2867_HCJTTCCX2_L2:i7_102_04,i5_01_C MZ168902_CKDL200165158-1B-AK7078-AK2867_HCJTTCCX2_L2:i7_102_03,i5_01_C TK134575_CKDL200165158-1B-AK7078-AK7506_HCJTTCCX2_L2:i7_102_03,i5_01_B RM125155_CKDL200165158-1B-5UDI301-AK13069_HCJTTCCX2_L2:i7_102_01,i5_01_D TK134582_CKDL200165158-1B-AK7758-AK10616_HCJTTCCX2_L2:i7_102_02,i5_01_H TK134578_CKDL200165158-1B-AK7758-AK9172_HCJTTCCX2_L2:i7_102_02,i5_01_F VPT632_CKDL200165158-1B-AK9143-AK8586_HCJTTCCX2_L2:i7_102_04,i5_01_E TK135323_CKDL200165158-1B-AK7758-5UDI267_HCJTTCCX2_L2:i7_102_02,i5_01_A MZ168894_CKDL200165158-1B-AK7078-AK8586_HCJTTCCX2_L2:i7_102_03,i5_01_E RM125144_CKDL200165158-1B-5UDI301-5UDI267_HCJTTCCX2_L2:i7_102_01,i5_01_A TK134577_CKDL200165158-1B-AK7758-AK8586_HCJTTCCX2_L2:i7_102_02,i5_01_E TK134589_CKDL200165158-1B-AK7758-5UDI197_HCJTTCCX2_L2:i7_102_02,i5_01_G VPT629_CKDL200165158-1B-AK9143-AK10616_HCJTTCCX2_L2:i7_102_04,i5_01_H RM125156_CKDL200165158-1B-5UDI301-AK8586_HCJTTCCX2_L2:i7_102_01,i5_01_E VPT637_CKDL200165158-1B-AK9143-5UDI267_HCJTTCCX2_L2:i7_102_04,i5_01_A TK134576_CKDL200165158-1B-5UDI301-5UDI197_HCJTTCCX2_L2:i7_102_01,i5_01_G TK135333_CKDL200165158-1B-AK9143-AK7506_HCJTTCCX2_L2:i7_102_04,i5_01_B RM125154_CKDL200165158-1B-5UDI301-AK2867_HCJTTCCX2_L2:i7_102_01,i5_01_C VPT633_CKDL200165158-1B-AK9143-AK13069_HCJTTCCX2_L2:i7_102_04,i5_01_D MZ168896_CKDL200165158-1B-AK7078-AK9172_HCJTTCCX2_L2:i7_102_03,i5_01_F TK135322_CKDL200165158-1B-AK7758-AK7506_HCJTTCCX2_L2:i7_102_02,i5_01_B VPT630_CKDL200165158-1B-AK9143-5UDI197_HCJTTCCX2_L2:i7_102_04,i5_01_G TK135325_CKDL200165158-1B-AK7078-5UDI267_HCJTTCCX2_L2:i7_102_03,i5_01_A MZ168897_CKDL200165158-1B-AK7078-AK13069_HCJTTCCX2_L2:i7_102_03,i5_01_D RM125145_CKDL200165158-1B-5UDI301-AK7506_HCJTTCCX2_L2:i7_102_01,i5_01_B VPT631_CKDL200165158-1B-AK9143-AK9172_HCJTTCCX2_L2:i7_102_04,i5_01_F TK135326_CKDL200165158-1B-5UDI301-AK10616_HCJTTCCX2_L2:i7_102_01,i5_01_H TK135327_CKDL200165158-1B-AK7758-AK2867_HCJTTCCX2_L2:i7_102_02,i5_01_C TK135330_CKDL200165158-1B-AK7758-AK13069_HCJTTCCX2_L2:i7_102_02,i5_01_D RM125159_CKDL200165158-1B-5UDI301-AK9172_HCJTTCCX2_L2:i7_102_01,i5_01_F MZ168895_CKDL200165158-1B-AK7078-AK10616_HCJTTCCX2_L2:i7_102_03,i5_01_H MZ168898_CKDL200165158-1B-AK7078-5UDI197_HCJTTCCX2_L2:i7_102_03,i5_01_G QCAZ14430_CKDL200165158-1B-AK3905-AK8571_HCJTTCCX2_L2:i7_101_02,i5_02_G QCAZ14427_CKDL200165158-1B-AK3905-AK6249_HCJTTCCX2_L2:i7_101_02,i5_02_E QCAZ14428_CKDL200165158-1B-AK3905-AK8591_HCJTTCCX2_L2:i7_101_02,i5_02_F QCAZ14429_CKDL200165158-1B-AK3905-AK8184_HCJTTCCX2_L2:i7_101_02,i5_02_D QCAZ15861_CKDL200165158-1B-AK9141-AK2867_HCJTTCCX2_L2:i7_101_01,i5_01_C QCAZ9324_CKDL200165158-1B-AK3905-5UDI254_HCJTTCCX2_L2:i7_101_02,i5_02_H QCAZ9331_CKDL200165158-1B-AK9141-AK7506_HCJTTCCX2_L2:i7_101_01,i5_01_B QCAZ5946_CKDL200165158-1B-AK9141-AK13069_HCJTTCCX2_L2:i7_101_01,i5_01_D

[names] VPT634_CKDL200165158-1B-AK9143-AK2867_HCJTTCCX2_L2:VPT634_Afrater MZ168902_CKDL200165158-1B-AK7078-AK2867_HCJTTCCX2_L2:MZ168902_Afrater TK134575_CKDL200165158-1B-AK7078-AK7506_HCJTTCCX2_L2:TK134575_Afrater RM125155_CKDL200165158-1B-5UDI301-AK13069_HCJTTCCX2_L2:RM125155_Afrater TK134582_CKDL200165158-1B-AK7758-AK10616_HCJTTCCX2_L2:TK134582_Afrater TK134578_CKDL200165158-1B-AK7758-AK9172_HCJTTCCX2_L2:TK134578_Afrater VPT632_CKDL200165158-1B-AK9143-AK8586_HCJTTCCX2_L2:VPT632_Afrater TK135323_CKDL200165158-1B-AK7758-5UDI267_HCJTTCCX2_L2:TK135323_Afrater MZ168894_CKDL200165158-1B-AK7078-AK8586_HCJTTCCX2_L2:MZ168894_Afrater RM125144_CKDL200165158-1B-5UDI301-5UDI267_HCJTTCCX2_L2:RM125144_Afrater TK134577_CKDL200165158-1B-AK7758-AK8586_HCJTTCCX2_L2:TK134577_Afrater TK134589_CKDL200165158-1B-AK7758-5UDI197_HCJTTCCX2_L2:TK134589_Afrater VPT629_CKDL200165158-1B-AK9143-AK10616_HCJTTCCX2_L2:VPT629_Afrater RM125156_CKDL200165158-1B-5UDI301-AK8586_HCJTTCCX2_L2:RM125156_Afrater VPT637_CKDL200165158-1B-AK9143-5UDI267_HCJTTCCX2_L2:VPT637_Afrater TK134576_CKDL200165158-1B-5UDI301-5UDI197_HCJTTCCX2_L2:TK134576_Afrater TK135333_CKDL200165158-1B-AK9143-AK7506_HCJTTCCX2_L2:TK135333_Afrater RM125154_CKDL200165158-1B-5UDI301-AK2867_HCJTTCCX2_L2:RM125154_Afrater VPT633_CKDL200165158-1B-AK9143-AK13069_HCJTTCCX2_L2:VPT633_Afrater MZ168896_CKDL200165158-1B-AK7078-AK9172_HCJTTCCX2_L2:MZ168896_Afrater TK135322_CKDL200165158-1B-AK7758-AK7506_HCJTTCCX2_L2:TK135322_Afrater VPT630_CKDL200165158-1B-AK9143-5UDI197_HCJTTCCX2_L2:VPT630_Afrater TK135325_CKDL200165158-1B-AK7078-5UDI267_HCJTTCCX2_L2:TK135325_Afrater MZ168897_CKDL200165158-1B-AK7078-AK13069_HCJTTCCX2_L2:MZ168897_Afrater RM125145_CKDL200165158-1B-5UDI301-AK7506_HCJTTCCX2_L2:RM125145_Afrater VPT631_CKDL200165158-1B-AK9143-AK9172_HCJTTCCX2_L2:VPT631_Afrater TK135326_CKDL200165158-1B-5UDI301-AK10616_HCJTTCCX2_L2:TK135326_Afrater TK135327_CKDL200165158-1B-AK7758-AK2867_HCJTTCCX2_L2::TK135327_Afrater TK135330_CKDL200165158-1B-AK7758-AK13069_HCJTTCCX2_L2:TK135330_Afrater RM125159_CKDL200165158-1B-5UDI301-AK9172_HCJTTCCX2_L2:RM125159_Afrater MZ168895_CKDL200165158-1B-AK7078-AK10616_HCJTTCCX2_L2:MZ168895_Afrater MZ168898_CKDL200165158-1B-AK7078-5UDI197_HCJTTCCX2_L2:MZ168898_Afrater QCAZ14430_CKDL200165158-1B-AK3905-AK8571_HCJTTCCX2_L2:QCAZ14430_Afrater QCAZ14427_CKDL200165158-1B-AK3905-AK6249_HCJTTCCX2_L2:QCAZ14427_Afrater QCAZ14428_CKDL200165158-1B-AK3905-AK8591_HCJTTCCX2_L2:QCAZ14428_Afrater QCAZ14429_CKDL200165158-1B-AK3905-AK8184_HCJTTCCX2_L2:QCAZ14429_Afrater QCAZ15861_CKDL200165158-1B-AK9141-AK2867_HCJTTCCX2_L2:QCAZ15861_Afrater QCAZ9324_CKDL200165158-1B-AK3905-5UDI254_HCJTTCCX2_L2:QCAZ9324_Afrater QCAZ9331_CKDL200165158-1B-AK9141-AK7506_HCJTTCCX2_L2:QCAZ9331_Afrater QCAZ5946_CKDL200165158-1B-AK9141-AK13069_HCJTTCCX2_L2:QCAZ5946_Afrater

brantfaircloth commented 4 years ago

ok - I would start by reducing your config file to a single set of reads, get that to work, and go from there.

NataliaCD commented 4 years ago

Ok, I will try that. Thank you very much.

NataliaCD commented 4 years ago

I tried to run it only for a set of reads and it worked. I realized that something should be wrong with the config file, so I reviewed it very carefully and I found a typo. After correcting it, the program ran smoothly. Thanks for your advice Dr. Faircloth.

brantfaircloth commented 4 years ago

Excellent 🎉