hisat2_read_statistics.py syntax error

tony-travis commented 4 years ago

Version info

bcbio version (bcbio_nextgen.py --version): 1.2.2
OS name and version: Ubuntu 18.04.4 LTS

To Reproduce

Exact bcbio command you have used /usr/local/bcbio/anaconda/bin/python3.6 /usr/local/bcbio/bin/bcbio_nextgen.py -t ipython -s slurm -q main -n 11 /work/EOC/paganetti/papin/Tau_KO_Papin-64260198/papin/config/papin.yaml -r timelimit=+30-00:00:00

Your sample configuration file

details:
- algorithm:
aligner: hisat2
bcbiornaseq:
  genomeBuild: GRCh38
  interesting_groups:
  - batch
  - group
  - timepoint
  organism: homo sapiens
kraken: minikraken
strandedness: firststrand
tools_on: bcbiornaseq
analysis: RNA-seq
description: _WT1_6hr
files:
- /work/EOC/paganetti/papin/Tau_KO_Papin-64260198/FASTQ_merge/_WT1_6hr_R1.fastq.gz
- /work/EOC/paganetti/papin/Tau_KO_Papin-64260198/FASTQ_merge/_WT1_6hr_R2.fastq.gz
genome_build: hg38
metadata:
batch: WT1
group: TAU
timepoint: 6hr

[...]

- algorithm:
aligner: hisat2
bcbiornaseq:
  genomeBuild: GRCh38
  interesting_groups:
  - batch
  - group
  - timepoint
  organism: homo sapiens
kraken: minikraken
strandedness: firststrand
tools_on: bcbiornaseq
analysis: RNA-seq
description: _231K_3hr
files:
- /work/EOC/paganetti/papin/Tau_KO_Papin-64260198/FASTQ_merge/_231K_3hr_R1.fastq.gz
- /work/EOC/paganetti/papin/Tau_KO_Papin-64260198/FASTQ_merge/_231K_3hr_R2.fastq.gz
genome_build: hg38
metadata:
batch: 231K
group: TAU-KO
timepoint: 3hr
fc_name: papin
upload:
dir: ../final

bcbio-logs.zip

Observed behavior

[2020-04-09T15:46Z] serenity: Aligning lane _WT1_6hr with hisat2 aligner
[2020-04-09T15:46Z] serenity: Aligning /work/EOC/paganetti/papin/Tau_KO_Papin-64260198/FASTQ_merge/_WT1_6hr_R1.fastq.gz and /work/EOC/paganetti/papin/Tau_KO_Papin-64260198/FASTQ_merge/_WT1_6hr_R2.fastq.gz with hisat2.
[2020-04-09T15:46Z] serenity:   File "/usr/local/bcbio/bin/../anaconda/envs/python2/bin/hisat2_read_statistics.py", line 182
[2020-04-09T15:46Z] serenity:     length_map = sorted(length_map.iteritems(), key=lambda (k,v):(v,k), reverse=True)
[2020-04-09T15:46Z] serenity:                                                            ^
[2020-04-09T15:46Z] serenity: SyntaxError: invalid syntax

Expected behavior No syntax error

Log files Please attach (10MB max): bcbio-nextgen.log, bcbio-nextgen-commands.log, and bcbio-nextgen-debug.log.

Additional context This issue has been raised as a Python 3 incompatibility in hisat2 but the "bcbio_nextgen" env should be running Python 2.

roryk commented 4 years ago

Hi Tony,

What happens if you don't preface the command with /usr/local/bcbio/anaconda/bin/python3.6? You should just be able to run bcbio_nextgen.py by itself.

tony-travis commented 4 years ago

On 11/04/2020 19:55, Rory Kirchner wrote:

Hi Tony,

What happens if you don't preface the command with |/usr/local/bcbio/anaconda/bin/python3.6|? You should just be able to run |bcbio_nextgen.py| by itself.

Hi, it was submitted to "slurm" as a batch file containing:

!/bin/bash

SBATCH -N 1

SBATCH --output=/work/EOC/log/paganetti_analysis.log

set -o nounset

PROJECTDIR="/work/EOC/paganetti/papin/Tau_KO_Papin-64260198/papin/" CONFIG="/work/EOC/paganetti/papin/Tau_KO_Papin-64260198/papin/config/papin.yaml" CORES=36

echo "Running on $SLURMD_NODENAME" export PATH=/usr/local/bcbio/bin:$PATH

cd "$PROJECTDIR/work"

bcbio_nextgen.py -t ipython -s slurm -q main -n $CORES $CONFIG -r timelimit=+30-00:00:00

I used "pgrep" to see how the job was being executed, and that is where the /usr/local/bcbio/anaconda/bin/python3.6 comes from. All the machines in the cluster default to Python 2:

manager@brigante:~$ dsh -Ma -- python -V brigante.local: Python 2.7.17 firefly.local: Python 2.7.17 serenity.local: Python 2.7.17 destiny.local: Python 2.7.17

That's why I thought it was "bcbio_nextgen.py" running "python3.6"...

We'll try it again, but not running the script as a "slurm" job.

Thanks,

Tony.

-- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548 http://minke-informatics.co.uk mob. +44(0)7985 078324 mailto:tony.travis@minke-informatics.co.uk

tony-travis commented 4 years ago

Hi, Rory. This might be relevant - There are several 'py' commands, all of which use python3.6:

bcbio@brigante:~$ ls -l /usr/local/bcbio/bin/py*
lrwxrwxrwx 1 bcbio bcbio 18 Mar 31 04:28 /usr/local/bcbio/bin/py -> ../anaconda/bin/py
lrwxrwxrwx 1 bcbio bcbio 19 Mar 31 04:28 /usr/local/bcbio/bin/py2 -> ../anaconda/bin/py2
lrwxrwxrwx 1 bcbio bcbio 21 Mar 31 04:28 /usr/local/bcbio/bin/py2.7 -> ../anaconda/bin/py2.7
lrwxrwxrwx 1 bcbio bcbio 27 Mar 31 04:28 /usr/local/bcbio/bin/pycompleter -> ../anaconda/bin/pycompleter
lrwxrwxrwx 1 bcbio bcbio 28 Mar 31 04:28 /usr/local/bcbio/bin/pycompleter2 -> ../anaconda/bin/pycompleter2
lrwxrwxrwx 1 bcbio bcbio 30 Mar 31 04:28 /usr/local/bcbio/bin/pycompleter2.7 -> ../anaconda/bin/pycompleter2.7
bcbio@brigante:~$ /usr/local/bcbio/bin/py -V
Pythonpy 0.4.11
Python 3.6.10
bcbio@brigante:~$ /usr/local/bcbio/bin/py2 -V
Pythonpy 0.4.11
Python 3.6.10
bcbio@brigante:~$ /usr/local/bcbio/bin/py2.7 -V
Pythonpy 0.4.11
Python 3.6.10

Should "py2" and "py2.7" not be using Python 2.7?

Thanks for looking into the problem,

Tony.

choosehappy commented 4 years ago

FYI, just installed and experiencing the same error

bcbio / bcbio-nextgen

hisat2_read_statistics.py syntax error #3181

!/bin/bash

SBATCH -N 1

SBATCH --output=/work/EOC/log/paganetti_analysis.log