search

CGATOxford / UMI-tools

Tools for handling Unique Molecular Identifiers in NGS data sets
MIT License
491 stars 190 forks source link

umi_tools dedup output empty bam and sort: unrecognized option '--no-PG' error #571

Closed janinelux closed 1 year ago

janinelux commented 1 year ago

Dear all, I am having similar problems as in #463 and #562 posts. I am currently working with the example data provided by umi-tools and executing the following on a HPC:

module load ..../UMI-tools/1.1.2 #load UMI tools 
pip install --user pysam==0.16.0.1 #did this to have the correct pysam version as advised in other posts
pip freeze
umi_tools dedup -I example.bam --output-stats=deduplicated -S deduplicated.bam --log=deduplicated.log

in my output file I see the following, which makes me feel like for me the pysam version is not the problem:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting pysam==0.16.0.1
  Downloading https://files.pythonhosted.org/packages/9b/ab/a7f2637b5bc8fb278aa9b5a5d4eab70856625c09b1d95328e3b78132847c/pysam-0.16.0.1-cp27-cp27mu-manylinux1_x86_64.whl (9.2MB)
Installing collected packages: pysam
  Found existing installation: pysam 0.20.0
    Uninstalling pysam-0.20.0:
      Successfully uninstalled pysam-0.20.0
Successfully installed pysam-0.16.0.1
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
absl-py==0.7.1
ad==1.3.2
alabaster==0.7.9
alignlib-lite==0.3
apipkg==1.4
appdirs==1.4.0
argcomplete==1.5.1
argh==0.26.1
asciitree==0.3.3
astor==0.8.0
astunparse==1.6.2
atomicwrites==1.1.5
attrs==17.4.0
autowrap==0.13.0
avro==1.8.2
Babel==2.3.4
backports-abc==0.5
backports.functools-lru-cache==1.5
backports.ssl-match-hostname==3.5.0.1
bcolz==1.1.2
beautifulsoup4==4.6.3
bgen-reader==2.0.5
bioformats==0.1.15
biolib==0.0.46
biopython==1.74
blessings==1.7
blist==1.3.6
blosc==1.5.1
Bottleneck==1.2.1
breathe==4.7.3
brent-search==1.0.33
brewer2mpl==1.4.1
build-capi==1.1.10
bumpy==0.4.3
bz2file==0.98
cachetools==2.1.0
capturer==2.4
cffi==1.11.5
cftime==1.0.1
chardet==3.0.4
CherryPy==3.5.0
click==6.7
cloudpickle==0.5.3
colorama==0.3.7
coloredlogs==10.0
colorlog==3.1.4
colormap==1.0.1
colormath==3.0.0
ConfigArgParse==0.12.0
configparser==3.5.0b2
Coopr==4.0.9597
cov-core==1.15.0
coverage==4.2
coveralls==1.2.0
cryptography==1.5.3
cssselect==0.9.2
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.29.6
cytoolz==0.9.0.1
cyvcf2==0.7.4
dask==0.19.0
DateTime==4.3
datrie==0.7.1
ddt==1.2.0
deap==1.2.2
decorator==4.3.0
deepsmiles==1.0.1
defusedxml==0.5.0
DendroPy==4.2.0
depinfo==1.4.0
distributed==1.22.0
Django==1.10.3
docopt==0.6.2
docutils==0.12
drmaa==0.7.7
easydev==0.9.37
enum34==1.0.4
et-xmlfile==1.0.1
eta===0.9.8f
execnet==1.4.1
Fabric==1.14.0
fastcache==1.0.2
fastcluster==1.1.23
fisher==0.1.5
Flask==0.11.1
forgi==1.0
freezegun==0.3.12
fudge==1.1.0
funcsigs==1.0.2
functools32==3.2.3.post2
future==0.16.0
futures==3.0.5
fuzzywuzzy==0.15.1
gast==0.2.2
GDAL==2.2.3
geneimpacts==0.3.2
gffutils==0.9
ggplot==0.11.5
gitdb2==2.0.3
GitPython==2.1.11
glimix-core==2.0.0
gmpy2==2.0.8
google-pasta==0.1.6
h5py==2.8.0
hdmedians==0.13
HeapDict==1.0.0
hmmlearn==0.2.0
html5lib==1.0.1
httplib2==0.9.2
humanfriendly==4.16.1
husl==4.0.3
idna==2.5
imagesize==0.7.1
inheritance==0.1.3
ipaddress==1.0.16
iso8601==0.1.11
isodate==0.5.4
itsdangerous==0.24
jdcal==1.3
Jinja2==2.10
joblib==0.12.3
jsonschema==2.5.1
keepalive==0.5
kiwisolver==1.0.1
libroadrunner==1.5.3
liknorm==1.1.13
limix==2.0.0a3
limix-core==1.0.1
limix-legacy==0.8.12
limix-plot==0.0.5
lit==0.6.0rc1
llvmlite==0.24.0
locket==0.2.0
lockfile==0.11.0
Logbook==1.0.0
logging==0.4.9.6
logging-tree==1.7
lxml==4.2.5
lz4==2.2.1
lzstring==1.0.3
Markdown==2.6.11
markdown2==2.3.5
MarkupSafe==0.23
mathstats==0.2.6
matplotlib==2.2.3
matplotlib-venn==0.11.5
mlpy==3.5.0
mock==2.0.0
MOFA==1.0
monotonic==1.5
more-itertools==4.1.0
mpi4py==2.0.0
mpld3==0.3
mpmath==0.19
msgpack==0.5.6
multiprocessing==2.6.2.1
mysql-connector-python==1.1.6
mysqlclient==1.3.7
natsort==5.0.1
nb2plots==0.6
ncephes==1.0.40
ndarray-listener==1.1.2
ndg-httpsclient==0.4.2
netCDF4==1.4.1
networkx==2.2
nose==1.3.7
nose-parameterized==0.3.5
nose-testconfig==0.10
nose2==0.6.5
numba==0.39.0
numexpr==2.6.1
numpy==1.16.2
numpy-sugar==1.2.8
numpydoc==0.5
olefile==0.45.1
openpyxl==2.4.4
optimix==2.0.0
optlang==1.4.2
palettable==3.0.0
pandas==0.23.4
pandas-plink==1.2.28
paramiko==2.1.2
partd==0.3.8
pathlib==1.0.1
pathlib2==2.3.2
patsy==0.4.1
pbr==1.10.0
pdbx-mmcif==0.0.2
PeakUtils==1.1.0
peddy==0.2.9
pep8==1.7.0
pexpect==4.1.0
Pillow==5.2.0
Pint==0.9
pipdeptree==0.12.1
plac==0.9.6
pluggy==0.8.1
ply==3.8
pomegranate==0.11.2
protobuf==3.7.1
psutil==5.2.2
psycopg2==2.6.2
ptyprocess==0.5.1
pwquality==1.4.0
py==1.5.4
py2bit==0.2.2
py4j==0.10.6
pyasn1==0.1.9
pyasn1-modules==0.0.8
pybedtools==0.7.10
pyBigWig==0.3.10
pybktree==1.1
pycmd==1.2
pycparser==2.14
pycurl==7.43.0
pydot==1.0.29
pydotplus==2.0.2
PyDP==0.2.4
pyfaidx==0.5.5.2
pyglet==1.2.4
Pygments==2.1.3
pygobject==3.22.0
pygraphviz==1.4rc1
pymatbridge==0.5.2
pymc==2.3.7
Pyomo==5.2
PyOpenGL==3.1.1a1
PyOpenGL-accelerate==3.1.1a1
pyOpenSSL==16.2.0
pyparsing==2.3.0
pyplink==1.3.4
pyqi==0.3.2
pyqtgraph==0.10.0
Pyrex==0.9.9
pysam==0.16.0.1
PySocks==1.5.7
pytest==3.9.3
pytest-cache==1.0
pytest-cov==2.5.1
pytest-doctestplus==0.1.3
pytest-pep8==1.0.6
pytest-runner==4.0
python-dateutil==2.5.3
python-distutils-extra==2.39
python-igraph==0.7.1.post6
python-Levenshtein==0.12.0
python-libsbml==5.18.0
python-lzo==1.11
python-qsoptex==0.5
python-snappy==0.5
pytz==2016.6.1
PyUtilib==5.5
PyVCF==0.6.8
PyYAML==4.2b4
pyzmq==17.0.0
ratelimiter==1.2.0.post0
rdflib==4.2.1
redis==2.10.6
rednose==1.2.2
regex==2016.9.22
reportlab==3.3.0
repoze.lru==0.4
requests==2.19.1
rrdtool==0.1.10
ruamel.ordereddict==0.4.9
ruamel.yaml==0.13.13
ruffus==2.6.3
scandir==1.9.0
scikit-learn==0.19.1
scipy==1.2.1
scipy-sugar==1.0.7
seaborn==0.9.0
sh==1.12.14
simplejson==3.10.0
singledispatch==3.4.0.3
six==1.15.0
smmap2==2.0.3
snowballstemmer==1.2.1
SOAPpy==0.12.22
sortedcontainers==2.0.4
SPARQLWrapper==1.7.6
spectra==0.0.11
Sphinx==1.4.8
sphinx-rtd-theme==0.1.9
sphinxtesters==0.2.1
spur==0.3.17
SQLAlchemy==1.1.2
statsmodels==0.9.0
subprocess32==3.2.6
sure==1.4.11
svgwrite==1.2.1
swalign==0.3.4
swiglpk==1.4.3
sympy==1.0
tables==3.3.0
tabulate==0.7.5
tblib==1.3.2
termcolor==1.1.0
termstyle==0.1.11
texext==0.6.1
Theano==0.9.0
toolshed==0.4.6
toolz==0.9.0
toposort==1.5
torch==0.4.1
torchvision==0.2.1
tornado==5.0.2
tox==2.3.1
tqdm==4.25.0
typing==3.6.1
ujson==2.0
umi-tools==1.1.2
urllib3==1.21.1
vcf-parser==1.6
virtualenv==15.0.3
web.py==0.38
weblogo==3.5.0
Werkzeug==0.14.1
wget==3.2
wrapt==1.10.10
wstools==0.4.8
wxPython==3.0.2.0
wxPython-common==3.0.2.0
xarray==0.10.8
xlrd==0.9.4
XlsxWriter==1.0.5
xlwt==1.1.2
xmlbuilder==1.0
xmltramp2==3.1.1
xopen==0.8.2
zict==0.1.3
zope.event==4.2.0
zope.interface==4.3.3
sort: unrecognized option '--no-PG'
Traceback (most recent call last):
  File "/software/UHTS/Analysis/UMI-tools/1.1.2/bin/umi_tools", line 11, in <module>
    load_entry_point('umi-tools==1.1.2', 'console_scripts', 'umi_tools')()
  File "/software/UHTS/Analysis/UMI-tools/1.1.2/lib64/python3.5/site-packages/umi_tools/umi_tools.py", line 61, in main
    module.main(sys.argv)
  File "/software/UHTS/Analysis/UMI-tools/1.1.2/lib64/python3.5/site-packages/umi_tools/dedup.py", line 373, in main
    pysam.sort("-o", sorted_out_name, "-O", sort_format, "--no-PG", out_name)
  File "/software/lib64/python3.5/site-packages/pysam/utils.py", line 75, in __call__
    stderr))
pysam.utils.SamtoolsError: 'samtools returned with error 1: stdout=, stderr=Usage: samtools sort [options...] [in.bam]\nOptions:\n  -l INT     Set compression level, from 0 (uncompressed) to 9 (best)\n  -m INT     Set maximum memory per thread; suffix K/M/G recognized [768M]\n  -n         Sort by read name\n  -t TAG     Sort by value of TAG. Uses position as secondary index (or read name if -n is set)\n  -o FILE    Write final output to FILE rather than standard output\n  -T PREFIX  Write temporary files to PREFIX.nnnn.bam\n      --input-fmt-option OPT[=VAL]\n               Specify a single input file format option in the form\n               of OPTION or OPTION=VALUE\n  -O, --output-fmt FORMAT[,OPT[=VAL]]...\n               Specify output format (SAM, BAM, CRAM)\n      --output-fmt-option OPT[=VAL]\n               Specify a single output file format option in the form\n               of OPTION or OPTION=VALUE\n      --reference FILE\n               Reference sequence FASTA FILE [null]\n  -@, --threads INT\n               Number of additional threads to use [0]\n'

I am happy for any suggestions on what I could try, Thank you

janinelux commented 1 year ago

also with pysam==0.20.0 I am getting the same problem

IanSudbery commented 1 year ago

One thing that raises my suspicions here is the references to python 2.7 in the output, and the cp27 in the download record for pysam, even though the error is coming from an umi-tools installation in a directory called python3.5.

I wonder if pip is installing pysam in the same place that umi-tools is calling it from.

Can you try the following?

$ python
> import pysam
> pysam.__version__
> exit()

$ /software/UHTS/Analysis/UMI-tools/1.1.2/bin/python
> import pysam
> pysam.__version__

and let me know the output of both starting python and running version.

for sort, pysam calls out to samtools. Properly installed, pysam installation should ensure the correct version of samtools is being installed, but I wonder if that is being overwritten. Could you also try:

$ samtools --version
janinelux commented 1 year ago

Thank you very much for your comment. I will try your suggestions. In the meantime I have also tried:

module load Anaconda3
eval "$(conda shell.bash hook)" # Alway use this command after loading Anaconda3 (also in your job script)
conda activate pysam_fix
module load vital-it/7
module load Python/3.9.5-GCCcore-10.3.0
module load UHTS/Analysis/UMI-tools/1.1.2
pip install --user pysam==0.16.0.1
pip freeze
umi_tools dedup -I example.bam --output-stats=deduplicated -S deduplicated.bam --log=deduplicated.log

with the output:


The following have been reloaded with a version change:
  1) zlib/.1.2.11 => zlib/.1.2.11-GCCcore-10.3.0

Collecting pysam==0.16.0.1
  Downloading pysam-0.16.0.1.tar.gz (3.6 MB)
Building wheels for collected packages: pysam
  Building wheel for pysam (setup.py): started
  Building wheel for pysam (setup.py): still running...
  Building wheel for pysam (setup.py): still running...
  Building wheel for pysam (setup.py): finished with status 'done'
  Created wheel for pysam: filename=pysam-0.16.0.1-cp39-cp39-linux_x86_64.whl size=9260872 sha256=9004b769b9d16f49bf3e818ecd80dd5bbde81f94eae49fd0cbb59578aa751c0a
  Stored in directory: /storage/homefs/jl20o752/.cache/pip/wheels/fb/96/5b/d62b12ccae437c41f0368bca424aa34b6d3f9fef4a38640edc
Successfully built pysam
Installing collected packages: pysam
Successfully installed pysam-0.16.0.1
WARNING: You are using pip version 21.1.1; however, version 22.3.1 is available.
You should consider upgrading via the '/software.el7/software/Python/3.9.5-GCCcore-10.3.0/bin/python3.9 -m pip install --upgrade pip' command.
alabaster @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/alabaster/alabaster-0.7.12
appdirs @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/appdirs/appdirs-1.4.4
asn1crypto @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/asn1crypto/asn1crypto-1.4.0
atomicwrites @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/atomicwrites/atomicwrites-1.4.0
attrs @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/attrs/attrs-21.2.0
Babel @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/Babel/Babel-2.9.1
bcrypt @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/bcrypt/bcrypt-3.2.0
biopython==1.78
bitstring @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/bitstring/bitstring-3.1.7
blist @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/blist/blist-1.3.6
CacheControl @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/CacheControl/CacheControl-0.12.6
cachy @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/cachy/cachy-0.3.0
certifi==2020.12.5
cffi @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/cffi/cffi-1.14.5
chardet @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/chardet/chardet-4.0.0
cleo @ file:///software.el7/sources/p/Python/extensions/cleo-0.8.1-py2.py3-none-any.whl
click==7.1.2
clikit @ file:///software.el7/sources/p/Python/extensions/clikit-0.6.2-py2.py3-none-any.whl
colorama @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/colorama/colorama-0.4.4
crashtest @ file:///software.el7/sources/p/Python/extensions/crashtest-0.3.1-py3-none-any.whl
cryptography @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/cryptography/cryptography-3.4.7
Cython @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/Cython/Cython-0.29.23
decorator @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/decorator/decorator-5.0.7
distlib @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/distlib/distlib-0.3.1
docopt @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/docopt/docopt-0.6.2
docutils @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/docutils/docutils-0.17.1
ecdsa @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/ecdsa/ecdsa-0.16.1
filelock==3.0.12
flit @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/flit/flit-3.2.0
flit-core @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/flitcore/flit_core-3.2.0
fsspec @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/fsspec/fsspec-2021.4.0
future @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/future/future-0.18.2
html5lib @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/html5lib/html5lib-1.1
idna==2.10
imagesize @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/imagesize/imagesize-1.2.0
importlib-metadata @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/importlib_metadata/importlib_metadata-4.0.1
iniconfig @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/iniconfig/iniconfig-1.1.1
intervaltree @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/intervaltree/intervaltree-3.1.0
intreehooks @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/intreehooks/intreehooks-1.0
ipaddress @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/ipaddress/ipaddress-1.0.23
jeepney @ file:///software.el7/sources/p/Python/extensions/jeepney-0.6.0-py3-none-any.whl
Jinja2 @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/Jinja2/Jinja2-2.11.3
joblib==1.0.0
jsonschema @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/jsonschema/jsonschema-3.2.0
keyring @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/keyring/keyring-21.8.0
keyrings.alt @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/keyringsalt/keyrings.alt-4.0.2
liac-arff @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/liacarff/liac-arff-2.5.0
lockfile @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/lockfile/lockfile-0.12.2
MarkupSafe @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/MarkupSafe/MarkupSafe-1.1.1
mock @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/mock/mock-4.0.3
more-itertools @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/moreitertools/more-itertools-8.7.0
msgpack @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/msgpack/msgpack-1.0.2
netaddr @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/netaddr/netaddr-0.8.0
netifaces @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/netifaces/netifaces-0.10.9
nose @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/nose/nose-1.3.7
numpy==1.19.4
packaging==20.8
pandas==1.1.5
paramiko @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/paramiko/paramiko-2.7.2
pastel @ file:///software.el7/sources/p/Python/extensions/pastel-0.2.1-py2.py3-none-any.whl
pathlib2 @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pathlib2/pathlib2-2.3.5
paycheck @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/paycheck/paycheck-1.0.2
pbr @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pbr/pbr-5.6.0
pepnn @ file:///storage/homefs/jl20o752/PepNN/pepnn
pexpect @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pexpect/pexpect-4.8.0
Pillow==8.0.1
pkginfo @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pkginfo/pkginfo-1.7.0
pluggy @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pluggy/pluggy-0.13.1
poetry @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/poetry/poetry-1.1.6
poetry-core @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/poetrycore/poetry-core-1.0.3
psutil @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/psutil/psutil-5.8.0
ptyprocess @ file:///software.el7/sources/p/Python/extensions/ptyprocess-0.7.0-py2.py3-none-any.whl
py @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/py/py-1.10.0
py-expression-eval @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/py_expression_eval/py_expression_eval-0.3.13
pyasn1 @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pyasn1/pyasn1-0.4.8
pycparser @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pycparser/pycparser-2.20
pycrypto @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pycrypto/pycrypto-2.6.1
Pygments @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/Pygments/Pygments-2.9.0
pylev @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pylev/pylev-1.3.0
PyNaCl @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/PyNaCl/PyNaCl-1.4.0
pyparsing==2.4.7
pyrsistent @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pyrsistent/pyrsistent-0.17.3
pysam==0.16.0.1
pytest @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pytest/pytest-6.2.4
python-dateutil==2.8.1
pytoml @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/pytoml/pytoml-0.1.21
pytz==2020.4
regex==2020.11.13
requests==2.25.1
requests-toolbelt @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/requeststoolbelt/requests-toolbelt-0.9.1
sacremoses==0.0.43
scandir @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/scandir/scandir-1.10.0
scipy==1.5.4
SecretStorage @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/SecretStorage/SecretStorage-3.3.1
semantic-version @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/semantic_version/semantic_version-2.8.5
setuptools-rust @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/setuptoolsrust/setuptools-rust-0.12.1
setuptools-scm @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/setuptools_scm/setuptools_scm-6.0.1
shellingham @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/shellingham/shellingham-1.4.0
simplegeneric @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/simplegeneric/simplegeneric-0.8.1
simplejson @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/simplejson/simplejson-3.17.2
six==1.15.0
snowballstemmer @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/snowballstemmer/snowballstemmer-2.1.0
sortedcontainers @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/sortedcontainers/sortedcontainers-2.3.0
Sphinx @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/Sphinx/Sphinx-4.0.0
sphinx-bootstrap-theme @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/sphinxbootstraptheme/sphinx-bootstrap-theme-0.7.1
sphinxcontrib-applehelp @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/sphinxcontribapplehelp/sphinxcontrib-applehelp-1.0.2
sphinxcontrib-devhelp @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/sphinxcontribdevhelp/sphinxcontrib-devhelp-1.0.2
sphinxcontrib-htmlhelp @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/sphinxcontribhtmlhelp/sphinxcontrib-htmlhelp-1.0.3
sphinxcontrib-jsmath @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/sphinxcontribjsmath/sphinxcontrib-jsmath-1.0.1
sphinxcontrib-qthelp @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/sphinxcontribqthelp/sphinxcontrib-qthelp-1.0.3
sphinxcontrib-serializinghtml @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/sphinxcontribserializinghtml/sphinxcontrib-serializinghtml-1.1.4
sphinxcontrib-websupport @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/sphinxcontribwebsupport/sphinxcontrib-websupport-1.2.4
tabulate @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/tabulate/tabulate-0.8.9
threadpoolctl @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/threadpoolctl/threadpoolctl-2.1.0
tokenizers==0.9.4
toml @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/toml/toml-0.10.2
tomlkit @ file:///software.el7/sources/p/Python/extensions/tomlkit-0.7.0-py2.py3-none-any.whl
torch==1.7.1
torchaudio==0.7.2
torchvision==0.8.2
tqdm==4.54.1
transformers==4.1.1
typing-extensions==3.7.4.3
ujson @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/ujson/ujson-4.0.2
umi-tools==1.1.2
urllib3==1.26.2
virtualenv @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/virtualenv/virtualenv-20.4.6
wcwidth @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/wcwidth/wcwidth-0.2.5
webencodings @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/webencodings/webencodings-0.5.1
wincertstore==0.2
xlrd @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/xlrd/xlrd-2.0.1
zipp @ file:///scratch/local/13250877/Python/3.9.5/GCCcore-10.3.0/zipp/zipp-3.4.1
sort: unrecognized option '--no-PG'
Traceback (most recent call last):
  File "/software/UHTS/Analysis/UMI-tools/1.1.2/bin/umi_tools", line 11, in <module>
    load_entry_point('umi-tools==1.1.2', 'console_scripts', 'umi_tools')()
  File "/software/UHTS/Analysis/UMI-tools/1.1.2/lib64/python3.5/site-packages/umi_tools/umi_tools.py", line 61, in main
    module.main(sys.argv)
  File "/software/UHTS/Analysis/UMI-tools/1.1.2/lib64/python3.5/site-packages/umi_tools/dedup.py", line 373, in main
    pysam.sort("-o", sorted_out_name, "-O", sort_format, "--no-PG", out_name)
  File "/software/lib64/python3.5/site-packages/pysam/utils.py", line 75, in __call__
    stderr))
pysam.utils.SamtoolsError: 'samtools returned with error 1: stdout=, stderr=Usage: samtools sort [options...] [in.bam]\nOptions:\n  -l INT     Set compression level, from 0 (uncompressed) to 9 (best)\n  -m INT     Set maximum memory per thread; suffix K/M/G recognized [768M]\n  -n         Sort by read name\n  -t TAG     Sort by value of TAG. Uses position as secondary index (or read name if -n is set)\n  -o FILE    Write final output to FILE rather than standard output\n  -T PREFIX  Write temporary files to PREFIX.nnnn.bam\n      --input-fmt-option OPT[=VAL]\n               Specify a single input file format option in the form\n               of OPTION or OPTION=VALUE\n  -O, --output-fmt FORMAT[,OPT[=VAL]]...\n               Specify output format (SAM, BAM, CRAM)\n      --output-fmt-option OPT[=VAL]\n               Specify a single output file format option in the form\n               of OPTION or OPTION=VALUE\n      --reference FILE\n               Reference sequence FASTA FILE [null]\n  -@, --threads INT\n               Number of additional threads to use [0]\n'
IanSudbery commented 1 year ago

Thats weird. All your depedencies are referencing a python 3.9 directory. Yet your UMI-tools appears to be installed in a python 3.5 directory.

janinelux commented 1 year ago

I tried 

$ python
> import pysam
> pysam.__version__
> exit()

with the output

>>> pysam.__version__
'0.16.0.1'
IanSudbery commented 1 year ago

What python version was quoted when you started python?

janinelux commented 1 year ago

then I did 

cd /software/UHTS/Analysis/UMI-tools/1.1.2/bin/

and there was no "python" folder, the only folder was "umi_tools"

janinelux commented 1 year ago

When I started python it was

Python 2.7.12 (default, Oct 18 2016, 09:15:58)
[GCC 6.1.1 20160510 (Red Hat 6.1.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
IanSudbery commented 1 year ago

Thats python 2.7. Since UMI-tools doesn't work with python 2.7, I'm pretty sure it must be picking up a different python installation when you start UMI-tools to the one that you are installing pysam into.

and there was no "python" folder, the only folder was "umi_tools"

Sorry, I didn't mean cd into a folder called python, but to run the program called python in the /software/UHTS/Analysis/UMI-tools/1.1.2/bin/ directory.

janinelux commented 1 year ago

ah, I am sorry, I misunderstood

cd /software/UHTS/Analysis/UMI-tools/1.1.2/bin/
python
Python 2.7.12 (default, Oct 18 2016, 09:15:58)
[GCC 6.1.1 20160510 (Red Hat 6.1.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysam
>>> pysam.__version__
'0.16.0.1'
IanSudbery commented 1 year ago

Sorry, you'll need to run the command 

$ /software/UHTS/Analysis/UMI-tools/1.1.2/bin/python

because what you did will run the system python, even if you are in the /software/UHTS/Analysis/UMI-tools/1.1.2/bin directory.

janinelux commented 1 year ago 
/software/UHTS/Analysis/UMI-tools/1.1.2/bin/python
-bash: /software/UHTS/Analysis/UMI-tools/1.1.2/bin/python: No such file or directory

There doesn't seem to be a python interpreter there

ls -lt /software/UHTS/Analysis/UMI-tools/1.1.2/bin/
total 1
-rwxr-xr-x 1 l_vitalit l_vitalit 397 Aug 16  2021 umi_tools

I did not install the umi_tools on the HPC, but it works by loading modules:

module load vital-it/7
module load UHTS/Analysis/UMI-tools/1.1.2
IanSudbery commented 1 year ago

Can you do 

$ cat /software/UHTS/Analysis/UMI-tools/1.1.2/bin/umi_tools
janinelux commented 1 year ago 
#!/software/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'umi-tools==1.1.2','console_scripts','umi_tools'
__requires__ = 'umi-tools==1.1.2'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('umi-tools==1.1.2', 'console_scripts', 'umi_tools')()
    )
IanSudbery commented 1 year ago

Right, so umi_tools is using the python install at /software/bin/python3.

Like all python tools that use pkg_resources entry_points as part of their packaging tool chain, it is hard coded to use the python that was active when it was installed. It will now continue to use that python irrespective of what modules or virtual environments you load.

However, /software/bin/python3 is not the currently active python on the system you are using. When you run pip install it installs it for the currently active python, which is python2.7 by default (i'm guessing /usr/bin/python), and /software.el7/software/Python/3.9.5-GCCcore-10.3.0/bin/python3.9 if you have loaded the Python/3.9.5-GCCcore-10.3.0.

If you want to install the correct pysam version for the python that umi_tools is using, I suggest trying 

$ /software/bin/python3 -m pip install pysam==0.16.0.1

Python packing is a mess. But my guess of how it came to this is that you installed umi_tools using a different method or under a different environment to how you are trying to run it?

janinelux commented 1 year ago 
$ /software/bin/python3 -m pip install pysam==0.16.0.1
Collecting pysam==0.16.0.1
  Downloading https://files.pythonhosted.org/packages/ca/09/ad780f4430777a7bfca63ee2d4d4584436d2f31f010d6bab743cfc97da48/pysam-0.16.0.1-cp35-cp35m-manylinux1_x86_64.whl (9.8MB)
     |████████████████████████████████| 9.8MB 91.8MB/s
Installing collected packages: pysam
  Found existing installation: pysam 0.15.2
    Uninstalling pysam-0.15.2:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'libctabix.cpython-35m-x86_64-linux-gnu.so'
Consider using the `--user` option or check the permissions.

Then I did 

$ /software/bin/python3 -m pip install --user pysam==0.16.0.1
Collecting pysam==0.16.0.1
  Using cached https://files.pythonhosted.org/packages/ca/09/ad780f4430777a7bfca63ee2d4d4584436d2f31f010d6bab743cfc97da48/pysam-0.16.0.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: pysam
Successfully installed pysam-0.16.0.1

and ran the script from the beginning again. Now I don't get any error messages in the .out file and the .bam file of the test data has been written with KB, in the .log file the job finally "# job finished in 20 second"

janinelux commented 1 year ago

Thank you very much for your help, I think it is working now. I will start a run with my sample now (which I think will take several hours), I think it might work now (:

IanSudbery commented 1 year ago

The error you got just means that you don't have write permission to the appropriate directory it was trying to install to. Probably /software/lib64/python3.5

janinelux commented 1 year ago

thank you again, it is working fine for me now, also with my data!