Guppy snakemake rocksdb error

[goja288]

Hi, first of all thanks for share your work!

I'm trying to run the snakemake guppy example without success.

This is the command I executed and I'm getting an error related to rocksdb: snakemake -s Guppy guppy_results/example_guppy-freq-perCG.tsv

ImportError: /opt/anaconda/envs/guppy_cpg_snakemake/lib/python3.6/site-packages/rocksdb/_rocksdb.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZNK7rocksdb24AssociativeMergeOperator12PartialMergeERKNS_5SliceES3_S3_PNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS_6LoggerE

I saw a similar comment in other issue #5 so I installed from the source different version (5.3.6, 5.11.3, 7.3.1, 7.4.0), but even so i was't lucky. Maybe the problem is here, in how i am installing rocksdb.

This is the message when i delete rockdb dependency in conda env :

Traceback (most recent call last):
  File "guppy_results/gcf52ref/scripts/extract_methylation_fast5.py", line 180, in <module>
    mdb = MethylDB(args.mod_data)
  File "guppy_results/gcf52ref/scripts/extract_methylation_fast5.py", line 39, in __init__
    import rocksdb
  File "/opt/anaconda/envs/guppy_cpg_snakemake/lib/python3.6/site-packages/rocksdb/__init__.py", line 1, in <module>
    from ._rocksdb import *
ImportError: librocksdb.so.5.3: cannot open shared object file: No such file or directory

Error in rule samtools_index_and_extract_methylation_from_fast5:
    jobid: 3
    output: guppy_results/mapped/example.bam.bai

RuleException:
CalledProcessError in line 17 of METEORE/Guppy:
Command ' set -euo pipefail;  samtools index guppy_results/mapped/example.bam && python guppy_results/gcf52ref/scripts/extract_methylation_fast5.py -p 10 guppy_results/workspace/*.fast5 ' returned non-zero exit status 1.
  File "METEORE/Guppy", line 17, in __rule_samtools_index_and_extract_methylation_from_fast5
  File "/opt/anaconda/envs/guppy_cpg_snakemake/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Removing output files of failed job samtools_index_and_extract_methylation_from_fast5 since they might be corrupted:
guppy_results/mapped/example.bam.bai

I tried in two different computers Ubuntu 20.04.4 LTS Guppy CPU Version 6.1.3

Ubuntu 22.04 Guppy GPU Version 6.1.7

I think there is something related to anaconda and linked lib, I tried to re-install everything several times, change LD_LIBRARY_PATH, use conda package, source package...

I'm still trying, but any hints or ideas are more than welcome.

Thanks in advance

[goja288]

Update: I created a Docker image based on ubuntu:20.04 (also with ubuntu:18.04) to rule out it being a problem with some library in my pc, but when I ran mamba env create -f guppy.yml I have problems with the rocks-db dependency also

  Running setup.py clean for python-rocksdb
Successfully built progressbar33
Failed to build python-rocksdb
Installing collected packages: cython, six, h5py, progressbar33, ont-fast5-api, python-rocksdb
  Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
  Running setup.py install for python-rocksdb: started
    Running setup.py install for python-rocksdb: finished with status 'error'

...
...
...

creating build/temp.linux-x86_64-3.6/rocksdb
    gcc -pthread -B /opt/conda/envs/guppy_cpg_snakemake/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Irocksdb -I./rocksdb -I/opt/conda/envs/guppy_cpg_snakemake/include/python3.6m -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-3.6/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    rocksdb/_rocksdb.cpp:762:10: fatal error: rocksdb/slice.h: No such file or directory
      762 | #include "rocksdb/slice.h"
          |          ^~~~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command "/opt/conda/envs/guppy_cpg_snakemake/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-sj1krrq0/python-rocksdb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-rivb7w7n/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-sj1krrq0/python-rocksdb/

failed

CondaEnvException: Pip failed

In this opportunity i used the release METEORE-1.0.0

I will try to add rocksdb from source to the Dockerfile and remove it from guppy.yml

But maybe you realize what library is missing as a dependency

Thanks again

[goja288]

I managed to make it work for me in a docker container with Ubuntu, I'll put it here in case someone finds it useful

First install rocks-db dependencies Rocksdb INSTALL.md

sudo apt install libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev 

After that, download and install rocksdb

git clone https://github.com/facebook/rocksdb.git && \
cd rocksdb && \
git checkout v5.3.6 && \
make shared_lib && \
mkdir -p /usr/local/rocksdb/lib && \
mkdir /usr/local/rocksdb/include && \
cp librocksdb.so* /usr/local/rocksdb/lib && \
cp /usr/local/rocksdb/lib/librocksdb.so* /usr/lib/ && \
cp -r include /usr/local/rocksdb/ && \
cp -r include/* /usr/include/ && \
rm -R /tmp/rocksdb/

Finally I removed the line - rocksdb=5.3.6=0 from guppy.yml and then continued with the README installation instructions

Thanks again for your work