How can I install Peregrine python packages?

[fmalmeida]

Hi, I was trying to sketch a genome using peregrine and shimmer as shown here however I am failling to install the python packages!

How can that be done? I even tried to use the install_with_conda.sh file but without success.

Any thoughts?

Thank you.

[cschin]

have you installed conda?

[fmalmeida]

Yes, I have.

 conda info

 active environment : base
 active env location : /Users/falmeida/miniconda
        shell level : 1
   user config file : /Users/falmeida/.condarc
 populated config files : /Users/falmeida/.condarc
      conda version : 4.8.3
 conda-build version : not installed
     python version : 3.7.6.final.0
   virtual packages : __osx=10.14.6
   base environment : /Users/falmeida/miniconda  (writable)
       channel URLs : https://conda.anaconda.org/bioconda/osx-64
                      https://conda.anaconda.org/bioconda/noarch
                      https://conda.anaconda.org/conda-forge/osx-64
                      https://conda.anaconda.org/conda-forge/noarch
                      https://repo.anaconda.com/pkgs/main/osx-64
                      https://repo.anaconda.com/pkgs/main/noarch
                      https://repo.anaconda.com/pkgs/r/osx-64
                      https://repo.anaconda.com/pkgs/r/noarch
                      https://conda.anaconda.org/r/osx-64
                      https://conda.anaconda.org/r/noarch
                      https://conda.anaconda.org/anaconda/osx-64
                      https://conda.anaconda.org/anaconda/noarch
      package cache : /Users/falmeida/miniconda/pkgs
                      /Users/falmeida/.conda/pkgs
      envs directories : /Users/falmeida/miniconda/envs
                      /Users/falmeida/.conda/envs
           platform : osx-64
         user-agent : conda/4.8.3 requests/2.23.0 CPython/3.7.6 Darwin/18.7.0 OSX/10.14.6
            UID:GID : 514:20
         netrc file : None
       offline mode : False

When I try to install it with the script everything goes ok until the make step, where I receive the follwing error:

~/Peregrine
~/Peregrine/src ~/Peregrine
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror   -c -o kalloc.o kalloc.c
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror   -c -o shmr_utils.o shmr_utils.c
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror    shmr_mkseqdb.c kalloc.o shmr_utils.o  -lz -o shmr_mkseqdb
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror   -c -o shmr_reduce.o shmr_reduce.c
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror   -c -o mm_sketch.o mm_sketch.c
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror   -c -o shmr_end_filter.o shmr_end_filter.c
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror    shmr_index.c kalloc.o shmr_reduce.o mm_sketch.o shmr_utils.o shmr_end_filter.o  -lz -o shmr_index
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror   -c -o shmr_gather_mc.o shmr_gather_mc.c
gcc   shmr_gather_mc.o kalloc.o shmr_utils.o  -lz -o shmr_gather_mc
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror   -c -o DWmatch.o DWmatch.c
gcc -O3  -Wall -Wno-unused-result -Wno-unused-function -Werror    shmr_overlap.c shmr_utils.o kalloc.o DWmatch.o  -lz -o shmr_overlap
shmr_overlap.c:135:13: error: taking the absolute value of unsigned type 'unsigned int' has no effect [-Werror,-Wabsolute-value]
           (abs(slen0 - q_end) < READ_END_FUZZINESS ||
            ^
shmr_overlap.c:135:13: note: remove the call to 'abs' since unsigned values cannot be negative
           (abs(slen0 - q_end) < READ_END_FUZZINESS ||
            ^~~
shmr_overlap.c:136:13: error: taking the absolute value of unsigned type 'unsigned int' has no effect [-Werror,-Wabsolute-value]
            abs(slen1 - t_end) < READ_END_FUZZINESS)) &&
            ^
shmr_overlap.c:136:13: note: remove the call to 'abs' since unsigned values cannot be negative
            abs(slen1 - t_end) < READ_END_FUZZINESS)) &&
            ^~~
shmr_overlap.c:142:13: error: taking the absolute value of unsigned type 'unsigned int' has no effect [-Werror,-Wabsolute-value]
        if (abs(rlen0 - (q_end - q_bgn)) < READ_END_FUZZINESS * 2 ||
            ^
shmr_overlap.c:142:13: note: remove the call to 'abs' since unsigned values cannot be negative
        if (abs(rlen0 - (q_end - q_bgn)) < READ_END_FUZZINESS * 2 ||
            ^~~
shmr_overlap.c:143:13: error: taking the absolute value of unsigned type 'unsigned int' has no effect [-Werror,-Wabsolute-value]
            abs(rlen1 - (t_end - t_bgn)) < READ_END_FUZZINESS * 2) {
            ^
shmr_overlap.c:143:13: note: remove the call to 'abs' since unsigned values cannot be negative
            abs(rlen1 - (t_end - t_bgn)) < READ_END_FUZZINESS * 2) {
            ^~~
4 errors generated.
make: *** [shmr_overlap] Error 1
cp shmr_mkseqdb shmr_index shmr_gather_mc shmr_overlap shmr_dedup shmr_map /Users/falmeida/miniconda/bin
cp: shmr_overlap: No such file or directory
cp: shmr_dedup: No such file or directory
cp: shmr_map: No such file or directory
make: *** [install] Error 1
~/Peregrine

Also, when I try to import the package from python I receive this error:

import peregrine.utils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/falmeida/miniconda/lib/python3.7/site-packages/peregrine-0.1.6.1+5.g68ddb4f.dirty-py3.7-macosx-10.9-x86_64.egg/peregrine/utils.py", line 4, in <module>
    from peregrine._shimmer4py import ffi as shimmer_ffi
ImportError: dynamic module does not define module export function (PyInit__shimmer4py)

[cschin]

Ok, it seems your compiler version is different from mine. Try to remove CFLAGS+=-Wall -Wno-unused-result -Wno-unused-function -Werror in the Makefile and test it again. In the meantime, I think your compiler catches a bug. I should have done a type casting for those variables before taking the absolute value.

[fmalmeida]

Nice! Thanks for the help.

Removing this line of code the error message changes:

gcc -O3    -c -o kalloc.o kalloc.c
gcc -O3    -c -o shmr_utils.o shmr_utils.c
gcc -O3     shmr_mkseqdb.c kalloc.o shmr_utils.o  -lz -o shmr_mkseqdb
gcc -O3    -c -o shmr_reduce.o shmr_reduce.c
gcc -O3    -c -o mm_sketch.o mm_sketch.c
gcc -O3    -c -o shmr_end_filter.o shmr_end_filter.c
gcc -O3     shmr_index.c kalloc.o shmr_reduce.o mm_sketch.o shmr_utils.o shmr_end_filter.o  -lz -o shmr_index
gcc -O3    -c -o shmr_gather_mc.o shmr_gather_mc.c
gcc   shmr_gather_mc.o kalloc.o shmr_utils.o  -lz -o shmr_gather_mc
gcc -O3    -c -o DWmatch.o DWmatch.c
gcc -O3     shmr_overlap.c shmr_utils.o kalloc.o DWmatch.o  -lz -o shmr_overlap
shmr_overlap.c:135:13: warning: taking the absolute value of unsigned type 'unsigned int' has no effect [-Wabsolute-value]
           (abs(slen0 - q_end) < READ_END_FUZZINESS ||
            ^
shmr_overlap.c:135:13: note: remove the call to 'abs' since unsigned values cannot be negative
           (abs(slen0 - q_end) < READ_END_FUZZINESS ||
            ^~~
shmr_overlap.c:136:13: warning: taking the absolute value of unsigned type 'unsigned int' has no effect [-Wabsolute-value]
            abs(slen1 - t_end) < READ_END_FUZZINESS)) &&
            ^
shmr_overlap.c:136:13: note: remove the call to 'abs' since unsigned values cannot be negative
            abs(slen1 - t_end) < READ_END_FUZZINESS)) &&
            ^~~
shmr_overlap.c:142:13: warning: taking the absolute value of unsigned type 'unsigned int' has no effect [-Wabsolute-value]
        if (abs(rlen0 - (q_end - q_bgn)) < READ_END_FUZZINESS * 2 ||
            ^
shmr_overlap.c:142:13: note: remove the call to 'abs' since unsigned values cannot be negative
        if (abs(rlen0 - (q_end - q_bgn)) < READ_END_FUZZINESS * 2 ||
            ^~~
shmr_overlap.c:143:13: warning: taking the absolute value of unsigned type 'unsigned int' has no effect [-Wabsolute-value]
            abs(rlen1 - (t_end - t_bgn)) < READ_END_FUZZINESS * 2) {
            ^
shmr_overlap.c:143:13: note: remove the call to 'abs' since unsigned values cannot be negative
            abs(rlen1 - (t_end - t_bgn)) < READ_END_FUZZINESS * 2) {
            ^~~
4 warnings generated.
gcc -O3     shmr_dedup.c kalloc.o  -lz -o shmr_dedup
gcc -O3     shmr_map.c shmr_utils.o kalloc.o DWmatch.o  -lz -o shmr_map

[cschin]

@fmalmeida I commit a fix. Also, I notice you are using a Mac OS. My suggestion is to try to the docker image build running on Mac. We have successfully assembled a plant genome of 1.2G bp size on a Mac before with the docker image built.

[fmalmeida]

Thanks @cschin for your help.

I'll follow your suggestion and use the Docker image that you provide.

Have a nice day!