search

vatlab / varianttools

software tool for the manipulation, annotation, selection, and analysis of variants in the context of next-gen sequencing analysis
https://vatlab.github.io/vat-docs/
GNU General Public License v3.0
31 stars 4 forks source link

Compile error on Linux server #50

Closed HenryLeongStat closed 6 years ago

HenryLeongStat commented 7 years ago

error only happens on Linux server...

seems like the problem is causing by those .nfsxxxxxxxxxxxxxxx files, cannot delete any of them even on command line... 

Processing variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg
removing '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg' (and everything under it)
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs00000002054938b000000026'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs00000002054938a700000024'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs00000002054938a000000022'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs000000020553350600000023'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs00000002054938ae00000025'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs000000020553350e00000027'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 39] Directory not empty: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 39] Directory not empty: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg'
Extracting variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg to /home/mleong/anaconda3/lib/python3.6/site-packages
variant-tools 3.0.0.dev0 is already the active version in easy-install.pth
Installing vtools script to /home/mleong/anaconda3/bin
Installing vtools_report script to /home/mleong/anaconda3/bin
HenryLeongStat commented 7 years ago

I am not sure whether it is the reason for the error in association test.

jma7 commented 7 years ago

In addition to removing "ENV_INCLUDE_DIRS" and "ENV_LIBRARY_DIRS" from 

ext_modules+=cythonize([Extension('variant_tools.io_vcf_read',
    sources=['src/variant_tools/io_vcf_read.pyx'],
    include_dirs=[np.get_include()],
    library_dirs = ["build"])])

I need to remove "ENV_INCLUDE_DIRS" and "ENV_LIBRARY_DIRS" from 

Extension('variant_tools._assoTests',
            sources = [ASSO_WRAPPER_CPP_FILE] + ASSOC_FILES,
            extra_compile_args = gccargs,
            libraries = libs + ['gsl', 'stat'], #, 'blas'],
            # library_dirs = ["build"] + ENV_LIBRARY_DIRS,
            library_dirs = ["build"],
            # include_dirs = ["src", "src/variant_tools", "src/gsl"] + ENV_INCLUDE_DIRS,
            include_dirs = ["src", "src/variant_tools", "src/gsl"],
        )

as well, to make it compile on my local desktop.

HenryLeongStat commented 7 years ago

Let me try that on my end.

HenryLeongStat commented 7 years ago

Same error after I remove "ENV_INCLUDE_DIRS" and "ENV_LIBRARY_DIRS" from 

        Extension('variant_tools._assoTests',
            sources = [ASSO_WRAPPER_CPP_FILE] + ASSOC_FILES,
            extra_compile_args = gccargs,
            libraries = libs + ['gsl', 'stat'], #, 'blas'],
            library_dirs = ["build"],
            include_dirs = ["src", "src/variant_tools", "src/gsl"],
        )
      ]

Error messages:

removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg
removing '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg' (and everything under it)
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs00000002054938b000000026'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs00000002054938a700000024'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs00000002054938a000000022'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs000000020553350600000023'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs00000002054938ae00000025'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 16] Device or resource busy: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools/.nfs000000020553350e00000027'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 39] Directory not empty: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools'
error removing /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg: [Errno 39] Directory not empty: '/home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg'
Extracting variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg to /home/mleong/anaconda3/lib/python3.6/site-packages
variant-tools 3.0.0.dev0 is already the active version in easy-install.pth
Installing vtools script to /home/mleong/anaconda3/bin
Installing vtools_report script to /home/mleong/anaconda3/bin

Not sure what are those .nfsxxxxxxxxxxxxxxx files...

HenryLeongStat commented 7 years ago

It seems like a mounting issue? Let me try to unmount them and then delete them...

Ref : stackexchange

HenryLeongStat commented 7 years ago

Can't even umount them: 

mleong@q1prpfs04:~/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools$ umount .
umount: /home/mleong/anaconda3/lib/python3.6/site-packages/variant_tools-3.0.0.dev0-py3.6-linux-x86_64.egg/variant_tools: umount failed: Operation not permitted
HenryLeongStat commented 7 years ago

Removed those .nfs files and complied successfully!

screen shot 2017-11-06 at 2 14 27 pm screen shot 2017-11-06 at 2 15 43 pm
HenryLeongStat commented 7 years ago

I tried to keep "ENV_INCLUDE_DIRS" and "ENV_LIBRARY_DIRS" from:

        Extension('variant_tools._assoTests',
            sources = [ASSO_WRAPPER_CPP_FILE] + ASSOC_FILES,
            extra_compile_args = gccargs,
            libraries = libs + ['gsl', 'stat'], #, 'blas'],
            library_dirs = ["build"]  + ENV_LIBRARY_DIRS,
            include_dirs = [".", "variant_tools", "gsl"] + ENV_INCLUDE_DIRS,
        )
      ]

and compile again on Linux. It still works...

jma7 commented 7 years ago

To make it compile on the server running gitlab-ci, I have to remove ENV_INCLUDE_DIRS and ENV_LIBRARY_DIRS:

# ENV_INCLUDE_DIRS = os.environ.get('LD_INCLUDE_PATH', '').split(os.pathsep)
# ENV_LIBRARY_DIRS = os.environ.get('LD_LIBRARY_PATH', '').split(os.pathsep)
ENV_INCLUDE_DIRS = []
ENV_LIBRARY_DIRS = []
BoPeng commented 7 years ago

Not sure why because if LD_INCLUDE_PATH is undefined, ENV_INCLUDE_DIRS should be empty anyway. What is the output of

echo $ENV_INCLUDE_PATH

on gitlab-ci?

jma7 commented 7 years ago

There is no output. There is error if ENV_INCLUDE_DIRS is ['']. SoI changed it to [] . 

ENV_INCLUDE_DIRS = os.environ.get('LD_INCLUDE_PATH', '').split(os.pathsep)
ENV_LIBRARY_DIRS = os.environ.get('LD_LIBRARY_PATH', '').split(os.pathsep)
if ENV_INCLUDE_DIRS[0]=='' or ENV_LIBRARY_DIRS[0]=='':
  ENV_INCLUDE_DIRS=[]
  ENV_LIBRARY_DIRS=[]
BoPeng commented 7 years ago

I see, then the code should be something like

ENV_INCLUDE_DIRS = [x for x in os.environ.get('LD_INCLUDE_PATH', '').split(os.pathsep) if x]