search

sneumann / xcms

This is the git repository matching the Bioconductor package xcms: LC/MS and GC/MS Data Analysis
Other
178 stars 81 forks source link

NetCDF: Not a valid ID error when using CentOS with NetCDF library 4.6.2 #363

Closed trivedi-group closed 5 years ago

trivedi-group commented 5 years ago

Hi,

I am using the code below on a CentOS node that is on computing cluster. It has installed and loaded R and required packages fine. It can also detect all my netCDF files in the location but as soon as it starts to create my xcmsSet, it fails with an error. 

Loading required package: Biobase
Loading required package: BiocGenerics
Loading required package: parallel

Attaching package: ‘BiocGenerics’

The following objects are masked from ‘package:parallel’:

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following objects are masked from ‘package:stats’:

    IQR, mad, sd, var, xtabs

The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, basename, cbind, colMeans,
    colnames, colSums, dirname, do.call, duplicated, eval, evalq,
    Filter, Find, get, grep, grepl, intersect, is.unsorted, lapply,
    lengths, Map, mapply, match, mget, order, paste, pmax, pmax.int,
    pmin, pmin.int, Position, rank, rbind, Reduce, rowMeans, rownames,
    rowSums, sapply, setdiff, sort, table, tapply, union, unique,
    unsplit, which, which.max, which.min

Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Loading required package: BiocParallel
Loading required package: MSnbase
Loading required package: mzR
Loading required package: Rcpp
Loading required package: S4Vectors
Loading required package: stats4

Attaching package: ‘S4Vectors’

The following object is masked from ‘package:base’:

    expand.grid

Loading required package: ProtGenerics

This is MSnbase version 2.8.3 
  Visit https://lgatto.github.io/MSnbase/ to get started.

Attaching package: ‘MSnbase’

The following object is masked from ‘package:stats’:

    smooth

The following object is masked from ‘package:base’:

    trimws

This is xcms version 3.4.4 

Attaching package: ‘xcms’

The following object is masked from ‘package:stats’:

    sigma

Scanning files in directory /net/lustre/mbdssdt3/test ... found 5 files
Error in R_nc4_close: NetCDF: Not a valid ID

Script:

#Set working directory
setwd("~/scratch/test")

#Load packages
library(xcms)
library(CAMERA)
library(MSnbase)
library(BiocParallel)

##Parallel Processing##
register(bpstart(MulticoreParam(32)))

#xcms
xset0 <- xcmsSet(BPPARAM = bpparam())

I have tried the same files on MacOS and they are OK. I have narrowed it down to being how XCMS works on centOS but am unable to put a finger on what is causing that error. Internet is of no use on this error code on netCDF library. Is this a familiar scenario?

Some of the raw files can be accessed here - https://www.dropbox.com/sh/m6um4ofsmvgw7ct/AABDcbjI_tr54ruHOYqGrLXHa?dl=0

Edit: I've tried (inspired from similar issue here - https://github.com/sneumann/xcms/issues/299) without parallel processing and with - nothing changes. Still get same error.

trivedi-group commented 5 years ago

Linking related issue that might help - I think the reported error 14 days ago might hint that the problem is related to changes that were merged? - https://github.com/sneumann/mzR/issues/185#issuecomment-470486668

sneumann commented 5 years ago

Hi, quick note, does it still fail if you use plain mzR ? openMSfile(). I guess so. If you have a docker container with exact versions that fails that could be helpful. Yours, Steffen

trivedi-group commented 5 years ago

Hi, quick note, does it still fail if you use plain mzR ? openMSfile(). I guess so. If you have a docker container with exact versions that fails that could be helpful. Yours, Steffen

Hi, 'openMSfile()' can handle the file. The output I get is:

Mass Spectrometry file handle. Filename: BATCH1_QC1301.CDF Number of scans: 11214

I've tried a few different files and they all seem to return no error.

jorainer commented 5 years ago

Can you please provide the output of your sessionInfo()?

jorainer commented 5 years ago

With the Bioconductor 3.8 release version of xcms (3.4.4) I can confirm the error:

library(xcms)
library(msdata)
fl <- system.file("cdf/ko15.CDF", package = "msdata")

raw <- xcmsRaw(fl)
Error in R_nc4_close: NetCDF: Not a valid ID
Create profile matrix with method 'bin' and step 1 ... OK

od <- readMSData(fl, mode = "onDisk")
sps <- spectra(od)

Since this does not happen with the new xcms3/MSnbase-way to process data it has to be something within the way we read data prior to xcms3. I'll have a look.

jorainer commented 5 years ago

I've fixed it in the master and RELEASE_3_8 branch of xcms. We were closing the netCDF file even if it was already closed.

You can install the fixed version (assuming you have R-3.5.2 and Bioconductor 3.8 packages) with

devtools::install_github("sneumann/xcms", ref = "RELEASE_3_8")
trivedi-group commented 5 years ago

Can you please provide the output of your sessionInfo()?

I am guessing you don't need this info any longer?

I've fixed it in the master and _RELEASE_38 branch of xcms. We were closing the netCDF file even if it was already closed.

You can install the fixed version (assuming you have R-3.5.2 and Bioconductor 3.8 packages) with

devtools::install_github("sneumann/xcms", ref = "RELEASE_3_8")

Thanks a lot! I've downloaded newer version, will give it a whirl!

jorainer commented 5 years ago

Please also close the issue when you're OK with it (i.e. we've fixed your problem).

trivedi-group commented 5 years ago

I have a feeling something is stopping the newer version to run. I've new set of text of wall and feel it is local machine issue rather than XCMS. Thoughts?

When I run xset = xcmsSet(), now I see this text and process stops. 

rm -f massifquant/xcms_massifquant.o massifquant/TrMgr.o massifquant/Tracker.o massifquant/SegProc.o massifquant/DataKeeper.o massifquant/OpOverload.o obiwarp/mat.o obiwarp/vec.o obiwarp/xcms_dynprog.o obiwarp/xcms_lmat.o xcms_obiwarp.o fastMatch.o mzClust_hclust.o mzROI.o util.o xcms.o binners.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/xcms_massifquant.cpp -o massifquant/xcms_massifquant.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/TrMgr.cpp -o massifquant/TrMgr.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/Tracker.cpp -o massifquant/Tracker.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/SegProc.cpp -o massifquant/SegProc.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/DataKeeper.cpp -o massifquant/DataKeeper.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/OpOverload.cpp -o massifquant/OpOverload.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c obiwarp/mat.cpp -o obiwarp/mat.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c obiwarp/vec.cpp -o obiwarp/vec.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c obiwarp/xcms_dynprog.cpp -o obiwarp/xcms_dynprog.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c obiwarp/xcms_lmat.cpp -o obiwarp/xcms_lmat.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c xcms_obiwarp.cpp -o xcms_obiwarp.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c fastMatch.c -o fastMatch.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c mzClust_hclust.c -o mzClust_hclust.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c mzROI.c -o mzROI.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c util.c -o util.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c xcms.c -o xcms.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c binners.c -o binners.o
/usr/bin/g++ -shared -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/blas/3.6.0/gcc-4.8.5 -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/lapack/3.5.0/gcc-4.8.5/lib -o xcms.so massifquant/xcms_massifquant.o massifquant/TrMgr.o massifquant/Tracker.o massifquant/SegProc.o massifquant/DataKeeper.o massifquant/OpOverload.o obiwarp/mat.o obiwarp/vec.o obiwarp/xcms_dynprog.o obiwarp/xcms_lmat.o xcms_obiwarp.o fastMatch.o mzClust_hclust.o mzROI.o util.o xcms.o binners.o -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -lR
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c fastMatch.c -o fastMatch.o
/usr/bin/gcc -std=gnu99 -shared -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/blas/3.6.0/gcc-4.8.5 -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/lapack/3.5.0/gcc-4.8.5/lib -o CAMERA.so fastMatch.o -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -lR
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c RcppExports.cpp -o RcppExports.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c Spectrum1_class.c -o Spectrum1_class.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c Spectrum2_class.c -o Spectrum2_class.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c imp_neighbour_avg.cpp -o imp_neighbour_avg.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c utils.c -o utils.o
/usr/bin/g++ -shared -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/blas/3.6.0/gcc-4.8.5 -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/lapack/3.5.0/gcc-4.8.5/lib -o MSnbase.so RcppExports.o Spectrum1_class.o Spectrum2_class.o imp_neighbour_avg.o utils.o -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -lR
jorainer commented 5 years ago

This looks like text printed during the installation of xcms. Where you able to install xcms? What version do you have now (i.e. run sessionInfo() after library(xcms))?

trivedi-group commented 5 years ago

This looks like text printed during the installation of xcms. Where you able to install xcms? What version do you have now (i.e. run sessionInfo() after library(xcms))?

You are right. It was an issue with installation of xcms. I've now installed it after removing previous package and it has sailed through. I think it has successfully read most of the files now (got over 600 x 2gb files) but it will take longer to finish. I will close this thread as the issue seems to be resolved. If something new pops up, I'll open another thread.

Thank you for resolving this so quickly!