Closed joseale2310 closed 3 years ago
Dear Jose
I’ve used IMAGE using glmnet version 2.0-16.
Good luck, Jesper
Best regards,
Jesper Grud Skat Madsen Postdoc, Mandrup group
Functional Genomics & Metabolism Research Unit Department of Biochemistry and Molecular Biology University of Southern Denmark
Email Phone
jgsm@bmb.sdu.dkmailto:jgsm@bmb.sdu.dk +45 65 50 23 36
Web
http://www.sdu.dk/bmb/functionalgenomics
[Description: SDU_BLACK_RGB_png] [Description: FuncGenMet_line_light_bg_small]
Campusvej 55 · 5230 Odense M · Tlf. 6550 1000 · www.sdu.dkhttp://www.sdu.dk/
From: joseale2310 [mailto:notifications@github.com] Sent: 27. maj 2019 09:47 To: JesperGrud/IMAGE Cc: Subscribed Subject: [JesperGrud/IMAGE] GLMNET errors while performing ridge regression (#1)
Dear Jesper,
I am struggling to use the IMAGE method to process data from my lab. I am getting the error below:
Error in UseMethod("predict") :
no applicable method for 'predict' applied to an object of class "list"
Calls: cv.glmnet -> do.call -> cv.elnet -> predict
Execution halted
It seems to be related to the glmnet package, but I am uncertain about where exactly and in which stage does the error occurs. This error is usually happening in the step 5. Performing ridge regression. My files have the same amount of columns and information as the example datasets, so I believe that the error might happen because I might have different versions of the packages used when the algorithm was developed. Could you share an R session with the versions of the packages you use?
Best,
Jose
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/JesperGrud/IMAGE/issues/1?email_source=notifications&email_token=AA7FA52HDV5DQQ2OCED6PCLPXOGXBA5CNFSM4HPZNOM2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GV62MJQ, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AA7FA553W4WNQTQW62ZYS7LPXOGXBANCNFSM4HPZNOMQ.
Thank you! I will give it a try. The version I am using is glmnet_2.0-18. While using this version, I have tried to check the process manually in R. I loaded the TestItManually.R and Stage 1 works good, so I am guessing the problem resides in stage 2 or 3.
Best, Jose.
Dear Jesper,
It seems that going back to the glmnet to your version partially works but I get another error. Please see below the outcome of the IMAGE pipeline.
#### Welcome to IMAGE ####
Your region file contains 68709 regions and 4 conditions across 8 files
Your exon file contains 52468 genes and 4 conditions across 12 files
Your fasta file is /home/dkv252/HOMER/data/genomes/mm9/mm9.fa
You are using 10 processors.
Starting the analysis
Setting up for parallizing motif search
Preparing input file for motif searching
Scanning for motifs
Running analysis in R
1. Reading data into R
2. Analyzing gene expression
3. Converting motif hits to matrix - Takes a while
4. Calculating motif activity - Full model stage
5. Performing ridge regression
Sample 1 completed
Sample 2 completed
Sample 3 completed
Sample 4 completed
Sample 5 completed
Sample 6 completed
Error in predmat[which, seq(nlami)] <- preds :
replacement has length zero
Calls: cv.glmnet -> do.call -> cv.elnet
Execution halted
Sample 7 completedError in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
I have tried to cut the RNA and Enhacer data to only the first two conditions, since it seemed that the first 6 samples worked fine. However, this gives me another error:
#### Welcome to IMAGE ####
Your region file contains 68709 regions and 2 conditions across 4 files
Your exon file contains 52468 genes and 2 conditions across 6 files
Your fasta file is /home/dkv252/HOMER/data/genomes/mm9/mm9.fa
You are using 10 processors.
Starting the analysis
Setting up for parallizing motif search
Preparing input file for motif searching
Scanning for motifs
Running analysis in R
1. Reading data into R
Error in estimateCommonDisp.default(y[tagsinbin, ], group = group, lib.size = lib.size, :
No genes satisfy rowsum filter
Calls: estimateTrendedDisp ... estimateTrendedDisp.default -> estimateCommonDisp -> estimateCommonDisp.default
Execution halted
2. Analyzing gene expressionError in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
Which I guess it is related to the bins that are formed inside the regression.R script. One of those bins must contain no genes with > 5 counts for all the samples. I then filtered the RNA data to have genes that have > 5 counts for all samples and run again IMAGE, without success:
#### Welcome to IMAGE ####
Your region file contains 68709 regions and 2 conditions across 4 files
Your exon file contains 21887 genes and 2 conditions across 6 files
Your fasta file is /home/dkv252/HOMER/data/genomes/mm9/mm9.fa
You are using 10 processors.
Starting the analysis
Setting up for parallizing motif search
Preparing input file for motif searching
Scanning for motifs
Running analysis in R
1. Reading data into R
2. Analyzing gene expression
3. Converting motif hits to matrix - Takes a while
4. Calculating motif activity - Full model stage
Error in { : task 10 failed - "cannot allocate vector of size 395.4 Mb"
Calls: cv.glmnet -> %dopar% -> <Anonymous>
Execution halted
5. Performing ridge regressionError in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
And this error I cannot figure it out. The data seems to be in the correct format, but the analysis fails at an earlier point than using all the samples. Do you have any suggestions of what could I try to make it work?
Best regards,
Jose
Dear Jose,
I’ve never seen these errors before. I think your approach at simplifying before going on is a good idea. If you want I can try to run your files if you send them to me if you want. Regardless, could you please show me the full command you’re issuing to IMAGE? Have you tried running the example files through?
The error in the last entry after filtering seems to be related to the amount of memory available. Can you try to either reduce the number of processors you are using or run IMAGE on a system with more resources?
Best regards, Jesper
Best regards,
Jesper Grud Skat Madsen Postdoc, Mandrup group
Functional Genomics & Metabolism Research Unit Department of Biochemistry and Molecular Biology University of Southern Denmark
Email Phone
jgsm@bmb.sdu.dkmailto:jgsm@bmb.sdu.dk +45 65 50 23 36
Web
http://www.sdu.dk/bmb/functionalgenomics
[Description: SDU_BLACK_RGB_png] [Description: FuncGenMet_line_light_bg_small]
Campusvej 55 · 5230 Odense M · Tlf. 6550 1000 · www.sdu.dkhttp://www.sdu.dk/
From: joseale2310 [mailto:notifications@github.com] Sent: 28. maj 2019 15:19 To: JesperGrud/IMAGE Cc: Jesper Grud Skat Madsen; Comment Subject: Re: [JesperGrud/IMAGE] GLMNET errors while performing ridge regression (#1)
Dear Jesper,
It seems that going back to the glmnet to your version partially works but I get another error. Please see below the outcome of the IMAGE pipeline.
Your region file contains 68709 regions and 4 conditions across 8 files
Your exon file contains 52468 genes and 4 conditions across 12 files
Your fasta file is /home/dkv252/HOMER/data/genomes/mm9/mm9.fa
You are using 10 processors.
Starting the analysis
Setting up for parallizing motif search
Preparing input file for motif searching
Scanning for motifs
Running analysis in R
1. Reading data into R
2. Analyzing gene expression
3. Converting motif hits to matrix - Takes a while
4. Calculating motif activity - Full model stage
5. Performing ridge regression
Sample 1 completed
Sample 2 completed
Sample 3 completed
Sample 4 completed
Sample 5 completed
Sample 6 completed
Error in predmat[which, seq(nlami)] <- preds :
replacement has length zero
Calls: cv.glmnet -> do.call -> cv.elnet
Execution halted
Sample 7 completedError in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
I have tried to cut the RNA and Enhacer data to only the first two conditions, since it seemed that the first 6 samples worked fine. However, this gives me another error:
Your region file contains 68709 regions and 2 conditions across 4 files
Your exon file contains 52468 genes and 2 conditions across 6 files
Your fasta file is /home/dkv252/HOMER/data/genomes/mm9/mm9.fa
You are using 10 processors.
Starting the analysis
Setting up for parallizing motif search
Preparing input file for motif searching
Scanning for motifs
Running analysis in R
1. Reading data into R
Error in estimateCommonDisp.default(y[tagsinbin, ], group = group, lib.size = lib.size, :
No genes satisfy rowsum filter
Calls: estimateTrendedDisp ... estimateTrendedDisp.default -> estimateCommonDisp -> estimateCommonDisp.default
Execution halted
2. Analyzing gene expressionError in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
Which I guess it is related to the bins that are formed inside the regression.R script. One of those bins must contain no genes with > 5 counts for all the samples. I then filtered the RNA data to have genes that have > 5 counts for all samples and run again IMAGE, without success:
Your region file contains 68709 regions and 2 conditions across 4 files
Your exon file contains 21887 genes and 2 conditions across 6 files
Your fasta file is /home/dkv252/HOMER/data/genomes/mm9/mm9.fa
You are using 10 processors.
Starting the analysis
Setting up for parallizing motif search
Preparing input file for motif searching
Scanning for motifs
Running analysis in R
1. Reading data into R
2. Analyzing gene expression
3. Converting motif hits to matrix - Takes a while
4. Calculating motif activity - Full model stage
Error in { : task 10 failed - "cannot allocate vector of size 395.4 Mb"
Calls: cv.glmnet -> %dopar% ->
Execution halted
5. Performing ridge regressionError in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
Calls: load -> readChar
Execution halted
And this error I cannot figure it out. The data seems to be in the correct format, but the analysis fails at an earlier point than using all the samples. Do you have any suggestions of what could I try to make it work?
Best regards,
Jose
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/JesperGrud/IMAGE/issues/1?email_source=notifications&email_token=AA7FA55U32JSBZBA4PJMIKDPXUWMRA5CNFSM4HPZNOM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWMC57Q#issuecomment-496512766, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AA7FA54NTMYN65GIDFBSJNTPXUWMRANCNFSM4HPZNOMQ.
Dear Jesper,
This is the full command I am running:
IMAGE.pl -region data/short_gt30.bed -expression data/rnaseq.txt -fasta /home/dkv252/HOMER/data/genomes/mm9/mm9.fa -RNADesign 1 1 1 2 2 2 -EnhancerDesign 1 1 2 2 -p 5 -n test
I am currently running it in a 12 processors, 32 Gb ram computer. Someone tried before me to run the analysis in a high-performance computer and it seemed to run indefinitely. Maybe I could take your suggestion and try to run the analysis. I have given it a couple more tries without success...
Dear Jesper,
I am struggling to use the IMAGE method to process data from my lab. I am getting the error below:
It seems to be related to the glmnet package, but I am uncertain about where exactly and in which stage does the error occurs. This error is usually happening in the step 5. Performing ridge regression. My files have the same number of columns and information as the example datasets, so I believe that the error might happen because I might have different versions of the packages used when the algorithm was developed. Could you share an R session with the versions of the packages you use?
Best,
Jose