Closed nzllim closed 4 years ago
ppavlidis
commented
5 years ago I believe you have to add id 2721 to the list in -factors - in other words, "use this factor in the analysis" and also "subset on it". It is confusing so I'll fix that (just makes the logic a little more complicated), but it would be helpful if that could be tested to make sure I'm understanding the behavior.
nzllim
commented
5 years ago Including id 2721 to the -factors parameter does not change the outcome - Gemma still performs a 3-factor DE, without subsetting by id 2721.
nzllim
commented
5 years ago I suspected that the underlying method call might be relying on "old/outdated" differential expression settings/design when running. This is proven true, when I define the DE to be performed on only two factors without subsetting whatsoever. It still returns a 3-factor DE. Log included below.
Loading jars from /space/grp/lib/Gemma ========= Gemma CLI invocation of diffExAnalyze ============ Options: -u [REDACTED] -p XXXXXX -e GSE17478 -factors 2722,2723 Gemma version 1.25.15 built 20190613-1016
[...TRIM...]
ppavlidis
commented
4 years ago I tested, but please confirm this is fixed as required and close issue as appropriate
nzllim
commented
4 years ago Command works as expected now. Case closed.
As title: The diffExAnalyze ignores the -subset flag while performing differential expression. An example of the output log is attached below; the logs do not indicate anything amiss, except of failure to subset. (The experiment in Gemma had DE performed via the web-interface, which respects the -subset flag).
Case-In-Point: https://gemma.msl.ubc.ca/expressionExperiment/showExpressionExperiment.html?id=1350
LOG:
$GEMMACMD diffExAnalyze -u $GemmaUsername -p $GEMMAPASSWORD -e GSE17478 -subset 2721 -factors 2722,2723 Loading jars from /space/grp/lib/Gemma OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0 ========= Gemma CLI invocation of diffExAnalyze ============ Options: -u [REDACTED] -p XXXXXX -e GSE17478 -subset 2721 -factors 2722,2723 Gemma version 1.25.11 built 20190328-1657 [Gemma 2019-04-30 11:48:46,586] INFO [main] org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(172) | Loading properties file from class path resource [default.properties] [Gemma 2019-04-30 11:48:51,455] INFO [main] org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(172) | Loading properties file from class path resource [default.properties] [Gemma 2019-04-30 11:48:51,852] INFO [main] org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(172) | Loading properties file from class path resource [default.properties] [Gemma 2019-04-30 11:48:51,902] INFO [main] org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(172) | Loading properties file from class path resource [default.properties] [Gemma 2019-04-30 11:48:51,945] INFO [main] org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(172) | Loading properties file from class path resource [default.properties] [Gemma 2019-04-30 11:48:52,001] INFO [main] org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(172) | Loading properties file from class path resource [default.properties] [Gemma 2019-04-30 11:48:52,031] INFO [main] org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(172) | Loading properties file from class path resource [default.properties] [Gemma 2019-04-30 11:48:52,062] INFO [main] org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(172) | Loading properties file from class path resource [default.properties] [Gemma 2019-04-30 11:48:52,087] INFO [main] org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(172) | Loading properties file from class path resource [default.properties] [Gemma 2019-04-30 11:48:56,820] INFO [fmaOntology_load_thread_KkdvU] ubic.basecode.ontology.OntologyLoader.loadMemoryModel(219) | Connecting to http://purl.obolibrary.org/obo/fma.owl [Gemma 2019-04-30 11:48:57,759] INFO [main] ubic.gemma.persistence.util.SpringContextUtil.getApplicationContext(64) | Got context in 13154ms [Gemma 2019-04-30 11:48:58,002] INFO [main] ubic.gemma.core.util.AbstractSpringAwareCLI.authenticate(226) | Logged in as [REDACTED] [Gemma 2019-04-30 11:49:07,394] INFO [main] ubic.gemma.persistence.service.expression.bioAssayData.DesignElementDataVectorDaoImpl.thawRawAndProcessed(165) | Thawed 10000 vector-associated probes 3985 ms [Gemma 2019-04-30 11:49:11,821] INFO [main] ubic.gemma.persistence.service.expression.bioAssayData.DesignElementDataVectorDaoImpl.thawRawAndProcessed(165) | Thawed 20000 vector-associated probes 8412 ms [Gemma 2019-04-30 11:49:11,899] INFO [fmaOntology_load_thread_KkdvU] ubic.basecode.ontology.OntologyLoader.loadMemoryModel(268) | Load model: 15287ms [Gemma 2019-04-30 11:49:12,173] INFO [fmaOntology_load_thread_KkdvU] ubic.basecode.ontology.OntologyLoader.loadMemoryModel(309) | Caching ontology to disk: fmaOntology [Gemma 2019-04-30 11:49:12,173] WARN [fmaOntology_load_thread_KkdvU] ubic.basecode.ontology.OntologyLoader.loadMemoryModel(320) | Ontology cache directory required to save to disk: ontology.cache.dir [Gemma 2019-04-30 11:49:15,531] INFO [main] ubic.gemma.persistence.service.expression.bioAssayData.DesignElementDataVectorDaoImpl.thawRawAndProcessed(165) | Thawed 30000 vector-associated probes 12122 ms [Gemma 2019-04-30 11:49:18,726] INFO [fmaOntology_load_thread_KkdvU] ubic.basecode.ontology.search.OntologyIndexer.indexOntology(93) | A valid index for fmaOntology already exists, using [Gemma 2019-04-30 11:49:19,531] INFO [main] ubic.gemma.persistence.service.expression.bioAssayData.DesignElementDataVectorDaoImpl.thawRawAndProcessed(165) | Thawed 40000 vector-associated probes 16122 ms [Gemma 2019-04-30 11:49:21,264] INFO [main] ubic.gemma.persistence.service.expression.bioAssayData.DesignElementDataVectorDaoImpl.thawRawAndProcessed(173) | Thaw phase 4 45101 vector-associated probes thawed in 17855ms [Gemma 2019-04-30 11:49:25,248] INFO [main] ubic.gemma.core.datastructure.matrix.ExpressionDataMatrixColumnSort.getBaselineLevels(163) | Falling back on choosing baseline arbitrarily: FactorValue 20112: OrganismPart:heart [Gemma 2019-04-30 11:49:25,250] INFO [main] ubic.gemma.core.datastructure.matrix.ExpressionDataMatrixColumnSort.getBaselineLevels(128) | Baseline chosen: FactorValue 20110: Genotype:wild type genotype [Gemma 2019-04-30 11:49:25,251] INFO [main] ubic.gemma.core.datastructure.matrix.ExpressionDataMatrixColumnSort.getBaselineLevels(128) | Baseline chosen: FactorValue 20107: Treatment:reference substance role | phosphate buffered saline solution | [Gemma 2019-04-30 11:49:25,260] INFO [main] ubic.gemma.core.datastructure.matrix.ExpressionDataDoubleMatrixUtil.ensureLog2Scale(105) | Data is already on a log2 scale [Gemma 2019-04-30 11:49:25,827] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.makeDesignMatrix(1247) | ExperimentalFactor Id=2721 Name=OrganismPart [Gemma 2019-04-30 11:49:25,828] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.makeDesignMatrix(1247) | ExperimentalFactor Id=2722 Name=Genotype [Gemma 2019-04-30 11:49:25,828] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.makeDesignMatrix(1247) | ExperimentalFactor Id=2723 Name=Treatment [Gemma 2019-04-30 11:49:26,235] INFO [pool-25-thread-1] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer$2.run(1439) | Model fit data matrix 45101 x 24: 404ms [Gemma 2019-04-30 11:49:30,894] INFO [pool-25-thread-1] ubic.basecode.math.linearmodels.LeastSquaresFit.summarize(560) | Summarizing [Gemma 2019-04-30 11:49:33,153] INFO [pool-25-thread-1] ubic.basecode.math.linearmodels.LeastSquaresFit.summarize(569) | Summzarized 45101 results [Gemma 2019-04-30 11:49:33,185] INFO [pool-25-thread-1] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer$2.run(1459) | Model summarize/ANOVA: 6949ms [Gemma 2019-04-30 11:49:33,194] INFO [pool-25-thread-1] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer$2.run(1461) | Analysis phase done ... [Gemma 2019-04-30 11:49:34,259] WARN [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.doAnalysis(863) | ANOVA could not be done for ExperimentalFactor Id=2723 Name=Treatment on CompositeSequence Id=89775 Name=1424403_a_at, further warnings suppressed [Gemma 2019-04-30 11:49:34,513] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.doAnalysis(748) | Processed results for 15000 elements ... [Gemma 2019-04-30 11:49:34,910] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.doAnalysis(748) | Processed results for 30000 elements ... [Gemma 2019-04-30 11:49:35,218] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.doAnalysis(748) | Processed results for 45000 elements ... [Gemma 2019-04-30 11:49:35,221] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.doAnalysis(888) | 3 elements or results were not usable - model could not be fit, etc. [Gemma 2019-04-30 11:49:35,221] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.getRanksAndQvalues(1028) | 45101 pvalues for fact.2722 [Gemma 2019-04-30 11:49:35,438] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.getRanksAndQvalues(1028) | 45101 pvalues for fact.2721 [Gemma 2019-04-30 11:49:35,555] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.getRanksAndQvalues(1028) | 45098 pvalues for fact.2723 [Gemma 2019-04-30 11:49:39,650] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.makeResultSets(1267) | Processing 3 resultSets [Gemma 2019-04-30 11:49:39,939] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.makeResultSets(1294) | Finished with result set for fact.2722 [Gemma 2019-04-30 11:49:40,477] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.makeResultSets(1294) | Finished with result set for fact.2721 [Gemma 2019-04-30 11:49:40,633] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.makeResultSets(1294) | Finished with result set for fact.2723 [Gemma 2019-04-30 11:49:40,633] INFO [main] ubic.gemma.core.analysis.expression.diff.LinearModelAnalyzer.doAnalysis(897) | Analysis processing phase done ... [Gemma 2019-04-30 11:49:40,668] INFO [main] ubic.gemma.core.analysis.expression.diff.DifferentialExpressionAnalyzerServiceImpl.deleteOldAnalyses(365) | No differential expression analyses to remove for GSE17478 [Gemma 2019-04-30 11:49:41,206] INFO [main] ubic.gemma.core.analysis.expression.diff.DifferentialExpressionAnalyzerServiceImpl.persistAnalysis(240) | Saving results [Gemma 2019-04-30 11:51:42,567] INFO [main] ubic.gemma.persistence.service.expression.arrayDesign.ArrayDesignDaoImpl.thaw(766) | Start initialize composite sequences [Gemma 2019-04-30 11:51:45,042] INFO [main] ubic.gemma.persistence.service.expression.arrayDesign.ArrayDesignDaoImpl.thaw(771) | Thaw array design stage 2: 2475ms [Gemma 2019-04-30 11:51:47,430] INFO [main] ubic.gemma.persistence.service.expression.arrayDesign.ArrayDesignDaoImpl.thaw(815) | Thaw array design stage 3: 2387 [Gemma 2019-04-30 11:51:47,661] INFO [main] ubic.gemma.core.analysis.service.ArrayDesignAnnotationServiceImpl.readAnnotationFileAsString(159) | Reading annotations from: /space/gemmaData/microAnnots/GPL1261.an.txt.gz [Gemma 2019-04-30 11:51:48,229] INFO [main] ubic.gemma.core.analysis.service.ExpressionDataFileServiceImpl.writeDiffExArchiveFile(309) | Creating differential expression analysis archive file: 1350_GSE17478_diffExpAnalysis_109426.zip [Gemma 2019-04-30 11:51:52,548] INFO [main] org.openjena.atlas.logging.Log.info(101) | Thaw resultSet 484715 took 4112ms [Gemma 2019-04-30 11:51:52,549] INFO [main] org.openjena.atlas.logging.Log.info(101) | Thawed 1/3 resultSets [Gemma 2019-04-30 11:51:56,295] INFO [main] org.openjena.atlas.logging.Log.info(101) | Thaw resultSet 484714 took 3746ms [Gemma 2019-04-30 11:51:56,296] INFO [main] org.openjena.atlas.logging.Log.info(101) | Thawed 2/3 resultSets [Gemma 2019-04-30 11:52:00,233] INFO [main] org.openjena.atlas.logging.Log.info(101) | Thaw resultSet 484713 took 3936ms [Gemma 2019-04-30 11:52:00,234] INFO [main] org.openjena.atlas.logging.Log.info(101) | Thawed 3/3 resultSets [Gemma 2019-04-30 11:52:01,910] INFO [main] ubic.gemma.persistence.service.analysis.expression.diff.DifferentialExpressionAnalysisDaoImpl.thaw(363) | Thaw: 1507ms [Gemma 2019-04-30 11:52:06,367] INFO [main] ubic.gemma.persistence.service.analysis.expression.diff.DifferentialExpressionAnalysisDaoImpl.thaw(363) | Thaw: 1634ms [Gemma 2019-04-30 11:52:09,756] INFO [main] ubic.gemma.core.analysis.expression.diff.DifferentialExpressionAnalyzerServiceImpl.persistAnalysis(268) | Save results: 149087ms [Gemma 2019-04-30 11:52:09,757] INFO [main] ubic.gemma.core.util.AbstractCLI.summarizeProcessing(515) |
Successfully processed 1 objects: Success ExpressionExperiment Id=1350 Name=Particulate Matter effect on Mouse Model of Cardiac Failure: Lung and Heart Left Ventricle (GSE17478)