search

opencobra / cobratoolbox

The COnstraint-Based Reconstruction and Analysis Toolbox. Documentation:
https://opencobra.github.io/cobratoolbox
Other
246 stars 308 forks source link

long time Microbiome modeling #1540

Open PejmanS opened 4 years ago

PejmanS commented 4 years ago

Hi I have ran a microbiome modeling data with 95 samples and about 200 taxons. But it's about 3weeks that is performing FVA with CPLEX 128 (fvatype=1) on 1 sample without any changing. Matlab shows Busy status and CPU is working too. I checked fvaCt and it shows structures for just 18 samples. I deleted that sample and ran again by another Matlab env but it happened again on another sample. Is it OK? Should this analyse takes long about 3weeks on a sample for FVA? @almut-heinken @rmtfleming

I hereby confirm that I have:

(Note: You may replace [ ] with [X] to check the box)

almut-heinken commented 4 years ago

Hi @PejmanS Are you using parallel computing and/or high performance computing? Did you normalize abundances so they sum up to 1 per samples? Best, Almut

PejmanS commented 4 years ago

Yes i'm using parallel pools. But as you know it doesnt work on PART3 and pools get shut down. I'm usnig HPC but now it works by just 2% of cpu performance for these analysis. Abundances for each sample sum up to 1.

On Sat, Jan 11, 2020, 9:26 PM almut-heinken notifications@github.com wrote:

Hi @PejmanS https://github.com/PejmanS Are you using parallel computing and/or high performance computing? Did you normalize abundances so they sum up to 1 per samples? Best, Almut

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opencobra/cobratoolbox/issues/1540?email_source=notifications&email_token=AKYJQYSOC6L2QBQQWL5GH4DQ5IB35A5CNFSM4KFTDUIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIWHLYA#issuecomment-573339104, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKYJQYRKBSJDCAHZ3B5I4FLQ5IB35ANCNFSM4KFTDUIA .

almut-heinken commented 4 years ago

Hi @PejmanS I am not sure what the problem is then. It should not take this long. Could you check if the models are in principle feasible? Try maximizing community biomass on the Average European diet. The pipeline constrains the community biomass between 0.4 and 1 so if it cannot reach 0.4, the model will be infeasible.

PejmanS commented 4 years ago

I know that in inFesMat there are some models which are infeasible. Is it right? But this model there is not in this mat. Also it should be there a simRes file but it is not there! I set rDiet=1, so does it help for getting enough biomass?

On Sat, Jan 11, 2020, 9:39 PM almut-heinken notifications@github.com wrote:

Hi @PejmanS https://github.com/PejmanS I am not sure what the problem is then. It should not take this long. Could you check if the models are in principle feasible? Try maximizing community biomass on the Average European diet. The pipeline constrains the community biomass between 0.4 and 1 so if it cannot reach 0.4, the model will be infeasible.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opencobra/cobratoolbox/issues/1540?email_source=notifications&email_token=AKYJQYRUJZRNPP7TZWTB4ZLQ5IDMRA5CNFSM4KFTDUIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIWHVTI#issuecomment-573340365, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKYJQYVHMQOJLJEZITVNLVLQ5IDMRANCNFSM4KFTDUIA .

almut-heinken commented 4 years ago

Hi, the simRes file is only created when the pipeline is finished. May you test if the models are feasible on the Average European diet without going through the pipeline? Best Almut

PejmanS commented 4 years ago

@almut-heinken Should i use useDiet to set constraints on model? the europe average diet file hasn't ub and lb and useDiet function need them.

almut-heinken commented 4 years ago

@PejmanS the code is as follows: adaptedDiet = adaptVMHDietToAGORA('AverageEuropeanDiet', 'Microbiota'); model=useDiet(model,adaptedDiet);

PejmanS commented 4 years ago

@almut-heinken I think there is a problem in toolbox. Wehn diet gets adopted, the format of reactions is Diet_EX_xxx and it is not compatible with model reactions.

Warning: Reaction Diet_EX_etoh[d] not in model

In changeRxnBounds (line 55)

almut-heinken commented 4 years ago

@PejmanS No, these IDs get automatically adapted in the pipeline, so that also needs to be done when growing the models outside the pipeline. I did not mention that, sorry. The code to do that is in https://github.com/opencobra/cobratoolbox/blob/master/src/analysis/multiSpecies/microbiomeModelingToolbox/mgPipe/microbiotaModelSimulator.m line 87-98.

PejmanS commented 4 years ago

Hi @almut-heinken Which one should be Obj for FBA? EX_microbeBiomass[fe] or communityBiomass?

almut-heinken commented 4 years ago

Either one works, it makes no difference.

PejmanS commented 4 years ago

Hi dear @almut-heinken I tested them. some of models show obj=0 and some show obj=1. 2 of models that stopped analyze in pipeline, in FBA had delay too. one time they didn't complete FBA and i stopped analyze, but when i ran again FBA for those models, it worked and finally obj was 0 for them. and i recently have another error for fastFVA in pipeline;

Error using cd Arguments must contain a character vector.

I ran generateMexFastFva() and then pipeline again. But it happened again!

almut-heinken commented 4 years ago

Hi @PejmanS I opened a pull request https://github.com/opencobra/cobratoolbox/pull/1547. Can you check again if the models are not producing biomass after pulling this branch? I am not having any problems with fastFVA.

PejmanS commented 4 years ago

Some Biomasses have changed from 0 to 1. But Biomass is logically yet, 1 or 0.

almut-heinken commented 4 years ago

What does that mean, that some models are still not growing? I would need to have a look at them to figure out why.

PejmanS commented 4 years ago

It does FVA in pipeline for models, but i think it has problem with loading some models. It gets freezing or hanging on reading models not FVA.

almut-heinken commented 4 years ago

Hi, that seems to be a problem of Matlab in general, not mgPipe. Can the models grow now?

PejmanS commented 4 years ago

Hi @almut-heinken Finally the time of analyse has been reduced and FVA is done in pipeline. But about 20 samples of 92 accounted as alive and others as dead! At the last code for mgSimResCollect function, I;ve got an error:

Conversion to cell from double is not possible.

Error in mgSimResCollect (line 88) fSp(:, k - 1) = sp;

Error in mgPipe (line 123) [Fsp,Y]= mgSimResCollect(resPath,ID,sampName,rDiet,0,patNumb,indInfoFilePath,fvaCt,figForm);

Error in initMgPipe (line 165) mgPipe

Thanks for your support.

PejmanS commented 4 years ago

I beleive that each sample should be considered as a live model, because the relative abundances are there and worked environmentally.

almut-heinken commented 4 years ago

Hi @PejmanS , yes, all models should be able to grow. Unfortunately, without seeing the models that cannot grow myself, I cannot say why that is happening.

PejmanS commented 4 years ago

What do you believe about last error? How could i send you models or taxonomic abundances file?

On Mon, Feb 17, 2020, 12:36 AM almut-heinken notifications@github.com wrote:

Hi @PejmanS https://github.com/PejmanS , yes, all models should be able to grow. Unfortunately, without seeing the models that cannot grow myself, I cannot say why that is happening.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opencobra/cobratoolbox/issues/1540?email_source=notifications&email_token=AKYJQYVPASLHP5F3MSBOPOTRDGTGBA5CNFSM4KFTDUIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4SFWQ#issuecomment-586752730, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKYJQYUGHIW46IW4T2XYD33RDGTGBANCNFSM4KFTDUIA .

almut-heinken commented 4 years ago

You could send one example to me via Google Drive. What does the content of intRes look like?

PejmanS commented 4 years ago

Hi dear @almut-heinken After the changes on modeling functions, I hadn't infeasible model. But unfortunately i have this error again.

inFesMat =

0×0 empty cell array

Conversion to cell from double is not possible.

Error in mgSimResCollect (line 88) fSp(:, k - 1) = sp;

Error in mgPipe (line 123) [Fsp,Y]= mgSimResCollect(resPath,ID,sampName,rDiet,0,patNumb,indInfoFilePath,fvaCt,figForm);

Error in initMgPipe (line 146) mgPipe

Error in startMgPipe (line 46) initMgPipe(modPath, toolboxPath, resPath, dietFilePath, abunFilePath,indInfoFilePath, objre, figForm, numWorkers, autoFix, compMod, rDiet,extSolve,fvaType,autorun);

almut-heinken commented 4 years ago

Hello, what does the content of intRes and inFeasMat look like? Best, Almut

On Tue, May 5, 2020 at 10:20 AM PejmanS notifications@github.com wrote:

Hi dear @almut-heinken https://github.com/almut-heinken After the changes on modeling functions, I hadn't infeasible model. But unfortunately i have this error again.

inFesMat =

0×0 empty cell array

Conversion to cell from double is not possible.

Error in mgSimResCollect (line 88) fSp(:, k - 1) = sp;

Error in mgPipe (line 123) [Fsp,Y]= mgSimResCollect(resPath,ID,sampName,rDiet,0,patNumb,indInfoFilePath,fvaCt,figForm);

Error in initMgPipe (line 146) mgPipe

Error in startMgPipe (line 46) initMgPipe(modPath, toolboxPath, resPath, dietFilePath, abunFilePath,indInfoFilePath, objre, figForm, numWorkers, autoFix, compMod, rDiet,extSolve,fvaType,autorun);

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opencobra/cobratoolbox/issues/1540#issuecomment-623948054, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACSOYHP5MDA4DKNT7D2RT3TRP7K5DANCNFSM4KFTDUIA .

PejmanS commented 4 years ago

@almut-heinken inFeas is free. But i sent files on your email.

almut-heinken commented 4 years ago

Hi, it seems there was an issue with not updating a variable. I will submit a pull request. I managed to extract your simulation results. Best, Almut

On Sun, May 10, 2020 at 7:35 PM PejmanS notifications@github.com wrote:

inFeas is free. But i sent files on your email.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opencobra/cobratoolbox/issues/1540#issuecomment-626370008, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACSOYHMJBTVA2FGKAJ6WWX3RQ3XXHANCNFSM4KFTDUIA .