search

StochasticNumerics / mimclib

A software library for UQ methods
GNU General Public License v2.0
6 stars 8 forks source link

Force MIMCLIB to print out intermediate results #85

Closed litvinen closed 7 years ago

litvinen commented 7 years ago

Very serious issue for me.

I have 5 levels of grids. The whole computations take 2 days. After 24 hours of running command "top" shows that 90% of RAM memory is used (100GB). Each process started with 1MB and now allocated already 8GB. But there is no results in mysql. Probably my TOL is too small and MIMCLIB requires finer and finer grids. Very soon it will be not enough memory and processes will be killed by the operation system. And I will not see any intermediate results in the table. So, I waited 48 hours and learned nothing about my problem. Therefore it would be nice to print out intermediate results (even if computations are not ready yet).

Virtakuono commented 7 years ago

How is this different from https://github.com/StochasticNumerics/mimclib/issues/59 ?

litvinen commented 7 years ago

Yes, it is different. Assume I compute for TOL1< ...<TOL10. Usually computations for TOL1,...,TOL3 are expensive and for all others cheap. I can interrupt in between and see results for TOL4...TOL10. If I am happy with results I compute for TOL1...TOL3 But computations for, say, TOL3 may take 2 days and if I interrupt after 36 hours I will see nothing. BUT, all computations on level 1-5 are done, this is levels 6-7 which take so long.

litvinen commented 7 years ago

Abdul! Think please to print bias, stat. error and other useful info even if not all levels are computed.

Virtakuono commented 7 years ago

So you say that TOL4...TOL10 are computed but not in the database? In my opinion this should not be possible, see https://github.com/StochasticNumerics/mimclib/issues/79#issuecomment-254760876 and the discussion around it.

litvinen commented 7 years ago

No, TOL4-TOL10 are computed and are in database, but show "pre-asymptotic regime" when MC is better as MLMC. So I run MLMC code again with TOL1.. TOL3 with hope to see that MLMC is better as MC. BUT(!) if I am not so lucky with the choice of TOL3, the computations can take 2-3-4-5-6 days and at the end will crash because not enough memory. So! I need intermediate results for TOL3 to understand my mistake in input randomness.

Virtakuono commented 7 years ago

How many threads are you running?

haji-ali commented 7 years ago

I think I understand what you want but it is not possible within the current framework. If you want to output half baked results then you can do so in your SampleLvl function by inspecting the expectation / variance estimates (run.last_itr.calcDeltaEl() and run.last_itr.calcDeltaVl())

litvinen commented 7 years ago

Thanks,