search

iut-ibk / DynaMind-ToolBox

DynaMind-ToolBox
www.dance4water.org
GNU General Public License v2.0
9 stars 6 forks source link

Dynamic Innsbruck Simulation runs. Test for DB #62

Closed christianurich closed 11 years ago

christianurich commented 11 years ago

next check on Monday

christianurich commented 11 years ago

first test just in cache ... successful test with db nodecache 100000 attribtecache 100000 after 9 hours still working on color_buildings (Attribute Calculator) without cache the module takes not even a second ... failed

zacharias2k commented 11 years ago

"Just in cache" - infinite cache or NO_DB_SYNC? Possible to resolve the state of worker & pool?

christianurich commented 11 years ago

The first run was in cache but with the flag set. I'll try to set up a couple of bigger test cases to figure out where the issues are.

christianurich commented 11 years ago

The first run was in cache but with the flag set. I'll try to set up a couple of bigger test cases to figure out where the issues are.

zacharias2k commented 11 years ago

You will always run in cache, no matter what you set. The difference is when it starts to write into the db-file, see docu. Try setting cache to 0 (means infinite).

christianurich commented 11 years ago

My post was probably confusing

so again:

Big simulation; infinite cache with DB -> works Big simulation; cache size 100 000 fails (see second post)

christianurich commented 11 years ago

I put a bigger simulation in the dropbox please ask @abroxos for access. You probably have to update the files ImportGDAL points to and close the links that are missing. If import gdal doesn't find the file the ports are not created and therefore the links are deleted. It works with infinite cache and db, it's extremely slow with --nodecache 100000 --attributecache 100000, not sure if it ever gets finished. But could also be my setup.

abroxos commented 11 years ago

Which modules (submodules) have do be enabled ?

christianurich commented 11 years ago

Cityblocks, PowerVIBe and maybe Sewer

christianurich commented 11 years ago

Could you reproduce the problem?

zacharias2k commented 11 years ago

Still on it, last msg of yesterday: https://github.com/iut-ibk/DynaMind-ToolBox/issues/37

zacharias2k commented 11 years ago

I tried the model with the standalone. Besides a general speed up, it is really taking very very long. I didn't manage to finish it within about 2 hours, so i decided to make some improvements in the modules. https://github.com/iut-ibk/DynaMind-BasicModules/commit/f7f71ff433468f7e338af2f0a4277f32dee23175 https://github.com/iut-ibk/DynaMind-Extensions/commit/38f7bd7bba348e2e50606ba7f6d6819523826b0c But it's just a minior improvement. With every iteration, a new successor state is created, copying all components from view, including it's attributes (a quick look revealed 60 attributes) which are copied one by one - as a consequence of our separated attribute concept - which is fairly good for our purpose. I will think about possibilities to make bulge-copies etc. But i'm not shure, by now, how to handle this without creating a new site/branch. Maybe we should move it to "after simenv"....

christianurich commented 11 years ago

Why are the Attributes and the components copied. I thought just when a component is touched the component is copied (not the Attributes) and when an Attribute is touched the Attribute?

We definitely have to solve this problem. All simulations that I have done so far look like this and therefore have lots of states. The test simulation is just a very small version of the monster I'm running here.

Without this running the db is useless for me! We should solve this issue BEFOR we start with simenv so we still can change something at the design if necessary. @abroxos

zacharias2k commented 11 years ago

it finished without an error.

INFO Mi 10. Apr 15:49:20 2013| >>>> config: INFO Mi 10. Apr 15:49:20 2013| node cache size: 10000000 INFO Mi 10. Apr 15:49:20 2013| attribute cache size: 10000000 INFO Mi 10. Apr 15:49:20 2013| sql query stack size: 100 INFO Mi 10. Apr 15:49:20 2013| sql write block size: 50 INFO Mi 10. Apr 16:08:17 2013| End Simulation INFO Mi 10. Apr 16:08:17 2013| >>>> finished simulation at an average of 1126339 +- 0 ms

Because the attribute calculator is touching attributes -> they get copied including their component. Introducing a successor at component level may lead to serious overload of the, already, huge component class and of course explode the API.

But as you can see, minior changes in your module code improved performance drastically, my simulation finished in 20 minutes, while it took over 2 hours without beeing finished before. Yes, WE got to improve this. I'll improve the core as best as i can and you'll have to improve the modules, you are using. I know was working to now. But now we got a different core, it is not designed to beat the performance of before - remember the design goals. And we have to improve imperformant code, as bottlenecks have changed from a massive map-performance bottlenecks to a hard-drive, thread syncronize AND cache bottleneck. That includes improving module code. We want performance, we need performant code.

Simenv doesn't touch the core classes, where this issue originates.

zacharias2k commented 11 years ago

https://github.com/iut-ibk/DynaMind/commit/1c71dd76b075560fbb651c39d06c9da301123437

christianurich commented 11 years ago

How does the state system work? If I create a new state and I touch a component for example getComponet what is happening with the attributes? When is the state of the attribute created?

Please make a proper performance analysis with this test file. I was using smaller node and attribute caches 100 000 and it didn't finish at all.

The state system is essential for my simulations and I create lots of them. Is this problem. If so how can we solve it?

When do you think can I use the dB for my simulations? Do I have to change code?

Until I have seen a roadmap of simenv I simply don't known what your plans are.

zacharias2k commented 11 years ago

As said, when touching a component via getComponent, it's copied, with all it's attributes (successors are on the level of system). There is no successor attribute (no component-level successor state).

I'll try smaller caches - but that should only decrease performance, not causing a crash.

The big issues are:

Solutions:

besides a lot of improvement can be done in the module code - as mentioned before.

(n->getX(), n->getY()), instead use n->get(double* v)

or

DM::Node n; n.setX(basenode->getX()) n.setY(basenode->getY()) n.setZ(basenode->getZ()) (seen in spatial linking, worst ever!)

  • do not use core containers as temporary variables (you can, it works, no doubt - but if performance is crucial, don't do it)

and so on, i will write a guideline for performance in the wiki soon.

I will do some profiling, maybe i can remove some bottlenecks. Meanwhile, use a bigger cache, and follow the suggestions above. And remember: successors are just one type of performance killers! You may want to take a look at https://github.com/iut-ibk/DynaMind-ToolBox/issues/58

You can use DB whenever you want, it's up to you.

Please don't get me wrong, but simenv is not going to be finished in the next weeks, don't look forward using it in your current simulations (AFAIK that was already communicated as you where in austria by michael and me).

Simenv does not change any behaviour of the core. It's the groundwork for CRC, a general cleanup/disentangling of module and simulation class, leading to an adaption of GUI and group classes. It should pave the path to a clean, stable and performant module processing with groundwork to the requested sandboxing.

christianurich commented 11 years ago

Thanks now I get it, or most of it.

A wiki entry of how to use the new core would be awesome. I would like to start integrating the new stuff but I'm a bit lost.

Moved DB discussion to https://github.com/iut-ibk/DynaMind-ToolBox/issues/76

Simenv:

I get the idea of what is planned for cleaning up the core (and it for sure needs to be done) but before starting with the major cleaning work we should make a roadmap -> have a discussion -> and agree on it. Could be a great opportunity to get rid of some design problems but therefore we need a more detailed discussion.

zacharias2k commented 11 years ago

I'm always in touch with @abroxos about it. As no API changes where made, don't worry. The exact result of the process strongly depends on the performance and the features of Qt. If there are some design issues by now, just tell me.

christianurich commented 11 years ago

I just made a test with a bigger cache and it is now 20 slower with --nodecache 1.000.000 --attributecache 1.000.000

The bottleneck seams to be after the first state is created

AttributeCalculator {11aeca5a-1803-4d41-935e-126e26b743e0} time 4.06 sec (cache)

AttributeCalculator {61afe637-5512-454a-a7b2-b182caba623a} time 617.243 sec (db)

Any idea why the module takes 150 times longer? The Attribute calculator is accessing approximately 12.000 components and 6.000 attributes goes from footprint to address and gets the number of stories.

Please do a proper performance testing! We need to develop a feeling of how the DB is reacting and where the problems are so we can resolve the bottle necks!

Also test small cache sizes to simulate big simulations and please test the stage system.

At the current stage I can't use the DB

christianurich commented 11 years ago

p.s. I used the latest updates

zacharias2k commented 11 years ago

Which attribute calculator is this (attribute name)?

christianurich commented 11 years ago

It's the one in the first loop group "color buildings", the first attribute calculator.

p.s.

The changes seem to run bug free for the big simulation

zacharias2k commented 11 years ago

And this increase is since my latest core update?

----- Reply message ----- Von: "Christian Urich" notifications@github.com An: "iut-ibk/DynaMind-ToolBox" DynaMind-ToolBox@noreply.github.com Cc: "zacharias2k" m.sengthaler@gmail.com Betreff: [DynaMind-ToolBox] Dynamic Innsbruck Simulation runs. Test for DB (#62) Datum: Do., Apr. 11, 2013 17:08 It's the one in the first loop group "color buildings", the first attribute calculator.

p.s.

The changes seem to run bug free for the big simulation

— Reply to this email directly or view it on GitHub.

christianurich commented 11 years ago

I only tried smaller caches before the update so I don't know.

I'm just running 100.000/100.000 and it is seems to be slow, but I don't have the numbers yet. It tries to import a shapefile since a couple of minutes.

christianurich commented 11 years ago

4th ImportGDAL after 20 mins still not finished

zacharias2k commented 11 years ago

the first attribute calculator in color_buildings in the gui (infinite cache):

INFO Do 11. Apr 18:03:44 2013| Success AttributeCalculator {dcd28a30-d151-4b10-9432-456de49b86e3} Counter 0 time 103.04

100k/100k is really slow as gdal cant import into cache (https://github.com/iut-ibk/DynaMind-ToolBox/issues/58) 1M/1M:

INFO Do 11. Apr 18:24:27 2013| Success AttributeCalculator {20a2be53-4947-46eb-90ae-83f3f04078ca} Counter 0 time 24.354

christianurich commented 11 years ago

I close this issue and move the discussions:

At the current stage Innsbruck will not run. Before it works we need to reduce the load on the DB.