a-b-street / abstreet

Transportation planning and traffic simulation software for creating cities friendlier to walking, biking, and public transit
https://a-b-street.github.io/docs/
Apache License 2.0
7.76k stars 345 forks source link

Consider multithreading #657

Closed vks closed 2 years ago

vks commented 3 years ago

As far as I can tell, abstreet currently only uses one core. Parallelizing agents would be non-trivial, and probably not worth it. However, for example the prebaking should be easy to parallelize, so that several maps can be simulated at the same time.

dabreegster commented 3 years ago

There are a few other ideas / considerations...

As far as I can tell, abstreet currently only uses one core.

There's a few places we use timer.parallelize, which uses scoped_threadpool underneath. But otherwise true.

Parallelizing agents would be non-trivial, and probably not worth it.

Yuuup. It's a cool algorithmic problem, though. I've come across some academic papers talking about running discrete-event traffic simulations in parallel. If you can partition the map carefully and have long roads along the boundary of two different execution contexts, then both machines/threads can simulate this overlap road. The amount of simulation time they could drift apart is bounded by the time it takes an agent to cross that road. Fun to think about, highly unlikely we'll ever try something like this.

prebaking should be easy to parallelize

That'd be a good one to include. The barrier here IMO is logging / UI. For prebaking it's maybe not as necessary to easily see the logs per map, but I'd like to figure out a general strategy for doing this. Prefixing each line with [map-name] might do the trick here. It'd be really nice to have some kind of floating progress bar showing the number of maps finished, with logs beneath it, but that means finding a console UI library.

dabreegster commented 3 years ago

A similar case to prebaking is importing maps. We're up to about 100 different maps, most of which can be parallelized. I'm in the process of moving this work to happen on multiple machines: https://github.com/a-b-street/abstreet/tree/master/cloud But the logging story isn't really solved. Ideally it'd be easy to both see an overall status of how many jobs are done, and click through to see logs for a single city.

262 is also related

vks commented 3 years ago

That'd be a good one to include. The barrier here IMO is logging / UI. For prebaking it's maybe not as necessary to easily see the logs per map, but I'd like to figure out a general strategy for doing this. Prefixing each line with [map-name] might do the trick here. It'd be really nice to have some kind of floating progress bar showing the number of maps finished, with logs beneath it, but that means finding a console UI library.

We could do something similar to Cargo here.

dabreegster commented 2 years ago

No plans to do anything here, so closing. FWIW, my workflow for importing maps now uses https://github.com/Nukesor/pueue to crank out some parallelism