search

camsas / firmament

The Firmament cluster scheduling platform
Apache License 2.0
412 stars 77 forks source link

Replace slow task assignment extraction code #18

Open ms705 opened 9 years ago

ms705 commented 9 years ago

The naive task mapping extraction code in quincy_dispatcher.cc is slowing things down at scale (most notably when running the simulator on the full Google trace) since it takes a few seconds to extract the task mappings.

We have an algorithmically superior implementation in Flowlessly, which we should back-port into the Firmament code base, so that all flow solvers benefit from it.

ms705 commented 8 years ago

@ICGog is this actually still an issue? I know that the dispatcher code changed significantly since, but it looks like GetMappings() in SolverDispatcher didn't change substantially. Did your recent scalability and performance improvement work would have hit the inefficient assignment extraction as an issue?

ICGog commented 8 years ago

I didn't hit this issue in my scalability tests because I was using Flowlessly which can use its fast assignment algorithm to just return task mappings. However, this is likely to still be a problem if we use other solvers.