[Automate Engine] Looking up objects in the database is expensive

Fryguy commented 4 years ago

When executing an automation request there are a TON of database queries, and most of them are duplicated. Additionally, since the content of the automate domain doesn't change frequently during execution, we should be able to do some simple caching to eliminate a lot of the performance penalty.

Some ideas...

wrapping an automation request in ActiveRecord::QueryCache.cache
general caching of the entire domain structure, and busting the cache when an domain is updated and saved.
https://github.com/ManageIQ/manageiq-automation_engine/issues/409
optimizing the most common lookups

I'm sure there are other ideas, but it feels like there are likely some really quick wins in here. cc @tinaafitz

Fryguy commented 4 years ago

@kbrock @NickLaMuro This is the higher order epic for automate's performance.

kbrock commented 4 years ago

One time I tested wrapping all MiqQueue#dispatch and I think it passed specs.

People were worried that the refresh would break. But this is how all our ui runs, not sure why the back end would be that different. (Also refresh used to bust the cache every query anyhow)

ManageIQ / manageiq-automation_engine

[Automate Engine] Looking up objects in the database is expensive #410