In QUADS 2.1.0 we moved to generating and hosting the dynamic wiki, inventory and pages directly from flask and our APIv3 via a separate gunicorn process e.g. quads-web instead of generating the same data in markdown as a separate process and then pushing that over a Python XMLRPC interface into Wordpress via an API.
Problem Details
We are seeing slow page render times for /assignments and /inventory when a large amount of hosts (1000+ or more) are actively scheduled. This has evaded us in our testing in smaller environments due to not running this at such a large scale in the Scale Lab (1200+ hosts, actively scheduled). This is not evident in smaller environments.
Page Load Observations
Page
Avg Load Time
Nginx Payload Render
Page Size
Scheduled Hosts
Total Hosts
Active Env
All Hosts Render per Host
Per Scheduled Host Render
/assignments
50 sec
1-3 sec
2584 KB
1064
1172
37
40ms
44ms
/inventory
35 sec
1-3 sec
829 KB
1064
1172
37
27ms
30ms
Page Render Mechanics
For /assignments each host across each active environment has to be queried for its schedule start, schedule end which is done sequentially for each active environment. Across a lot of active hosts with schedules this adds up. There is other date calculation and content structuring but this is relatively fast. Nginx also delivers great performance and most of the time is simply waiting for the payload to generate so it can be rendered.
There is other date calculation and content structuring but this is relatively fast. Nginx also delivers great performance and most of the time is simply waiting for the payload to generate so it can be rendered.
For /inventory each host has its cloud owner and active current cloud environment queried sequentially and iterated by rack.
Use Ajax / jquery 12 to only render the summary parts of /assignments and only when someone drills into a specific environment is that part of the page loaded or otherwise use intelligent / background loading on-demand.
Look into asyncio for potentially expensive iterative loops
Look into using asyncio in the jinja2 content rendering calls and methods where possible, this is done in many other parts of the code-base with great results.
Other Rendering Optimizations
Potentially look at rendering /inventory first by cloud membership (--cloud-only) to potentially lessen amount of calls, then sort by rack to speed up render times for jinja2 content generation.
Add additional helper unification methods for src/quads/web/app.py to reduce the amount of Rest API calls required in pulling in all the data needed for page render payloads.
Summary
In QUADS 2.1.0 we moved to generating and hosting the dynamic wiki, inventory and pages directly from flask and our APIv3 via a separate gunicorn process e.g.
quads-webinstead of generating the same data in markdown as a separate process and then pushing that over a Python XMLRPC interface into Wordpress via an API.Problem Details
We are seeing slow page render times for
/assignmentsand/inventorywhen a large amount of hosts (1000+ or more) are actively scheduled. This has evaded us in our testing in smaller environments due to not running this at such a large scale in the Scale Lab (1200+ hosts, actively scheduled). This is not evident in smaller environments.Page Load Observations
/assignments/inventoryPage Render Mechanics
For
/assignmentseach host across each active environment has to be queried for its schedule start, schedule end which is done sequentially for each active environment. Across a lot of active hosts with schedules this adds up. There is otherdatecalculation and content structuring but this is relatively fast. Nginx also delivers great performance and most of the time is simply waiting for the payload to generate so it can be rendered.https://github.com/redhat-performance/quads/blob/latest/src/quads/web/app.py#L163 https://github.com/redhat-performance/quads/blob/latest/src/quads/web/controller/CloudOperations.py#L95 https://github.com/redhat-performance/quads/blob/latest/src/quads/web/controller/CloudOperations.py#L44 https://github.com/redhat-performance/quads/blob/latest/src/quads/web/controller/CloudOperations.py#L109
There is other
datecalculation and content structuring but this is relatively fast. Nginx also delivers great performance and most of the time is simply waiting for the payload to generate so it can be rendered.For
/inventoryeach host has its cloud owner and active current cloud environment queried sequentially and iterated by rack.https://github.com/redhat-performance/quads/blob/latest/src/quads/web/app.py#L256 https://github.com/redhat-performance/quads/blob/latest/src/quads/web/app.py#L279
Suggestions for Improvement
Ajax / JQuery / Intelligent Rendering
Use Ajax / jquery 1 2 to only render the summary parts of
/assignmentsand only when someone drills into a specific environment is that part of the page loaded or otherwise use intelligent / background loading on-demand.Look into asyncio for potentially expensive iterative loops
Look into using asyncio in the jinja2 content rendering calls and methods where possible, this is done in many other parts of the code-base with great results.
Other Rendering Optimizations
Potentially look at rendering
/inventoryfirst by cloud membership (--cloud-only) to potentially lessen amount of calls, then sort by rack to speed up render times for jinja2 content generation.src/quads/web/app.pyto reduce the amount of Rest API calls required in pulling in all the data needed for page render payloads.