epic: Scale Testing (unidimensional)

[dgarros]

Component

No response

Describe the Feature Request

As we get closer to the beta, we need to start testing and validating how Infrahub is performing at Scale and we need to start publishing guidelines regarding what is "supported" As much as possible the tests needs to be automated because as the system evolves we'll need to run them on a regular basis to validate how the latest features are impacting the performance.

Scale testing is a very complex problems as we need to consider many elements (dimension) and how they influence each other. At first, we need to focus on unidimensional testing where we scale only one dimension at a time, in a second phase we'll build more complex scenarios where we'll scale multiple dimension at once (multidimensional)

Below is a non-exhaustive list of dimension that we need to consider to test infrahub at Scale:

• Number of objects
• Number of Attributes
• Number of Relationships (schema)
• Number of Revision of an attribute
• Number of Revision of a relationship
• Number of Element in a Diff
• Number of Choice
• Number of Branch
• Number of Artifacts
• Number of Concurrent request to the API (Read)
• Number of Concurrent request to the API (Write)

For each dimension we need to assess how the system is behaving at different scale level and specifically we need to validate the :

• Performance Impact
• Usability Impact (mainly in the frontend)

Testing protocol

• Choose 3 to 5 scale level for the specific dimension we are trying to test
• For each level :
  • Load the system with the test data
  • Measure the response time of various actions, in most cases it will be interesting to try all CRUD operations and to record the response time of the application
  • Check the frontend to validate its behavior, take screenshot as needed

Metrics to monitor during each test

• CPU & Memory of the server
• CPU & Memory of the database
• Size of the database
• Number of nodes and edges in the database

Testing Framework

Currently we are using Locust for some basic tests but if a better solution exist we should consider it.

Describe the Use Case

As we get closer to the beta, we need to start testing and validating how Infrahub is performing at Scale and we need to start publishing guidelines regarding what is "supported"

Additional Information

No response

[petercrocker]

Not part of scale testing, but other performance related tests to consider in the future:

• Network usage to upload an entire network from another tool
• Time to suck in netbox —> Infrahub
• Latency testing (Something like kffl/speedbump: TCP proxy for simulating variable, yet predictable network latency :globe_with_meridians::hourglass_flowing_sand:)