[SCD] Create a configurable load test

[LeslieW]

Is your feature request related to a problem? Please describe. Load tests are crucial for ensuring systems operate well at scale. An individual USS setting up load testing infrastructure will always be insufficient -- they do not have the appropriate means to load test with other USSs in their ecosystem due to insufficient information about another USSs operations or underlying implementation. InterUSS Automated Testing is in a position to help USSs prepare for prod operations at scale by providing a configurable load test with the test suite.

Describe the solution you'd like There are likely many approaches to creating and conducting a load test. Here's what I would love to see at a high level:

• Each USS has the ability to input the scale they would like tested, perhaps something along the lines of # of flights per minute per 10sqkm, with a default of 1 if not proactively configured by a USS.
• The load test generates flights based on the inputs from each USS at roughly the same time.
• The load test has some amount of overlap between each USS (so op intents need to be exchanged).

While the reservation volume itself is not representative, running a test with the above parameters provides a signal on the performance of infrastructure at scale and could reduce the likelihood of SEV1 and SEV2 issues for USSs.

[BenjaminPelletier]

Could you elaborate on what the requested tool's actions would be? Would it simply call each USS's flight_planning API to instruct the USS to create a flight every so often, ensuring that the flight areas provided required interaction between all participants? Or something else/in addition?

[number] of flights per minute per 10sqkm

What does "per 10sqkm" mean? I was initially thinking to put flights all essentially at one single point (though technically deconflicted/not intersecting), but "per 10sqkm" seems to imply some amount of spatial spread. How would the spread be accomplished or specified?

The load test generates flights based on the inputs from each USS at roughly the same time.

What kind of inputs would the load test accept from each USS? I would have expected a configuration specifying the parameters of the flight plan minus some dynamically-generated information like volumes, but I wouldn't have expected that to be per USS.

[LeslieW]

Thanks for the clarifying questions!

Could you elaborate on what the requested tool's actions would be? Would it simply call each USS's flight_planning API to instruct the USS to create a flight every so often, ensuring that the flight areas provided required interaction between all participants? Or something else/in addition?

Correct, I think the requested tool's actions would be to simply call each USS's flight_planning API to instruct the USS to create a flight every so often, ensuring that the flight areas provided required interaction between all participants. Welcome other ideas, but that is what I had in mind.

What does "per 10sqkm" mean? I was initially thinking to put flights all essentially at one single point (though technically deconflicted/not intersecting), but "per 10sqkm" seems to imply some amount of spatial spread. How would the spread be accomplished or specified?

Ah -- good question. I phrased it that way as I thought that it might be a helpful constraint when prompting USSs to answer the "how much load/number of flights would you like tested?". I propose that the answer to that question is a configurable param. Definitely open to other ways of phrasing/obtaining an answer to that question. In terms of implementation for this test, I think it is perfectly reasonable to essentially generate flights at one single point (though technically deconflicted/not intersecting).

What kind of inputs would the load test accept from each USS? I would have expected a configuration specifying the parameters of the flight plan minus some dynamically-generated information like volumes, but I wouldn't have expected that to be per USS.

I think it would be helpful for each USS to share what load/number of flights they would like tested, which could be a new param that is used to generate the flights that are sent to each USS's flight_planning API. That said, I'm also open to this just being a single number for an ecosystem that Governance committees can periodically decide to update based on prompting from USSs. Example below (doesn't need to be exactly this):

New field name: total_flight_load Description: This parameter is used to generate a load test. It should reflect the max number of flights per minute a single USS (or the ecosystem, depending which direction this implementation heads) expects to support during production operations. Type: int

[BenjaminPelletier]

I think the above all makes sense. From an offline conversation, I wanted to bring up an additional question: Currently, InterUSS has a load test that is a stand-alone binary leveraging Locust to apply the load. Locust sets up a webserver that presents a UI where the user can start and stop the load and see the status of the query attempts. Fulfilling this request using that existing tool seems fairly straightforward -- we would add locus files for "flights" as a resource, and make a client capable of CRUDing those flight as applicable.

However, it sounded like the desire might to be to put this tool into uss_qualifier (the tool that conducts requirements-based automated testing). If that is the desire, I think that's probably harder and we'll need to explore further. One thing that would certainly be needed is a time/tasks bound as automated tests are supposed to run relatively quickly. The bound would be user-configurable, but would the need be satisified by a load over, say, 1-2 minutes? If not, uss_qualifier may not be a good place to put it.

We have a large-concurrency test for NetRID operations in the DSS -- would that kind of thing satisfy this request if the DSS requests were instead flight creation requests to USSs, and those requests were rate-limited according to user specification?