search

parallaxsecond / parsec

Platform AbstRaction for SECurity service
https://parsec.community/
Apache License 2.0
472 stars 68 forks source link

Create a null/mock provider to assist with performance measurement and integration testing #140

Open paulhowardarm opened 4 years ago

paulhowardarm commented 4 years ago

Summary

Create a non-functional provider that just returns canned responses to all API requests, possibly making use of a mocking framework for greater control of responses.

Purpose

A non-functional provider would have no purpose in production, but can be useful in testing. If the provider performs a negligible amount of work, then we can use it to help us learn the performance profile of the Parsec service and its wire protocol. This will help us to characterise the performance of Parsec relative to the back-end providers that it abstracts over.

Scope

The scope of this issue is to create a null or mock provider that supports at least as many opcodes as the existing real-world providers. It could support more. There is no specific requirement to adopt a mocking framework, although this could be included. It would be good to have some example tests, but not necessarily a full performance test suite initially. It would be good to seed some idea for how performance testing would be achieved within the CI pipeline.

For now, the focus is on the provider. In the future, it might also be useful to have stubs/mocks for other functional pieces, such as the authenticator and Key ID manager, but that's not necessarily required here.

Definition of Done

A null/mock provider module is introduced and added to the CI build with appropriate tests, including some examples of performance tests (eg. measuring round trips per second for the various opcodes).

ionut-arm commented 4 years ago

The provider should be hidden behind a feature, or possibly behind cfg(test) if it's only intended for tests.

If profiling is the main purpose of this tool, a stress test of sorts could be run with the use of pprof for generating a flamegraph. This could be done with various configurations around the provider (different key ID managers, authenticators...).

If we want to run the tests in CI and with the pprof crate mentioned above, we can probably release the flamegraph SVG as a build artifact. Although this should probably run on a nightly basis on master rather than on each PR.