RTLS
commented
3 weeks ago Sequin Testing
Simulated runtime
- Can we provide client SDKs that simulate ingesting and delivering messages?
- This pattern is popular with ie. Kafka so you can test Kafka “end to end” without actually running Kafka
- Seems like a high surface area both for our development and for developer UX
- Ie. Does the developer have to sprinkle mock ingestion code?
- How would a developer reasonably mock an insert performed in their database with a trigger + function?
- Because we don’t explicitly have developers publish to Sequin, I think this is ruled out as an option
Unit testing with mocks
- Mocking for unit tests allows lightweight testing of business logic
- Mock delivery for consume
- HTTP response mocking for Consume and Sync APIs
- HTTP requests for Webhook Subscriptions
- Mocking does not test transforms, preloads, etc. The developer manually has to make sure mocks align with Sequin configuration
- Assertion of ingested messages?
- ie. the user performs an action in their code then wants to assert that Sequin has ingested a message
- Is it sufficient to JIT read from a records table to assert that a record is there?
- Can this additionally apply sequence / consumer filtering to ensure they match?
- This simulates Sequin business logic which has similar challenges to simulating the full runtime, so probably not recommended
- Asserting WAL Pipelines ingestion
- Even harder than asserting records table ingestion
End to end testing
- End to end testing is a complement to more comprehensive unit testing
- End to end testing additionally tests Sequin configuration, especially with declarative configuration between dev/test/prod
- Run Sequin in a docker container locally or in CI/CD
- Configuration
- Might reasonably require both imperative and declarative configuration to cover the full range of test scenarios
- Imperatively using client API
- Enables test isolation for synchronous tests, if desired
- Runtime/dynamic configuration for randomized test scenarios
- Becomes essential to support dynamic sequence/consumers in ie. a white label streaming solution (think Ix), but we haven’t seen that as a need yet
- ChatGPT assisted user stories:
- As a developer, I want to programmatically configure Sequin sequences and consumers using the client SDK within my test code, so that I can tailor the environment to specific test cases.
- As a developer, I want to adjust Sequin configurations at runtime through the client SDK to simulate different scenarios and edge cases, so that I can thoroughly test the application's behavior under various conditions.
- As a developer new to the project, I want to use the client SDK to configure Sequin without needing to learn IaC tools, so that I can start testing immediately with minimal overhead.
- Declaratively with IaC
- All that is needed if you have a static or build time configuration for Sequin
- All of our current customers
- Guaranteed to match dev/prod configuration
- We almost certainly want to support this anyways for dev/prod
- More or less requires synchronous tests because transaction isolation must be disabled
- ChatGPT assisted user stories:
- As a DevOps engineer, I want to define the entire Sequin test environment using Infrastructure as Code (IaC) so that I can ensure consistent and repeatable deployments across all testing stages.
- As a team lead, I want all Sequin configurations in the test environment to be managed through IaC and stored in our version control system, so that we have an auditable history of changes and can easily roll back if needed.
- As a developer, I want our test environment to mirror the production Sequin setup using IaC, so that we can catch environment-specific issues early in the development cycle.
- Transaction test isolation
- Perhaps the greatest challenge for e2e testing Sequin is test isolation via transactions
- Nearly every one of the most popular ORMs uses transactions for test isolation
- These prevent Sequin for ingesting via the WAL or even reading from the database, because Sequin as a sidecar will be outside the transaction
- So we cannot configure Sequin to rely on pulls only (instead of WAL)
- We need to craft guides for each ORM on how to test without transactions
E2E? Stub? What would a guide look like?