colmsnowplow
commented
2 hours ago 👋 Interesting suggestion! You wouldn't get the nice UI experience this way, but in case you don't know about it, --output-tsv will print the events to terminal. So if you run Micro with:
docker run -p 9090:9090 snowplow/snowplow-micro:2.1.2 --output-tsv > output.tsvThen you'll get the events in a file. Appreciate that comparing events in a text file isn't the nicest experience in the world, but it might be useful to know that a clunky version of the workflow you're describing can be done this way. :)
Is your feature request related to a problem? Please describe. Snowplow Micro appears to store events entirely in memory using two lists: one for good events and one for bad. As events are stored in memory, they are not persisted beyond the lifecycle of a Snowplow Micro container.
This can pose challenges to customers using Snowplow Micro to test schemas quickly and locally by mounting them to a Snowplow Micro container. Because the container must be stopped and restarted to pick up changes to a schema, any in-memory events are lost, and comparison of pre- and post-schema-change events becomes difficult.
Describe the solution you'd like To improve the above experience, we propose that Snowplow Micro expose optional configuration, disabled by default, to persist events to a lightweight, embedded, and file-based database system such as MapDB or SwayDB. Storing events in this way would allow users to persist events beyond the lifecycle of a single container through the use of a volume mounting the DB file.
Describe alternatives you've considered We briefly considered a weightier approach using an additional component/container for data storage (DB server, Redis, etc) - but we believe this to overcomplicate the Snowplow Micro development experience. Since Micro is not meant for production use, a file-based persistence system seems most transparent, portable, and potentially simplest to implement.
Additional context N/A