Persisting data - Githubissues

Admiral-Piett / goaws

AWS (SQS/SNS) Clone for Development testing

MIT License

779 stars 144 forks source link

Persisting data #169

Closed mozey closed 4 years ago

mozey commented 6 years ago

Please excuse me if this has been discussed before, but I could not find anything related. Are there any plans to make data persistent when running the docker version?

Maybe a -snapshot path/to/file.json param, this can be used to specify queue messages must be serialized to disk when the container is stopped, and it can be used for initialization when it is started?

p4tin commented 6 years ago

As a developer tool, I never imagined a reason to save messages in a queue to disk or any other media. Can you provide me a use case please?

mozey commented 6 years ago

I would use it for a couple of things

if my application expects queues to already exist, I could create them, make a snapshot and set it to recreate the queues when the container is started
setup of queues and mock messages for integration testing, the serialized test data might be stored in a repos
sniff the content of messages created by an unfamiliar application for analysis

I could do all of the above by writing some scripts, maybe using the aws cli tool, that is executed before or after the container starts. Just thought it could be useful if this was built in

p4tin commented 6 years ago

The first point can already be done with the JSON configuration file, as far as 2 and 3 I like... Did you intend on submitting a PR for this?

mozey commented 6 years ago

can already be done with the JSON configuration file

Thanks, I missed that you can specify queues in the config file

Did you intend on submitting a PR for this?

I can look into it, wanted to find out if I'm missing something that already exists. And I'm not sure what would be an appropriate format for the serialized data. Local DynamoDB uses one or more SQLite db files for persisting data

p4tin commented 6 years ago

If your intent is to

sniff the content of messages created by an unfamiliar application for analysis

then storing the data in a SQL DB type storage like Dynamo or SQLite might prove to be cumbersome - I'd suggest a JSON array in a file for each queue (by queueName) that way you can also restore queues without the help of the config but you might be missing other configs on those queues like subscriptions, etc,.. I'll be looking forward to the PR and don't hesitate to ask any questions you might have.