phaistos-networks / TANK

A very high performance distributed log service
Apache License 2.0
938 stars 70 forks source link

Temporary topics support #73

Open markpapadakis opened 4 years ago

markpapadakis commented 4 years ago

There may be situations that call for some kind of ad-hoc analysis, where it makes a lot of sense to produce events into a TANK topic, so that you can then later process them, and when you are done, delete the topic and release whatever resources were associated with it.

You can always create new topics, and because of TANK2's lazy initialisation of partitions and RO segments(and because TANK will auto-close a partition that hasn't been accessed for sometime), this won't have any real overhead.

However, a better way is to support transient/temporary topics. Topics that are created and designated as temporary, and such topics can be easily created and deleted. The reason for having that designation is that transient topics will be delete-able, whereas regular topics won't be, and that is because it's very unsafe to allow for deletion of regular topics.

This requires further discussion.