search

JeffersonLab / jaws-admin-gui

Web admin interface for JAWS
https://ace.jlab.org/jaws
MIT License
0 stars 0 forks source link

Rely on Kafka __consumer_offsets? #22

Closed slominskir closed 1 month ago

slominskir commented 2 years ago

Kafka already tracks a consumer's read offset for a topic so it can resume later. This is keyed by groupId, topicName, and partition. Currently we store an offset for each topic in IndexDB and clients supply those offsets to the Server. This works, however, we may be able to instead have clients store just the groupId and provide that to the server. This approach of allowing Kafka to handle resuming from an intermediate offset may be conceptually simpler, but does have the downside that clients are providing a groupId, and could accidentally or maliciously provide a groupId that collides with someone else's.

slominskir commented 2 years ago

This probably is only reasonable if only authenticated users were able to provide a groupId, and would mean login would be required even to just view read-only data. Not sure we want that restriction.

slominskir commented 2 years ago

On second thought, just use a less predictable naming strategy (very long random number). Hmm...

slominskir commented 2 years ago

Committing offsets might be more trouble than it's worth though. We'd have to disable auto commit offsets and then wait for client browsers to notify the server of the committed offsets, which limits the benefit as the client probably must send the offsets to the server either way.

slominskir commented 1 month ago

Switched to smoothness weblib in v4.6.0 so this issue is now obsolete.