awslabs / amazon-kinesis-agent

Continuously monitors a set of log files and sends new data to the Amazon Kinesis Stream and Amazon Kinesis Firehose in near-real-time.
Other
352 stars 218 forks source link

Is it possible to user a specific value from the data as the partition key #266

Open aoflanna opened 9 months ago

aoflanna commented 9 months ago

Currently it looks like the if the deterministic option is used, the entire data object is hashed. https://github.com/awslabs/amazon-kinesis-agent/blob/master/src/com/amazon/kinesis/streaming/agent/tailing/KinesisRecord.java#L60

Is there an option anywhere to use a value from the data, let's say if your data always had some user_id or device_id and you wanted to partition by that and ensure data from the same device or user etc was used the same shard.