[QUESTION]Increasing speed of 1 xadd vs multiple for waveform data

[bigsamich]

I am attempting to get waveform data into a redis stream. The waveform consists of 250,000 floats and is being ingested at 1hz

What I have attempted so far is to do a single xadd with said waveform as a byte string. This fits in nicely with my time constraint, but makes it difficult to digest into applications reading from redis since they have to read the entire waveform. This xadd with 1MB payload takes ~5ms

What I'd like to do is a single xadd per element. That way I could leverage xrange when digesting the data. This is taking a long time, even on the loopback. Utilizing a pipeline , I can get the xadd to average out around 50 microseconds; however I am still off by a factor of 10 to even be close to fitting into 1 second.

Is this even possible with redis / redis plus plus? Am I using the wrong data structure?

Thanks in advance

Side note: i can get roughly 10 % better by providing a timestamp to xadd compared to using "*"

Environment : Ubuntu-latest docker container gcc 8.1 redis-stable hiredis-master redispluplus-master

[sewenew]

Your first solution puts all data as a single stream item, while your second solution puts each element as a stream item. If both solution fail, you can try to put a small batch of elements as a stream item. Say, put 10 or 100 elements with a single XADD command.

Regards

[sewenew]

Since there's no update, I'll close this issue.

Regards