search

zeroc-ice / datastorm

Data centric pub/sub framework based on Ice
GNU General Public License v2.0
13 stars 7 forks source link

is there a way to achieve zero-copy with datastorm #11

Closed zengdaiwei closed 2 years ago

zengdaiwei commented 2 years ago

using [cpp::array] we can achieve zero-copy with Ice, but I cant figure out a way to achieve zero-copy with datastorm?

zengdaiwei commented 2 years ago

I have a using case for very large data transform, so my app is sensitive to memory copy; If datastorm does not support zero-copy, is there any work-around, like getting ice-proxy from reader/writer? Thanks.

bentoi commented 2 years ago

It's not possible to avoid the copy when receiving of a data sample. DataStorm keeps a single sample which is then made available to potentially multiple readers either using the getNextUnread pull method or the onSamples callback.

Can you describe a little more the architecture of your application? Which model are you using to obtain the samples from the DataStorm reader? Are you using getNextUnread or onSamples? Do you have multiple readers per process?

zengdaiwei commented 2 years ago

The main architecture of my application is: one writer process with multiple reader processes(one reader per-process).
I prefer to use onSamples to obtain samples from DataStorm reader. Each process has one single reader, which is designed to receive high-throughput stream data(several hundred megabytes per-second). I want to avoid reader's extra copy to std::vector, just like ice with [cpp::array]. Do you have any ways to work around? Thanks.

bentoi commented 2 years ago

DataStorm doesn't support this.

However, you could use DataStorm to provide the proxy of your receivers to the sender. The sender would send the data to each of the receiver directly using Ice zero-copy APIs.

DataStorm wouldn't be used anymore to transfer the stream data. It would be used to discover and manage the different receivers. Each receiver would map to a data element with distinct keys. The data would contain the proxy and possibly other data members that could be useful to the sender for sending the stream data.

zengdaiwei commented 2 years ago

Thanks for your help. I got it with your help.

---- 回复的原邮件 ---- | 发件人 | Benoit @.> | | 日期 | 2022年04月13日 22:36 | | 收件人 | @.> | | 抄送至 | @.**@.> | | 主题 | Re: [zeroc-ice/datastorm] is there a way to achieve zero-copy with datastorm (Issue #11) |

DataStorm doesn't support this.

However, you could use DataStorm to provide the proxy of your receivers to the sender. The sender would send the data to each of the receiver directly using Ice zero-copy APIs.

DataStorm wouldn't be used anymore to transfer the stream data. It would be used to discover and manage the different receivers. Each receiver would map to a data element with distinct keys. The data would contain the proxy and possibly other data members that could be useful to the sender for sending the stream data.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>