search

StormHub / NetUV

.Net standard/Core binding for Libuv
Other
76 stars 20 forks source link

How to consume data #53

Closed Green7 closed 7 years ago

Green7 commented 7 years ago

I'm trying to apply NetUV in my project. I am sending large amounts of data and I need entire data to process them.

Now I handle data in OnRead(StreamHandle streamHandle, ReadableBuffer data) event but in ReadableBuffer data is (of course) chunked. What is the best and most effective "NetUV" way to collect this data and process them ?

For example: I want to get and process 10 000 bytes. Now I'm using ReadableBuffer.ReadBytes, read current portion of data to temporary array and copy them to my big array allocated before. But in this situation I have a lot of unnecessary operations. How to do it better ?

StormHub commented 7 years ago

Hi, what is the average size of the data in bytes? The readable buffer internally has reference counting, it can be exposed to retain without copying.

Green7 commented 7 years ago

Averge size depends on several factors (compression, user activity etc.) but in current solution in some cases we sending data in 150 KB chunks. (and I need this full 150 KB chunk to process it).

How to use ReadableBuffer without coping ?

StormHub commented 7 years ago

In this case, I think the readable buffer needs to expose increment reference count and you can get hold of the bytes without copying. I have to add some new methods into the readable buffer.

StormHub commented 7 years ago

I have added a method in ReadableBuffer public void ReadBytes(Action<ArraySegment> processor, int length) so that you don't have to copy data out to process.

Do you think this would be enough for your processing chunks?

fryderykhuang commented 7 years ago

I my case, I need to calculate some aggregation of the buffer e.g. md5 without copy out the data, this callback syntax is not very convenient to use. Since the underlying buffer of ReadableBuffer is reference counted, it's guarenteed not being recycled before the end of stream consumer function. I don't see the point of using a callback.

StormHub commented 7 years ago

I am trying to avoid returning buffer since the underlying buffer might not be a heap array, it could be unmanaged memory.

Green7 commented 7 years ago

I have added a method in ReadableBuffer public void ReadBytes(Action processor, int length) so that you don't have to copy data out to process. Do you think this would be enough for your processing chunks?

I think it should. But now I do not use NetUv. There are two reasons: first: https://github.com/StormHub/NetUV/issues/55 Because I cant have client and server in this same application its impossible to do unit tests etc. I tried System.IO.Pipelines.Networking.Libuv and this problem does not exist there. Second (worst): Our application must work on Windows XP. So we cant use net standard :( Its hard to port NetUv to NetFramework 4, standard Net sockets are to slow for us. We are still looking for a solution to this problem ....

StormHub commented 7 years ago

Kestrel does the thread marshalling. Windows XP support is quite a stretch.