search

rsocket / rsocket-kotlin

RSocket Kotlin multi-platform implementation
http://rsocket.io
Apache License 2.0
553 stars 37 forks source link

Research usage of big Payloads #110

Open whyoleg opened 4 years ago

whyoleg commented 4 years ago

On current moment Payload contains 2 properties data and metadata which can store bytes. The problem is, that those properties store ByteReadPacket which works like in-memory buffer. So f.e. to send big file through in one Payload(+fragmentation), it will be needed to firstly read everything from file to ByteReadPacket, so in memory.

Need to research, if and how it's possible to send such big payloads without reading everything to memory

yschimke commented 4 years ago

My mental model has been around an individual payload being available via a stream API, like Channel, InputStream/OutputStream or similar.

whyoleg commented 4 years ago

For inspiration: ktor-io byte channels Also it has integration with File IO and JVM IO-streams.

yschimke commented 4 years ago

I'd be interested in hooking it into the CLI once it's barely working. Can testing with streaming STDIN/STDOUT pipes etc.

olme04 commented 3 years ago

better to postpone this task until ktor/ktor-io will have 2.0.0 version, which will change some Input API.

olme04 commented 3 years ago

POC api change: https://github.com/olme04/rsocket-kotlin/commit/5946241c3997cf3f711152d37ddd2b05719d19e3#diff-b20323f12dc29bebdc7845234a305c8f22b468cb7c98964cb14ad808696ec65f Main breaking change is that when using Input instead of BRP we will loose possibility to copy payload (or data/metadata). This will affect user code.

whyoleg commented 1 year ago

Someday I think we will need to migrate to https://github.com/Kotlin/kotlinx-io/issues/131, though, it depends on how well it will interop with ktor and so ktor-io, as this is the main transport now.