search

facebook / wdt

Warp speed Data Transfer (WDT) is an embeddedable library (and command line tool) aiming to transfer data between 2 systems as fast as possible over multiple TCP paths.
https://www.facebook.com/WdtOpenSource
Other
2.86k stars 391 forks source link

tail -f like behaviour #151

Closed dimitry-ishenko closed 7 years ago

dimitry-ishenko commented 7 years ago

I'd like to send a large and growing file across the network. Is this something wdt is suitable for?

For example, I have an MPEG-TS video file being recorded. I'd like to transfer this file over the network as fast as possible. As the network bandwidth varies, it is possible that wdt will hit EOF and will need to wait for new data to arrive.

ldemailly commented 7 years ago

Yes you could but you would have to write code as it's not yet supported out of the box. A combination of our support for offsets/size and download resumption feature would be the building blocks for this.

dimitry-ishenko commented 7 years ago

@ldemailly I can take a stab at it, but will need some help finding the starting point. Would you be able to point me in the right direction?

nikunjy commented 7 years ago

This is an example of making a receiver with the download resumption https://github.com/facebook/wdt/blob/master/wdtCmdLine.cpp#L305

And fileinfo, the name of the file, the offsets, etc can be set here. https://github.com/facebook/wdt/blob/master/WdtTransferRequest.h#L177

I would imagine that a wrapper could be written around Sender/receiver that uses the download resumption, or uses specified transfer request. I am not sure if you would need to use both @ldemailly ?

ldemailly commented 7 years ago

Sorry for the delays, I was (and still am) traveling, and thanks Nikunj for stepping in.

You could possibly change the directory source queue to keep producing blocks - how would it end though ? how do you tell the difference between the file being slow to grow and the file has stopped growing ?

dimitry-ishenko commented 7 years ago

@ldemailly in this case it would only be interrupted by an external signal