search

catid / wirehair

Wirehair : O(N) Fountain Code for Large Data
http://wirehairfec.com
BSD 3-Clause "New" or "Revised" License
268 stars 56 forks source link

(wishlist) stream api #30

Closed danieagle closed 3 years ago

danieagle commented 3 years ago

Hi! Very Very Very Thanks for this lib.

I am planning to use WireHair with files from hundreds of megabytes to tens of gigabytes or more, ranging from 1 to 300 files simultaneously. this clearly does not fit in the server ram.

Is there any way to not have to load entire files into ram?

Thanks! Best Wishes, Daniel.

catid commented 3 years ago

I think you'd have to use swap for that or split it into separate pieces. The row/column mixing travels all over the dataset. Might be improved a bit but at the end of the day requires everything to produce the intermediate symbols and then produce output!

danieagle commented 3 years ago

Hi! The internal data per si can be "splited" in less lenghty embedded chunks. From your sugestion to other user , I'll test siameses pack and ir Cauchy Caterpillar. Siameses package more advantage: I can use It as a internal format for files and to package data in udp datagrams. ....... Note: Each day to day I realize that You Catid is The Bether and The Best programmer in The fec area from more than 20+ years with extensive Internet log recomendations to use yours fec systems!

You have my personal Gratitude!!! Very Very Very Thanks and Thanks to all yours continuous and hard work.

Best Wishes, Daniel.

catid commented 3 years ago

Wow that's nice of you to say thank you