Open jkleint opened 10 years ago
This makes sense for very large data sets. I would implement this after some general changes, which will come with 4.0.
Of course I will not keep anyone from working on this, so if there are patches, we could have it in 3.10 or 4.0.
How large are your test files?
A few GB.
I'm making predictions with a command like this:
When running with a large "test-file", it uses a lot of memory; with a small test-file, it uses a little memory. I'm not familiar with the internals of MyMediaLite, but can't each prediction be made independently, facilitating a streaming read of the test-file? This would be both faster and enable the use of larger data sets.
It would also be awesome if I could read from a process or fifo (i.e., streaming read without seeking) so I could store my data compressed and uncompress it on the fly.