search

AlfredoSequeida / fvid

fvid is a project that aims to encode any file as a video using 1-bit color images to survive compression algorithms for data retrieval.
MIT License
354 stars 43 forks source link

Converting large files is not feasible due to high memory usage #30

Open dylanpdx opened 3 years ago

dylanpdx commented 3 years ago

This tool works great for small files, but trying to convert a relatively large file (1GB) will cause a MemoryError, even with 32 GB of RAM

Theelx commented 3 years ago

Indeed. There's a bit of a tradeoff between memory and speed, and we've mostly gone the speed route since it was envisioned that people wouldn't try to encode a file above a few tens of megabytes. If there's an interest in adding modes for memory or speed, I'd be happy to work on that. For now though, can you run a memory profiler (scalene or memprof work well) to tell us where the code uses most of the memory on your machine, or send the 1GB file here so I can test?