It seems the output files are always compressed. Is it possible to write them uncompressed.
Part of my concern is speed but I have also noticed and issue with increasing memory while writing.
For one example, I read in a 300 Mbyte csv file (1250 columns, 7500 rows) and ty to write it as a MatLab struct. The memory steadily increases until it bombs with over 10Gbytes of memory allocated (per task manager.) I have not debugged into the MatFileHandler code but I am guessing this might be part of the process of building the compressed image in memory.
On the other hand, I realize the memory issue could be completely separate from the compression process.
I just published a new release (1.2.0) that makes it possible to write uncompressed files. You can try it and see if it helps. Meanwhile, I'll still try to investigate the memory allocation issue.
It seems the output files are always compressed. Is it possible to write them uncompressed.
Part of my concern is speed but I have also noticed and issue with increasing memory while writing.
For one example, I read in a 300 Mbyte csv file (1250 columns, 7500 rows) and ty to write it as a MatLab struct. The memory steadily increases until it bombs with over 10Gbytes of memory allocated (per task manager.) I have not debugged into the MatFileHandler code but I am guessing this might be part of the process of building the compressed image in memory.
On the other hand, I realize the memory issue could be completely separate from the compression process.
Any advice appreciated.