search

yulqen / bcompiler-engine

MOVED: A Python library to alleviate the pain of using Excel spreadsheets to collect data from your stakeholders.
https://git.sr.ht/~yulqen/bcompiler-engine
MIT License
4 stars 1 forks source link

Fix memory usage on export #28

Closed yulqen closed 3 years ago

yulqen commented 4 years ago

Ref https://github.com/yulqen/datamaps/issues/11

Use of a blank template in region of 3/4Mb leads to a RAM max-out at around 18 templates and subsequent machine lock.

Test files are in dft/bcompiler_engine_issue_28.zip.

banillie commented 4 years ago

Thanks Matt.

Do you know which part of the template requires some much memory? I think in the past you have used a method for looking into workbooks to see which worksheets hold a lot of memory. If you can explain/show me how you do that, that would be very useful.

yulqen commented 4 years ago

"sheet16.xml" (which I think is 5A...) is 76Mb uncompressed for some reason. There were millions of empty rows in the XML file - to the point where I couldn't jump to the bottom of the file in my editor. However I was able to delete them and resave and I've just run the export and it ran the lot in about 3 minutes. I don't know what's caused this. I'll send everything over to you, including the fixed blank_template.xlsm. Be warned - that file may now be even more unstable due to me rippings it's internals out, but it seems to have worked for now.

banillie commented 4 years ago

Thanks Matt. Files corrupting is an occupational hazard when working with excel.