pausing/crashing programme due to size of blank_template

[banillie]

Hi Matt, we've run into an issue where datamaps just pauses between extracting the data and placing it in the blank template. I'll send you the documents so you can see/test for yourself. Its due to the size of the blank template. Two questions pls:

• can you look at the template and see which part of it is using so much memory. If you could do this soon that would be very much appreciated. sorry!
• might be another handling error exception. but that's not urgent. Thanks, Will

[yulqen]

Some questions!

• what leads you to think it's to do with memory (RAM)? Have you run out of memory on your machine? Do free -m on Ubuntu to check.
• what command did you run to see this?
• is there anything different about the scenario in which you see this compared to what you usually do?
• what is the exception you've seen and what command are you running to see it?

All helpful - of course, I'll take a look when I get the files.

[banillie]

Hi Matt, not sure its the memory - sorry probably using the wrong terminology. I meant the size of the template file. We think it's the template size because when we revert back to an older version of the template, which is about a 1/3 of the size then every runs just fine. Did you get the test files ok? Thanks, Will.

[yulqen]

Well it could ultimately be a RAM issue if the template file is a 1/3 bigger, depending on what you're doing with it. I got the files and will take a look. Are you running datamaps export when this problem occurs?

[banillie]

yes datamaps export. Thanks Matt.

[yulqen]

Ok, so it's a RAM issue. I got to about 19 templates and my machine locked up - I think it had eaten up over 12Gb of RAM. I only have 16Gb on my machine - it's likely you have far less (8?) on your laptop so it's going to die well before that.

A really interesting and useful problem to have found - thank you. In order to be able to process limitless numbers of templates, which is ideally what I want, it's going to require a fundamental rewrite to the parts of the code that populate the files. Not a quick job but definitely a worthwhile one.

In the meantime, you need to break your export job into much smaller batches.

First thing you should be doing is running with as little memory used as possible, so close down applications like Pycharm and any browser like Chrome or Firefox, which are massive RAM hoggers.

Do free -m (on Ubuntu) to see how much RAM you have available. Look in the "free" column - anything three figures here means you're on the verge of locking up your machine. You want the used number to be as low as possible and the free to be high. For the guys on Windows, just instruct them to close down as many apps as possible before running.

Try batches of five files to start with.

Sorry this is inconvenient - it's fundamental stuff. The way datamaps works at the moment is that it does all the master-reading first and holds the data in memory, then it has to load the blank template each time it writes it into the template, without freeing all that memory first. Another reason why storing all your data in a database would help - this wouldn't be an issue. It's a fundamental design problem anyway which is going to be interesting to solve.

Hope you can find a way to still make it useable in the meantime. We could write a little bash script that breaks the batch down automatically - I'll have a think about that too.

[yulqen]

Closed and transferred to https://github.com/yulqen/bcompiler-engine/issues/28.