Memory issues in library view (twine 2.1.0b2)

Originally reported by: tryguy (Bitbucket: tryguy, GitHub: tryguy)

Hi, I'm using twine_2.1.0b2_win64.exe on a win7-64 system, with only about 4 Gb ram.

What I'm experiencing is runaway memory when and only when I view the library page. If I don't choose a story to edit, and just let the program sit there, I see 2 processes begin to claim more and more memory. Typically, they both ramp up to about 1,000k each before one closes, and the other exponentially expands to about 2,000k before shutting down, and closing the app with it. This is repeatable, and happens every time.

What I can do though, is avoid too much time with that view and go ahead and work on a story. In that case, I see the 2 processes hover around 300k and 37k. This is with a 2Mb story file.

If I go back to library view, those two slowly start to increase again. If I do nothing, they'll both ramp up until they crash the app.

What's more, if I have things using the memory, I find there just isn't enough available memory to start the app. All I see is it attempting to load, and then quit without giving a reason. I could see this as very irritating when/if this is released as a stable release.

Before I understood the issue, I was deleting the app settings (under AppData/local). This allowed me to start the app when most of my memory was devoted to other things. I don't think this is necessary though, if you have enough avaliable memory for it. But, most users may not make that connection.

Anyway, I think this is a real problem at the moment, since users might choose to leave the app going, and come back to find it closed. And, if they're in my position, they might then spend time attempting to restart it, and seeing it close and close again, without giving reason.

So... what do we look at here? Might be win7. Might be that others are experiencing it, but they either have enough memory to not notice, or they don't stay on the library screen for any extended amount of time.

I think I just have to stay on there for 2 or so minutes before the app crashes. Anyone else have this problem? For me, it's repeatable. Right now, I have to just skip to editing, and the app seems nice and stable then. But, I don't think it should be released to the general public like this. Unless my system is the only one affected...

Bitbucket: https://bitbucket.org/klembot/twinejs/issue/292

Original comment by Chris Klimas (Bitbucket: klembot, GitHub: klembot):

Properly disable testing local storage quota :-/

The hook for when the gauge would begin testing moved at some point from ready() to created(). Because of this, the NW.js patch that disabled this became ineffective, and instead of testing the browser local storage quota, it instead tested the user's hard drive quota.

Whoops.

If you were affected by this bug, you should delete Twine's local storage on disk. The way Twine tests the quota is by creating increasingly large dummy data, then quitting and then deleting it all when it hits an out-of-space error. (This is because browsers don't provide a way to ask nicely for the available quota -- a bit like testing how much water is in a well by drawing it all up and measuring the buckets, then dumping it back into the well.) So you likely have a lot of dummy data wasting space on your disk.

On macOS, the local storage is saved to /Users/[your username]/Library/Application/Twine/Default/Local Storage. On Windows, press Windows-R and type "%APPDATA%\Twine\Default\Local Storage" and press Enter. On Linux, it is in /home/[your username]/.config/Twine/Default/Local Storage.

This will unfortunately have the side effect of resetting any preferences and custom story formats you have set, but should have no effect on your saved stories (which are elsewhere). Just to be on the safe side, I would make a copy of your stories to be sure.

Sorry everyone for this goof-up.

Resolves #292

Original comment by tryguy (Bitbucket: tryguy, GitHub: tryguy):

Well, with just the first stories in there, I'm seeing some strange behaviour. The files themselves are 1, 21 and 406 KB. And on start up, I see two twine processes consume memory up until about 100 KB each, and then they sink down again to stable levels of 45 KB and 16 KB. So, they those threads are attempting to do something, possibly succeed, and then release the memory again. I guess I'd have a stable system if I only had those three stories, since I wouldn't even miss 200 KB at start up. But, the problem probably compounds with the number of stories involved.

Any idea of what those processes do, when they use that amount of memory? Like, in the time where they're needing memory before releasing it and becoming stable? They seem to be doing it when the display is already drawn. Oh, and it also does it if I enter a story to edit and come back out into library view. Is it some kind of save checking it does? A check to see if anything needs to be saved? I think it might be something like that, since saving is done automatically.

What think?

Edit: Anyway, I think I've played with this enough for now. So, I won't test all the other files. It could just be the number of stories. I have 18, and 18 seems to break things (because it requires more and more memory to handle them (check them?)).

Original comment by tryguy (Bitbucket: tryguy, GitHub: tryguy):

So... what does the stuff in Twine/User Data/Default/Local Storage do?

There's two files that relate to chrome, with the extension .localstorage

If I delete User Data, and empty my Stories out, I have no issues with Twine. I can restart it, create a new story, go back to library view and things are pretty stable in memory, the biggest process taking around 40k.

But, if exit out and put my actual stories back in, then I see the memory stealing start. I quit to stop it from crashing, and if I then delete all my stories again, thereby making Stories empty, I see the same behavior. Even though there's no stories in there, it's gobbling up memory.

So, I did a comparison between the User Data folders from these two states. I could upload these if you want, but I think the real difference between them is the file I asked about. When it was stable, the file itself is only 13 KB. When it's gobbling memory, the file is 232,232 KB.

So, I simply copied in the smaller one over the top of the bigger one (I had them copied elsewhere), and ran Twine again. And I see it's completely stable again.

So, there are two things. (1) my actual stories seem to cause a blowout to the .localstorage file when I put them in the Stories directory and run Twine. (2) Regardless of if I take the stories out, the .localstorage file remains the same, and leads to this memory gobbling.

So... does this give you any leads? My actual stories aren't that big, I could potentially zip them up and you could try them for yourself. I haven't tracked down if there is one in particular that upsets things, or if it's a number of them, or the whole lot of them. Also, I'm running in Japanese mode, so that might be contributing somehow.

I might do what I said I was going to, and drop a story in one at a time, and see how this .localstorage file reacts to it. But, that's what I know at this stage. Probably the issue is only of concern to me, since it's only a problem if I put my own stories in there. But, I'll test, and we'll check.

Original comment by tryguy (Bitbucket: tryguy, GitHub: tryguy):

Hey, actually, you know what? I went ahead and deleted the User Data folder that's under AppData/Local/Twine. And then I restarted Twine. I got that welcome message, skipped it and went onto the library view. Since I don't have any stories in there, it's showing 0 of them, and it seems to be very stable.

So, it might have everything to do with my current story files.

I put my real story files in and restarted Twine, and I can see it grabbing more and more memory, but it's very slow. I didn't let it crash again, so I'm not sure how far it goes up, but definitely past 400k. The time when I didn't have any stories, it was stable at around 41k or something.

What I might do is selectively bring in my stories and see which one(s) might be causing the app to misbehave. This might take some time.

Original comment by tryguy (Bitbucket: tryguy, GitHub: tryguy):

Yeah, I gave beta 3 a shot. It's doing the same thing. I don't think the amount of stories in Stories is important. I dragged all my stories out of there, or I dragged the whole folder out, and when I started TWINE again, it created that folder for me, and there's no stories in there for it to display.

And yet, it exhibits the same memory usage. Ramping exponentially from 300k up to about 1500k before crashing.

I find that if I let it crash that first time, the next time I restart the app, it doesn't even make it to displaying the library stories properly, the memory is just stolen far more rapidly and it crashes. That's why I mentioned in my first post that if normal users encountered this, then they might have no idea of why the application refuses to open for them. Try, try again, and still nothing. The only reason I worked it out, was because I saw what was happening with the memory. To correct this, I have to delete settings in AppData so I get that welcome help message at the start. If it loads that, then I can normally navigate to the library and then onto a story before the memory usage gets too high.

I'm not sure what happens if I restart my computer. It might do the same thing. I haven't wanted to delete the settings, because I think it may erase the link to sugarcube 2 I have, so I've kind of been avoiding crashing it. But... to test beta 3, I let it run on, and it crashed.

Actually, is library view supposed to show all the stories in the same lime-green color? When I used it in the past, there were an assortment of colors, each story having a unique look to it. But, right now, and in beta 2, I've only seen lime-green, for all the stories. Maybe... that's involved? I have 18 stories. There's 2 2MB ones, and 2 1MB ones, and the rest very small, prototypes, or stories I haven't worked on much.

Other than what I've said, I'm not sure what else I can say about it.

Points of interest: (1) library stories are all displayed in varying shades of lime green, and (2) the rate of memory consumption doubles if the app crashes that first time. Subsequent restarts (of the app), run out of memory before much of anything is displayed (only the app window itself, nothing inside, and no stories or menu).

So, does anyone else get this? Or am I the only one seeing this behaviour? I didn't use the recent versions, so I haven't tried all the releases. I think I deleted the version I was using when I downloaded the current one. But, it may have been one from six months ago or something.

Edit: Oh yeah, I should say... when it's crashed and is consuming memory super fast, I only see one TWINE process steal the memory. On the first, I see two processes, each stealing it at the same rate. But, that's different when I'm attempting to restart it after a crash. There's definitely only one. Or, I should say, there's only one main one. There could be other TWINE processes using far lower amounts of memory, but things go too fast for me to check that.

Original comment by Chris Klimas (Bitbucket: klembot, GitHub: klembot):

I don't know what is causing this problem, but I agree it's serious. Can you tell me how large your Twine stories folder is?

I've updated the version of the underlying engine the Twine app is running on in beta 3 and am hoping this may help out. Please try installing that and let me know if it improves at all.

klembot / twinejs

Memory issues in library view (twine 2.1.0b2) #292