Excessive disk access and desktop lagging when testing story

AmberFuse commented 7 months ago

Describe the bug.

The latest version of Twine is slower than 2.7 and accesses the disk much more frequently than before. It seems to slow down the other apps too, causing choppiness and the cursor jumping from side to side.

Steps to reproduce:

Open any story with attachments and test it a few times while the browser is active. Sometimes the desktop will freeze for a brief instant, hard to notice but very annoying and somewhat worrying.

Expected behavior:

The desktop and the cursor should keep moving smoothly.

Additional context on this problem.

The laptop has a dual core processor with at least 8 GB RAM and a SSD drive. Asked a friend technician and he told the computer is OK but probably the app has some bug. I'm not the administrator and I'm not expert but this never happened with other apps before.

Twine version number

2.8.1

Does this problem occur with the web version of Twine or the desktop app?

Desktop app

What operating system does this problem occur on?

Linux

If this problem is occurring with the web version of Twine, what browser does it occur on?

None

Presubmission checklist

[ ] I am interested in working on code that would fix this bug. (This is not required to submit a bug report.)
[X] I have done a search and believe that an issue does not already exist for this bug in the GitHub repository.
[X] If this problem is occurring with the app version of Twine, it still happens after I remove the user.css file from my Twine folder. (You would've added this file yourself. If you don't understand this question, you can safely check this box.)
[X] I have read and agree to abide by this project's Code of Conduct.

greyelf commented 7 months ago

@AmberFuse

Open any story with attachments

What exactly do you mean by "attachments"?

AmberFuse commented 7 months ago

What exactly do you mean by "attachments"?

Hi, only basic stuff like still images, background textures and gifs.

greyelf commented 7 months ago

Which Story Format are you using?

still images, background textures and gifs.

Are those different types of images being stored as external files, that you are refencing using Relative URLs combined with HTML / CSS something like the following...

<img src="forest.png">

or

background-image: url("forest.png")

...or are you storing those different types of images as Base64 data within the project itself?

If the different types of images are being stored as external files, exactly where are files being stored?

AmberFuse commented 7 months ago

Which Story Format are you using?

Sugarcube.

are you storing those different types of images as Base64 data within the project itself?

Yes, inside the story.

If the different types of images are being stored as external files, exactly where are files being stored?

No this never worked reliably as the images often disappeared between sessions so I started using the other method. Its much better because the result is a single file.

I'm doing maybe less than a 100 tests in each session but the problem is persistent and the choppiness seems to have worsened a bit over time. Is it safe to downgrade to 2.7?

greyelf commented 7 months ago

Yes, inside the story.

The Twine 2.x application constantly needs to save the Passage changes you make to the HTML file associated with your project. This involves combining the content of all the project's Passages (including those storing Base64 data) with the template supplied by the selected Story Format, and (in the case of the Desktop release of the application) the result is saved to the hard-drive.

For an average SugarCube based project that doesn't include Base64 data this Project HTML file has a size that ranges somewhere between ~600KB (1) and less than 5MBs. So depending on the write speed of your hard-drive each save should be fairly quick, potentially less than a second.

But once you start storing Base64 data in a project the size of the HTML file greatly increases, thus increasing how long it takes to save that file on the hard-drive. It also greatly increases the amount of memory the application needs to use to keep a copy of the project in memory while it is being worked on.

And converting an image file to Base64 data generally increases the storage size of its data by around 30%. eg. the data of 100KB image file would become around 130KB when converted to Base64.

(1) SugarCube's template itself is around 550KB in size.

AmberFuse commented 7 months ago

greatly increases the amount of memory the application needs to use to keep a copy of the project in memory while it is being worked on.

The Gnome System Monitor says the Swap is empty so maybe low memory isn't the cause. Also checked /tmp/ (the folder where all temporary files are stored) and there was plenty of free space and nothing out of the ordinary.

once you start storing Base64 data in a project the size of the HTML file greatly increases, thus increasing how long it takes to save that file on the hard-drive.

The laptop contains a fast SSD and no other apps caused stuttering, even when saving heavy documents. Twine 2.7 didn't have this issue and was working smoothly without hindering the other apps. I'd like to go back and just keep working using the previous release. Is it safe to go back from 2.8.1 to 2.7?

klembot commented 7 months ago

@AmberFuse couple thoughts. This is going to be tricky for me to address as it seems fairly situational and I don't have a Linux computer to test on myself. (If anyone out there who might be lurking uses Linux and would like to help, that would be much appreciated!)

First off, could you give a rough idea as to the size of your story, both number of passages and overall file size? I could at least try to create a test story myself to see if I see the same problems.

I am suspicious that, if you didn't see this problem in 2.7 and it only happens when testing a story, it is related to 2.8's change that a) made the location for test stories configurable b) changed the default from the temp directory to under ~/Twine. See more info here and directions on how to configure them here. Please mind the warnings in the docs about using dedicated locations!

I don't know for sure that this will fix your problem, though. I do agree with greyelf that embedding multimedia directly into Twine stories is not something I recommend, though I understand that many people in the community have done this because Twine 2 doesn't have facilities for managing it itself. In fact, changing the scratch folder can help with this--you can put your media in one folder, set the scratch folder to a folder near it, and reference the files using relative URLs. As long as your published version uses the same folder structure, it'll work there too with no changes.

Finally, I can't say for sure that downgrading from 2.8 to 2.7 will cause no problems as I haven't tested that. But there were no breaking changes between the two versions, so I'd guess it would work fine. I would keep backups carefully if you take this route.

klembot commented 7 months ago

@volkeng I respect that you disagree with the change in 2.8 and am not going to try to change your mind about it, but I want to correct one inaccurate part of what you wrote.

Twine stopped using this standard folder, so all autosaves, all temporary files, ... are constantly written directly on disk while Twine 2.8 is active.

The only thing that changed between 2.7 and 2.8 was where the files were saved when using the Test or Play buttons from inside Twine. Those now go into the Twine directory. The mechanism by which stories are saved during editing didn't change, which is the only other part of Twine that uses a temporary directory.

I'll also reiterate that it's possible to have 2.8 use the 2.7 behavior by making a configuration change as explained in the docs I linked.

klembot / twinejs