Add cron job to docker image for graph calculations

secure-77 / Perlite

A web-based markdown viewer optimized for Obsidian

https://perlite.secure77.de/

MIT License

1.06k stars 92 forks source link

Add cron job to docker image for graph calculations #123

Open NotGovernor opened 5 months ago

NotGovernor commented 5 months ago

This is a real dead-in-the-water issue for me as I cannot shrink my site / vault. Any help would be greatly appreciated.

Describe the bug As I added more and more files to this vault over time the initial load times have climed such that recently I've started getting the following error on the console:

[11-Apr-2024 01:26:09] NOTICE: fpm is running, pid 1
[11-Apr-2024 01:26:09] NOTICE: ready to handle connections
172.19.0.3 -  11/Apr/2024:01:26:26 +0000 "GET /index.php" 500
NOTICE: PHP message: PHP Fatal error:  Maximum execution time of 30 seconds exceeded in /var/www/perlite/helper.php on line 478

To Reproduce I'm not entirely sure, but it seems to be related to the quantity of files in the vault. I'm currently around 600 md's and another 600 images.

Device (please complete the following information): Client device doesnt matter. It's a server issue. Running in docker. 2 vcpu's 4 gb ram.

NotGovernor commented 5 months ago

Might be related to the graph? I'll try deleting the metadata.json file as a temporary workaround so I can get the site back online in the meantime. It's gotten pretty big...

[edit] That worked. Site is functional again. So the pressure is off for me. :D:D Though I have only about half of the sites content up so far -- the likely size of the metadata will probably be double what it is now. (about a 1 MB file, tens of thousands of lines)

NotGovernor commented 5 months ago

Related to issue #97 ?

secure-77 commented 5 months ago

yes, it seems to be the same problem. The problem is that your php enigne can't calculate the graph relations in 30 seconds. It only need to do it one time after the metadata.json has changed, after this the final html is stored in a tmp file.

So two solutions:

increase the execution time of php (you can change it back after the first page load and tmp file creation is done)
use this python script (you only need to adjust the path line 11 to your metadata.json) https://github.com/secure-77/Perlite/blob/main/perlite/.scripts/create_GraphLinking.py

You only need to run this script one time after the metadata.json has changed

NotGovernor commented 5 months ago

Thank you for the response and the python file. Might it be a good idea to remove the graph calculation from php and have a cron or something in the docker that runs the python every few minutes, processing only if the file has been modified?

secure-77 commented 4 months ago

I don't want to remove it completely from PHP, because if you don't have to many notes and running the non docker setup you always need then to setup a cron job, which I want to avoid. But I will think about to add a cron job to the docker image.