🛑 DLF Wiki is down - Githubissues

clirdlf / uptime

Uptime for CLIR projects

https://clirdlf.github.io/uptime/

MIT License

0 stars 0 forks source link

🛑 DLF Wiki is down #6

Closed waynegraham closed 6 months ago

waynegraham commented 6 months ago

In 7ed3566, DLF Wiki (https://wiki.diglib.org) was down:

HTTP code: 500
Response time: 396 ms

waynegraham commented 6 months ago

Resolved: DLF Wiki is back up in 42953e6 after 17 minutes.

waynegraham commented 6 months ago

The solution was to run

touch includes/json/FormatJson.php

I also pulled the latest and verified that the file existed.

Only thing in the error logs are access_compat:error lines with client denied by server configuration to languages/i18n/tr.json.

Investigating suggested two changes to the conf:

php_flag register_globals off

and Require all granted to the Directory block.

waynegraham commented 6 months ago

Restarted phpfpm and issue reappeared. Looking at https://phabricator.wikimedia.org/T307816, I checked the permissions that were set to me and not www-data.

ls -Flask includes/json

Ran chown -R www-data:www-data . to update permissions.

Restarted and with a hard refresh and the issue came back.

waynegraham commented 6 months ago

This is pointing to some kind of OPcache issue.

One sledge hammer solution while I track down OPcache settings is to set a crontab to touch the file every n minutes.

waynegraham commented 6 months ago

Reconfigured cache to use memcached instead of OPCache as it appeared this may have been some kind of weird cache invalidation issue. Will continue to monitor.