502 Bad Gateway

[belatedly]

Describe the bug

I restarted the Calckey service because I was having issues with federation and got the errors below in the log. I have no idea how to resolve this and my instance is completely inaccessible.

Context

• VPS running Debian 11
• YunoHost 11.1.15 (stable)
• Calckey 13.1.2~ynh1

Steps to reproduce

Log into Yunohost admin panel Click on tools > services Click on calckey Click restart button

Expected behavior

The service would restart and reload the UI

Logs

https://paste.yunohost.org/tesahetaya

Edit: there are more logs that occurred after that one: https://paste.yunohost.org/unuvaxabem

[belatedly]

I had installed 13.0.5 so I have the pre-upgrade backup of that, I am going to try to restore it. My users are getting antsy.

[belatedly]

The restore worked

[lapineige]

That pnpm : not found is really strange, this should not happen. Almost like it wasn't running in the right folder/with adequate right, or if pnpm is not here which would be even more serious.

One way to get more complete log would have been to launch the service manually (don't ask me why launching the service don't give the full log, I don't understand). I can't give detailed instructions right now, but the big idea is to get the command from /etc/systemd.conf.d/calckey.service or similar (I don't remember) and running it in /var/www/calckey with sudo -u calckey [TheCommand].

I had installed 13.0.5 so I have the pre-upgrade backup of that, I am going to try to restore it.

That's good, and for such a service if it's critical I would advice you to make very regular backups, and maybe one before any potentially breaking change.

My users are getting antsy.

Please not that this package is still very recent, with many major changes recently (v13 for instance, a switch from npm to pnpm…) and we can't guarantee it to work 100% of the time perfectly fine. Also please make sure to make proper backup regularly if maintaining a working service is critical. Yunohost does it automatically on upgrade, but that's not a silver bullet.

[belatedly]

I don't think the upgrade from 13.0.5 to 13.1.2 went through correctly because everything is fine now that I am back on 13.0.5. I was able to federate with the domain I added to my allow list (not being able to is what prompted me to attempt to restart the service in the first place), and I am able to restart the Calckey service without any issues.

[lapineige]

I don't really know how to help you further here.

[belatedly]

I tried to do a manual install with my existing database but the build failed, so if you can advise how I can get that to work, that'd be awesome.

[lapineige]

What do you mean by "manual install" ?

[belatedly]

I wanted to move my instance away from Yunohost and install Calckey on my VPS manually using my existing database. I was able to export and import my database to the other server, but the install wouldn't work with that database so I just started fresh instead.

[lapineige]

Ok, that's a pitty :( I cannot provide guidance on that specific matter, I don't know what can be the issue.

[belatedly]

This issue happened again when I restarted the Calckey service yesterday. I had done the update to 13.1.4.1 on April 16th and the logs were saying pnpm was not found when I restarted the service last night. I had done a backup immediately beforehand after learning from last time, but it seems like the upgrades are removing pnpm or something and breaking everything

[lapineige]

but it seems like the upgrades are removing pnpm or something and breaking everything

Wow that a big problem…

Can you share the upgrade log ?

[belatedly]

https://paste.yunohost.org/raw/oqibapakoc

Edit: I did a backup and restarted the service again to try and get the logs with the "pnpm not found" issue, but it restarted without a problem this time. Here are the logs from the service restart though: https://paste.yunohost.org/toqekuvesa - it has been throwing that bad service call issue for a long time, but nothing seems to be broken despite it

[lapineige]

So it's working now ?

That is very strange, Schrodinger Calckey…

[belatedly]

It is working yes, was there anything noteworthy in my upgrade logs? Both times were tied to an upgrade having been done and then later restarting the Calckey service from within the YH admin UI

[lapineige]

I didn't see anything suspicious, apart from that :

Removed all cached metadata files 2023-04-16 03:07:50,394: DEBUG -  ENOENT  ENOENT: no such file or directory, scandir '/var/www/calckey/.local/share/pnpm/store/v3/files'

Not sure it's a real problem 🤔

But I didn't have the time to read it in detail.

Both times were tied to an upgrade having been done and then later restarting the Calckey service from within the YH admin UI

What could be done when it fails is to launch Calckey by hand:

• service calckey status will give you the link to the systemd service that contains the command to launch Calckey.
• then you go to /var/www/calckey and launch that command. This should be a lot more verbose.

Note that it might take a few seconds to start, and give a 502 error in the midtime.

[belatedly]

Thanks, once the next update comes through I will test it and report back if the pnpm not found thing happens again. I'll close this again in the meantime!

[lapineige]

Let's hope it's closed once and for good ! :D