search

YunoHost-Apps / snserver_ynh

Standard Notes Syncing Server
https://standardnotes.org/
GNU Affero General Public License v3.0
6 stars 4 forks source link

Synching issue after failed update #115

Closed edco100 closed 1 year ago

edco100 commented 1 year ago

An update from the YH console from 2022.11.18-ynh1 to 2022.12.20-ynh1 failed.

Since I've tried the update, I have synching issues, it won't sync between devices anymore ("Unable to sync"). I also tried to add a new device, and it didn't work.

Error after the failed update:

Error: "500" Action: "PUT" /yunohost/api/apps/snserver/upgrade

Context

Logs

Full log: https://paste.yunohost.org/raw/fujanokoto

Log in Webadmin:

Now upgrading snserver...
Loading installation settings...
Checking version...
Backing up the app before upgrading (may take a while)...
Stopping a systemd service...
Ensuring downward compatibility...
Making sure dedicated system user exists...
Upgrading source files...
Upgrading dependencies...
dpkg: warning: version 'v8.2.0' has bad syntax: version number does not start with digit
Upgrading NGINX web server configuration...
Adding swap...
The main mountpoint of your system '/' is on an SD card, swap will not be added to prevent some damage of this one, but that can cause troubles for the app snserver. If you still want activate the swap, you can relaunch the command preceded by 'SD_CARD_CAN_SWAP=1'
Updating a configuration file...
Installing Standard Notes - Syncing Server...
[Error] Upgrade failed.
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
This action broke dpkg/APT (the system package managers)... You can try to solve this issue by connecting through SSH and running `sudo apt install --fix-broken` and/or `sudo dpkg --configure -a`.
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
114030 The main mountpoint of your system '/' is on an SD card, swap will not be added to prevent some damage of this one, but that can cause troubles for the app snserver. If you still want activate the swap, you can relaunch the command preceded by 'SD_CARD_CAN_SWAP=1'
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
The app was restored to the way it was before the failed upgrade.
Could not upgrade snserver: An error occurred inside the app upgrade script
Could not complete the operation 'Upgrade the 'snserver' app'.
FabianWilkens commented 1 year ago

Hi,

so the upgrade failed and the app was restored. Now you get "Unable to sync". Is that right?

Can you give me the output of these commands? sudo systemctl status snserver* sudo tail /var/log/snserver/*

Are visitors allowed? sudo yunohost user permission list snserver

permissions:
  snserver.main:
    allowed: visitors
edco100 commented 1 year ago

Thank you, Fabian. Yes, that is right.

Here's the output you requested:

sudo systemctl status snserver*

`● snserver-auth.service - StandardNotes auth Loaded: loaded (/etc/systemd/system/snserver-auth.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-01-03 12:23:58 CET; 4h 6min ago Main PID: 1909173 (node) Tasks: 62 (limit: 8986) CPU: 35.219s CGroup: /system.slice/snserver-auth.service ├─1909173 node /usr/share/yarn/bin/yarn.js start:auth ├─1909193 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1909210 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1909229 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1909242 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> └─1909260 /opt/node_n/n/versions/node/16.13.1/bin/node dist/bin/server.js

Jan 03 12:23:58 easisi.com systemd[1]: Started StandardNotes auth.

● snserver-api-gateway.service - StandardNotes api-gateway Loaded: loaded (/etc/systemd/system/snserver-api-gateway.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-01-03 12:23:41 CET; 4h 7min ago Main PID: 1908883 (node) Tasks: 62 (limit: 8986) CPU: 21.581s CGroup: /system.slice/snserver-api-gateway.service ├─1908883 node /usr/share/yarn/bin/yarn.js start:api-gateway ├─1908904 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1908921 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1908943 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1908956 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> └─1908973 /opt/node_n/n/versions/node/16.13.1/bin/node dist/bin/server.js

Jan 03 12:23:41 easisi.com systemd[1]: Started StandardNotes api-gateway.

● snserver-auth-worker.service - StandardNotes auth worker Loaded: loaded (/etc/systemd/system/snserver-auth-worker.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-01-03 12:24:25 CET; 4h 6min ago Main PID: 1909481 (node) Tasks: 62 (limit: 8986) CPU: 32.311s CGroup: /system.slice/snserver-auth-worker.service ├─1909481 node /usr/share/yarn/bin/yarn.js start:auth-worker ├─1909501 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1909518 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1909537 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1909550 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> └─1909569 /opt/node_n/n/versions/node/16.13.1/bin/node dist/bin/worker.js

Jan 03 12:24:25 easisi.com systemd[1]: Started StandardNotes auth worker.

● snserver-syncing-server.service - StandardNotes syncing server Loaded: loaded (/etc/systemd/system/snserver-syncing-server.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-01-03 12:25:11 CET; 4h 5min ago Main PID: 1910110 (node) Tasks: 62 (limit: 8986) CPU: 29.656s CGroup: /system.slice/snserver-syncing-server.service ├─1910110 node /usr/share/yarn/bin/yarn.js start:syncing-server ├─1910130 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1910148 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1910167 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1910180 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> └─1910197 /opt/node_n/n/versions/node/16.13.1/bin/node dist/bin/server.js

Jan 03 12:25:11 easisi.com systemd[1]: Started StandardNotes syncing server.

● snserver-files.service - StandardNotes files Loaded: loaded (/etc/systemd/system/snserver-files.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-01-03 12:24:52 CET; 4h 6min ago Main PID: 1909802 (node) Tasks: 62 (limit: 8986) CPU: 23.974s CGroup: /system.slice/snserver-files.service ├─1909802 node /usr/share/yarn/bin/yarn.js start:files ├─1909822 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1909839 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1909860 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1909873 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> └─1909905 /opt/node_n/n/versions/node/16.13.1/bin/node dist/bin/server.js

Jan 03 12:24:52 easisi.com systemd[1]: Started StandardNotes files.

● snserver-syncing-server-worker.service - StandardNotes syncing server worker Loaded: loaded (/etc/systemd/system/snserver-syncing-server-worker.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-01-03 12:25:36 CET; 4h 5min ago Main PID: 1910416 (node) Tasks: 62 (limit: 8986) CPU: 29.930s CGroup: /system.slice/snserver-syncing-server-worker.service ├─1910416 node /usr/share/yarn/bin/yarn.js start:syncing-server-worker ├─1910436 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1910454 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1910473 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1910486 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> └─1910509 /opt/node_n/n/versions/node/16.13.1/bin/node dist/bin/worker.js

Jan 03 12:25:36 easisi.com systemd[1]: Started StandardNotes syncing server worker.

● snserver-workspace.service - StandardNotes workspace Loaded: loaded (/etc/systemd/system/snserver-workspace.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-01-03 12:26:00 CET; 4h 4min ago Main PID: 1910723 (node) Tasks: 62 (limit: 8986) CPU: 30.523s CGroup: /system.slice/snserver-workspace.service ├─1910723 node /usr/share/yarn/bin/yarn.js start:workspace ├─1910743 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1910760 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1910780 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> ├─1910795 /opt/node_n/n/versions/node/16.13.1/bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-> └─1910813 /opt/node_n/n/versions/node/16.13.1/bin/node dist/bin/server.js

Jan 03 12:26:00 easisi.com systemd[1]: Started StandardNotes workspace. ESCOC rvice; enabled; vendor preset: enabled) 2 CET; 4h 6min ago

art:files /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs start:files /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs workspace @standardnotes/files-server start /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs node dist/bin/server.js /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs node dist/bin/server.js /bin/node dist/bin/server.js

es files.

yncing server worker server-worker.service; enabled; vendor preset: enabled) 6 CET; 4h 5min ago

service art:syncing-server-worker /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs start:syncing-server-worker /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs workspace @standardnotes/syncing-server wo> /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs node dist/bin/worker.js /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs node dist/bin/worker.js /bin/node dist/bin/worker.js

es syncing server worker.

e.service; enabled; vendor preset: enabled) 0 CET; 4h 4min ago

art:workspace /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs start:workspace /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs workspace @standardnotes/workspace-server > /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs node dist/bin/server.js /bin/node /opt/yunohost/snserver/live/.yarn/releases/yarn-4.0.0-rc.25.cjs node dist/bin/server.js /bin/node dist/bin/server.js

es workspace. lines 66-111/111 (END)`

sudo tail /var/log/snserver/*

`==> /var/log/snserver/api-gateway.log <== {"level":"info","message":"Server started on port 3000"}

==> /var/log/snserver/auth.log <== {"level":"info","message":"Server started on port 3001"}

==> /var/log/snserver/auth-worker.log <== {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"}

==> /var/log/snserver/cron.log <==

==> /var/log/snserver/files.log <== {"level":"info","message":"Server started on port 3003"}

==> /var/log/snserver/syncing-server.log <== {"level":"info","message":"Server started on port 3004"}

==> /var/log/snserver/syncing-server-worker.log <== {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"} {"level":"info","message":"Alive and kicking!"}

==> /var/log/snserver/workspace.log <== {"level":"info","message":"Server started on port 3006"}`

sudo yunohost user permission list snserver

permissions: 
  snserver.main: 
    allowed: 
      - all_users
      - visitors
FabianWilkens commented 1 year ago

I seriously can't explain why you are having this error. You can try restarting all services. sudo systemctl restart snserver*

On the other hand, I tried updating 2022.11.18-ynh1 to 2022.12.20-ynh1 and that worked for me. Did you try the upgrade a second time? In the logs I can see that there was a problem downloading a dependency, maybe this was a temporary issue.

edco100 commented 1 year ago

Unfortunately, restart of snserver (and even the whole Raspi) didn't work. It's still not syncing.

I tried to update once again (the 3rd time), it again fails with the same errors.

What else can I try?

Thanks in advance.

FabianWilkens commented 1 year ago

In the logs of your first post you get this error:

2023-01-03 12:20:20,846: DEBUG - ➤ YN0007: │ @sentry/profiling-node@npm:0.0.12 must be built because it never has been before or the last one failed
2023-01-03 12:20:25,122: DEBUG - ➤ YN0009: │ @sentry/profiling-node@npm:0.0.12 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-844e1790/build.log)
2023-01-03 12:20:37,250: DEBUG - ➤ YN0000: └ Completed in 22s 227ms
2023-01-03 12:20:37,641: DEBUG - ➤ YN0000: Failed with errors in 26s 433ms

Can you search for the same fault in the new log file from the last tried update. In this example you have at the end of the second line /tmp/xfs-844e1790/build.log This file was deleted after the raspi restart, but you should have a new one with a different path.

Can you send this logfile. In there we should get some more information.

edco100 commented 1 year ago

Thanks again.

Is it this one?

# This file contains the result of Yarn building a package (@sentry/profiling-node@npm:0.0.12)
# Script name: postinstall

/opt/yunohost/snserver/live/.pnp.cjs:23648
      Error.captureStackTrace(firstError);
            ^

Error: Qualified path resolution failed: we looked for the following paths, but none could be accessed.

Source path: /opt/yunohost/snserver/live/.yarn/unplugged/@sentry-profiling-node-npm-0.0.12-8aeba55fc9/node_modules/@>
Not found: /opt/yunohost/snserver/live/.yarn/unplugged/@sentry-profiling-node-npm-0.0.12-8aeba55fc9/node_modules/@se>
Not found: /opt/yunohost/snserver/live/.yarn/unplugged/@sentry-profiling-node-npm-0.0.12-8aeba55fc9/node_modules/@se>
Not found: /opt/yunohost/snserver/live/.yarn/unplugged/@sentry-profiling-node-npm-0.0.12-8aeba55fc9/node_modules/@se>
Not found: /opt/yunohost/snserver/live/.yarn/unplugged/@sentry-profiling-node-npm-0.0.12-8aeba55fc9/node_modules/@se>

Require stack:
- /opt/yunohost/snserver/live/.yarn/unplugged/@sentry-profiling-node-npm-0.0.12-8aeba55fc9/node_modules/@sentry/prof>
    at Function.require$$0.Module._resolveFilename (/opt/yunohost/snserver/live/.pnp.cjs:23648:13)
    at Function.require$$0.Module._load (/opt/yunohost/snserver/live/.pnp.cjs:23499:42)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/opt/yunohost/snserver/live/.yarn/unplugged/@sentry-profiling-node-npm-0.0.12-8aeba55fc9/>
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Object.require$$0.Module._extensions..js (/opt/yunohost/snserver/live/.pnp.cjs:23692:33)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.require$$0.Module._load (/opt/yunohost/snserver/live/.pnp.cjs:23512:22)
FabianWilkens commented 1 year ago

Ok,

I found out that the same problem was reported to the StandardNotes Dev team and they removed that package again. The package which is causing the problem is not compatible with arm. So it is not working on any raspis. With the next update this problem is fixed.

In the mean time it should be possible to use your existing installation. We just have to figure out why you have no access anymore.

To check all the services again with an other method, you can do this. Each of the following command should return OK

curl localhost:3000/healthcheck
curl localhost:3001/healthcheck
curl localhost:3003/healthcheck
curl localhost:3004/healthcheck
curl localhost:3006/healthcheck
edco100 commented 1 year ago

I found out that the same problem was reported to the StandardNotes Dev team and they removed that package again. The package which is causing the problem is not compatible with arm. So it is not working on any raspis. With the next update this problem is fixed.

Awesome.

To check all the services again with an other method, you can do this. Each of the following command should return OK

OK on all 5 of them.

FabianWilkens commented 1 year ago

All services are running and reachable from your server. Maybe the nginx config is not correct after the failed upgrade.

If you check this file: /etc/nginx/conf.d/SUBDOMAIN.YOUDOMAIN.TLD.d/snserver.conf

In your case line 3 should look like this: proxy_pass http://127.0.0.1:3000/;

If this is also correct i have no idea where the problem could be.

On thing you can do is to upgrade your snserver with the same version again.

git clone https://github.com/YunoHost-Apps/snserver_ynh
cd snserver_ynh
git checkout 98794a6bc726071a3ab3c66acd5741de853c26a0
sudo yunohost app upgrade snserver -f . --force
edco100 commented 1 year ago

In your case line 3 should look like this: proxy_pass http://127.0.0.1:3000/;

No, it's:

proxy_pass http://127.0.0.1:3001/;

It's the port 3001 instead of 3000. Can I change it safely to 3000?

FabianWilkens commented 1 year ago

sudo tail /var/log/snserver/*

`==> /var/log/snserver/api-gateway.log <== {"level":"info","message":"Server started on port 3000"}

You can rerun this command and check if it is still on 3000, like at the beginning. If so, just change the 3001 to 3000. Than it should work again. Than there is some issue in the restore script, which I have to find.

Edit: And after that run: sudo systemctl reload nginx

edco100 commented 1 year ago

Awesome Fabian, it's working again. Thank you so much for your time and help.

FabianWilkens commented 1 year ago

Nice, With the version (maybe next week) you should hopefully have no problems to upgrade.