PATH variable is buggy

[tituspijean]

While helping out a user on the chat room for an unrelated issue, we discovered that the PATH variable stored in the Environment lines of the services was:

PATH=/opt/rbenv/versions/mastodon/bin:/opt/node_n/n/versions/node/16/bin:/opt/node_n/bin:/opt/rbenv/shims:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

I think something funky is going on, like the standard PATH is appended again upon every upgrade of the app.

[panomaki]

How did you find that, in other words how could I check my own installation for this?

[isAAAc]

How did you find that, in other words how could I check my own installation for this?

grep Environment /etc/systemd/system/mastodon-web.service

[panomaki]

Can't reproduce:

~# grep Environment /etc/systemd/system/mastodon-web.service
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so"
Environment="RAILS_ENV=production"
Environment="PORT=3000"
Environment="PATH=/opt/rbenv/versions/mastodon/bin:/opt/node_n/n/versions/node/16/bin:/opt/node_n/bin:/opt/rbenv/shims:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

[isAAAc]

ok , i edited the path manually in /etc/systemd/system/mastodon-web.service && restarted all the mastodon services. everything looks ok ,

thx for your help ;)

[isAAAc]

please reopen , it happens again:

 root@krashboyz:~# grep Environment /etc/systemd/system/mastodon-web.service
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so"
Environment="RAILS_ENV=production"
Environment="PORT=3000"
Environment="PATH=/opt/rbenv/versions/mastodon/bin:/opt/node_n/n/versions/node/16/bin:/opt/node_n/bin:/opt/rbenv/shims:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/rbenv/shims:/opt/rbenv/bin:/opt/rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

[isAAAc]

looks i have an issue related to ruby now when i want clear cache and old files > 30 days

mastodon@krashboyz:/root$ cd /var/www/mastodon/live
mastodon@krashboyz:~/live$ export RAILS_ENV=production
mastodon@krashboyz:~/live$ bin/tootctl cache clear
/usr/bin/env: ‘ruby’: No such file or directory

EDIT:

name: Mastodon
version: 4.2.7~ynh1     //  updated yesterday

root@krashboyz:~# yunohost --version
yunohost: 
  repo: stable
  version: 11.2.10.1
yunohost-admin: 
  repo: stable
  version: 11.2.4.1
moulinette: 
  repo: stable
  version: 11.2
ssowat: 
  repo: stable
  version: 11.2

[panomaki]

The following command works with me, can you try it?

cd /var/www/mastodon/live
sudo -u mastodon RAILS_ENV=production PATH=/opt/rbenv/versions/mastodon/bin bin/tootctl cache clear

[Tagadda]

Also you could try upgrading to the testing branch and use yunohost app shell mastodon

[isAAAc]

upgrade to testing in progress ;)

[isAAAc]

upgrade to testing in progress ;)

but it failed: https://paste.yunohost.org/raw/ajizerezic

i relaunch without the testing option (and in a tmux because i'm monving), i'll give a feedback in few hours

[Tagadda]

but it failed: https://paste.yunohost.org/raw/ajizerezic

@isAAAc Can you share the upgrade logs please? (above is the remove action after the upgrade)

[isAAAc]

oh, i need to grep the logs to find the good link

we had a free space issue with the vm disk, i relaunched the restoration @ 14:23 (paris) wip

[isAAAc]

@Tagadda i think it is this one https://paste.yunohost.org/raw/imugopugal

[Tagadda]

2024-02-27 04:28:24,150: WARNING - cp: error writing '/var/cache/yunohost/files_to_keep_during_setup_source/mastodon/public/system/cache/media_attachments/files/111/916/176/098/697/155/original/86cd5fd5ec17cbb7.mp4': No space left on device

[isAAAc]

damned, upgrade failed again ... https://paste.yunohost.org/raw/pihoqeduzo

i don't understand the error... perhaps becuase the current user were in the bad folder when i launched the upgrade, the prompt return

Info: The service mastodon-streaming has correctly executed the action start.
Info: [#################+..] > Reloading NGINX web server...
Info: [####################] > Upgrade of mastodon completed
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
sh: 0: getcwd() failed: No such file or directory
Error: Could not upgrade mastodon: This action broke dpkg/APT (the system package managers)... You can try to solve this issue by connecting through SSH and running `sudo apt install --fix-broken` and/or `sudo dpkg --configure -a`.
Info: The operation 'Upgrade the 'mastodon' app' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20240228-235233-app_upgrade-mastodon' to get help
Error: The operation 'Upgrade the 'mastodon' app' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20240228-235233-app_upgrade-mastodon' to get help

apt install --fix-broken && dpkg --configure -a do nothing much

root@krashboyz:/var/www/mastodon/live# sudo -u mastodon RAILS_ENV=production PATH=/opt/rbenv/versions/mastodon/bin bin/tootctl --version
4.2.8

isn't it the good version ?

yunohost tells:

root@krashboyz:/var/www/mastodon/live# yunohost app info mastodon
description: Libre and federated social network
domain_path: masto.krashboyz.org/
name: Mastodon
version: 4.2.7~ynh1

4.2.8 vs 4.2.7$~ynh1

i'm a bit lost ^^

[isAAAc]

i'll try a testing upgrade again root@krashboyz:/var/www/mastodon/live# df -h

Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/krashboyz--vg-root  688G  425G  231G  65% /

-> enough free space

root@krashboyz:/var/www#

the running folder is not mastodon

:crossed_fingers:

[isAAAc]

Message: "cp: error writing '/var/www/mastodon/live/public/system/cache/media_attachments/files/111/940/381/327/700/774/small/9456640bdfbcd102.webp': No space left on device"
Arguments: ()
39885169 WARNING cp: error writing '/var/www/mastodon/live/public/system/cache/media_attachments/files/111/940/381/327/700/774/small/9456640bdfbcd102.webp': No space left on device

damn , how could it be possible with 231G free space ??? oO

[isAAAc]

well, i just tried again a simple upgrade of mastodon, and it is failing again https://paste.yunohost.org/raw/ufazumanop

2024-03-03 16:08:34,363: DEBUG - 3475 + ynh_print_warn '--message=Source path '\'''\'' does not exist'
2024-03-03 16:08:34,379: DEBUG - 3490 + ynh_print_log 'Source path '\'''\'' does not exist'
2024-03-03 16:08:34,379: DEBUG - 3491 + echo -e 'Source path '\'''\'' does not exist'
2024-03-03 16:08:34,380: DEBUG - 3491 + '[' 0 == 0 ']'
2024-03-03 16:08:34,395: WARNING - 3498 Source path '' does not exist
2024-03-03 16:08:34,396: DEBUG - 3499 + echo ''
2024-03-03 16:08:34,396: DEBUG - 3499 + grep --quiet /etc/fail2ban
2024-03-03 16:08:34,396: DEBUG - 3500 + return 1
2024-03-03 16:08:34,397: DEBUG - 3500 + ynh_exit_properly
2024-03-03 16:08:34,397: DEBUG - 3501 + [[ backup =~ ^install$|^upgrade$|^restore$ ]]
2024-03-03 16:08:34,886: DEBUG - 3997 + ynh_clean_setup
2024-03-03 16:08:34,887: DEBUG - 3998 + true
2024-03-03 16:08:34,888: DEBUG - 3999 + exit 1
2024-03-03 16:08:35,891: DEBUG - 5002 Could not run script: /var/cache/yunohost/app_tmp_work_dirs/app_74pm_z6q/scripts/backup
2024-03-03 16:08:35,894: WARNING - 5006 Could not back up mastodon

second try: https://paste.yunohost.org/raw/rokuxoweqi

2024-03-03 16:15:32,742: DEBUG - 1207 ++ [[ do_not_backup_data =~ (unprotected|protected|skipped)_ ]]
2024-03-03 16:15:32,743: DEBUG - 1208 ++ ynh_app_setting get mastodon do_not_backup_data
2024-03-03 16:15:32,837: DEBUG - 1302 + do_not_backup_data=1
2024-03-03 16:15:32,838: DEBUG - 1303 + '[' '!' -e '' ']'
2024-03-03 16:15:32,838: DEBUG - 1303 + ynh_print_warn '--message=Source path '\'''\'' does not exist'
2024-03-03 16:15:32,853: DEBUG - 1318 + ynh_print_log 'Source path '\'''\'' does not exist'
2024-03-03 16:15:32,854: DEBUG - 1318 + echo -e 'Source path '\'''\'' does not exist'
2024-03-03 16:15:32,854: DEBUG - 1319 + '[' 0 == 0 ']'
2024-03-03 16:15:32,871: WARNING - 1328 Source path '' does not exist
2024-03-03 16:15:32,871: DEBUG - 1328 + echo ''
2024-03-03 16:15:32,872: DEBUG - 1329 + grep --quiet /etc/fail2ban
2024-03-03 16:15:32,872: DEBUG - 1329 + return 1
2024-03-03 16:15:32,872: DEBUG - 1329 + ynh_exit_properly
2024-03-03 16:15:32,873: DEBUG - 1330 + [[ backup =~ ^install$|^upgrade$|^restore$ ]]
2024-03-03 16:15:33,360: DEBUG - 1825 + ynh_clean_setup
2024-03-03 16:15:33,361: DEBUG - 1826 + true
2024-03-03 16:15:33,361: DEBUG - 1826 + exit 1
2024-03-03 16:15:34,364: DEBUG - 2828 Could not run script: /var/cache/yunohost/app_tmp_work_dirs/app_429z_j5b/scripts/backup
2024-03-03 16:15:34,367: WARNING - 2832 Could not back up mastodon

finally i tried without backup but it fails too https://paste.yunohost.org/raw/ubototacoq

yunohost app upgrade mastodon -b

[...]

2024-03-03 16:20:29,865: DEBUG - + grep -q 'ii  postgresql-13'
2024-03-03 16:20:29,926: DEBUG - + yunohost tools regen-conf postgresql
2024-03-03 16:20:40,820: DEBUG - + sudo --login --user=postgres psql '-cALTER user mastodon WITH PASSWORD '\''**********'\''' postgres
2024-03-03 16:20:41,030: DEBUG - ALTER ROLE
2024-03-03 16:20:41,034: DEBUG - + ynh_replace_string --match_string=DB_PASS= --replace_string=DB_PASS=********** --target_file=/live/.env.production
2024-03-03 16:20:41,149: DEBUG - + sed --in-place $'s\001DB_PASS=\001DB_PASS=**********\001g' /live/.env.production
2024-03-03 16:20:41,151: WARNING - sed: can't read /live/.env.production: No such file or directory

[isAAAc]

thx to @Tagadda for the rescue last weeks <3

everything looks ok now, the present tissue can be closed, (i can't do it by myself)

thx to all for your (as always) incredible work !