Upgrade failed: more space needed on the / filesystem yet only 26% in use

[julian-tr]

Part way through installing the packages, the process failed with this message:

2024-06-12 11:26:07,410 - ERROR - Command ['/usr/bin/leapp', 'upgrade'] failed with return code 1
2024-06-12 11:26:07,411 - ERROR - Failed: doing the conversion. The reason: Command '['/usr/bin/leapp', 'upgrade']' returned non-zero exit status 1.

Error Summary states:

2024-06-12 11:26:07,388 - INFO - Disk Requirements:
2024-06-12 11:26:07,388 - INFO - At least 358MB more space needed on the / filesystem.

..however, the output of df -m shows:

Filesystem              1M-blocks  Used Available Use% Mounted on
devtmpfs                     3971     0      3971   0% /dev
tmpfs                        3982     0      3982   0% /dev/shm
tmpfs                        3982    10      3973   1% /run
tmpfs                        3982     0      3982   0% /sys/fs/cgroup
/dev/mapper/centos-root    120318 30485     89834  26% /
/dev/vda1                     497   254       244  51% /boot
tmpfs                         797     0       797   0% /run/user/0

so should be plenty of space? centos2alma_feedback.zip

[julian-tr]

Tried the upgrade on a different server today, and got the same failure error at the same point (while installing packages). This server has over 100GB of free space on the / filesystem.

[SandakovMM]

I found these errors in the log file:

2024-06-12 11:26:07,271 - INFO - Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
2024-06-12 11:26:07,271 - INFO - Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system

It seems that SELinux might be preventing the conversion. A similar issue is described in #273. The "not enough memory" error is unusual but could theoretically be caused by SELinux.

Please disable SELinux before performing the conversion as described in issue #273.

[paulhermanseu]

We have the same problem and SELinux was already disabled

[stijneikelboom]

The same issue is occurring for us. It seems to occur upon installing packages with DNF.

The size required by the last package corresponds with the amount of 'more space needed' in the error.

2024-06-17 19:31:45,944 - INFO - installing package nss-softokn-freebl-3.90.0-7.el8_10.i686 needs 656MB on the / filesystem
2024-06-17 19:31:45,944 - INFO - Error Summary
2024-06-17 19:31:45,944 - INFO - -------------
2024-06-17 19:31:45,944 - INFO - Disk Requirements:
2024-06-17 19:31:45,944 - INFO - At least 656MB more space needed on the / filesystem.
2024-06-17 19:31:45,979 - INFO - ============================================================
2024-06-17 19:31:45,980 - INFO - ERRORS
2024-06-17 19:31:45,980 - INFO - ============================================================
2024-06-17 19:31:45,980 - INFO - 2024-06-17 19:31:45.676732 [ERROR] Actor: dnf_package_download
2024-06-17 19:31:45,980 - INFO - Message: DNF execution failed with non zero exit code.

Another conversion attempt fails at the exact same package nss-softokn-freebl-3.90.0-7.el8_10.i686.

This is currently blocking our upgrade during scheduled maintenance. Any ideas? What does @SandakovMM think?

[Klaus-UR]

We have the same problem too! One server we could convert without problems, two others failed with more space needed on the / filesystem, while there are 70GB free! 2024-06-18 19:33:24,863 - INFO - installing package make-devel-1:4.2.1-11.el8.x86_64 needs 853MB on the / filesystem 2024-06-18 19:33:24,863 - INFO - Error Summary 2024-06-18 19:33:24,863 - INFO - ------------- 2024-06-18 19:33:24,864 - INFO - Disk Requirements: 2024-06-18 19:33:24,864 - INFO - At least 873MB more space needed on the / filesystem. 2024-06-18 19:33:24,864 - INFO - ============================================================ 2024-06-18 19:33:24,864 - INFO - END OF ERRORS 2024-06-18 19:33:24,864 - INFO - ============================================================ 2024-06-18 19:33:24,866 - INFO - Debug output2024-06-18 19:33:24.861 ERROR PID: 13739 leapp: Upgrade workflow failed, check log for details 2024-06-18 19:33:24,871 - INFO - written to /var/log/leapp/leapp-upgrade.log 2024-06-18 19:33:24,871 - INFO - ============================================================ 2024-06-18 19:33:24,871 - INFO - REPORT 2024-06-18 19:33:24,871 - INFO - ============================================================ 2024-06-18 19:33:24,871 - INFO - A report has been generated at /var/log/leapp/leapp-report.json 2024-06-18 19:33:24,871 - INFO - A report has been generated at /var/log/leapp/leapp-report.txt 2024-06-18 19:33:24,871 - INFO - ============================================================ 2024-06-18 19:33:24,871 - INFO - END OF REPORT 2024-06-18 19:33:24,872 - INFO - ============================================================ 2024-06-18 19:33:24,872 - INFO - Answerfile has been generated at /var/log/leapp/answerfile 2024-06-18 19:33:24,912 - ERROR - Command ['/usr/bin/leapp', 'upgrade'] failed with return code 1 2024-06-18 19:33:24,918 - ERROR - Failed: doing the conversion. The reason: Command '['/usr/bin/leapp', 'upgrade']' returned non-zero exit status 1. 2024-06-18 19:33:25,745 - DEBUG - Going to write json '/usr/local/psa/var/centos2alma/actions.json' with new data

selinux is disabled: SELinux status: disabled What can we do?

[Klaus-UR]

the only difference between the server who works and the servers they want work is an installed plesk acronis. Can this be the problem (dkms?)

[paulhermanseu]

We do not use Acronis so it is unrelated i think

[stijneikelboom]

We are currently rerunning the upgrade and have managed to make it proceed by increasing LEAPP_OVL_SIZE.

This can be done by adding the below to the root user's .bash_profile and by running it manually before ./centos2alma.

export LEAPP_OVL_SIZE=4096

Apparently, this was brought up earlier in https://github.com/plesk/centos2alma/issues/166, but was assumed to be fixed on the Leapp side.

Once again, our upgrade is currently still in progress. I will report back here with our final experiences afterwards.

[julian-tr]

I can confirm that increasing LEAPP_OVL_SIZE allowed our upgrade to proceed and complete successfully.

After the final reboot, Apache refused to start and was giving 502 Bad Gateway errors. Eventually tracked this down in the /var/log/httpd/error_log to 'mod_watchdog' not being enabled for some reason:

https://www.plesk.com/kb/support/unable-to-start-apache-on-a-plesk-server-ah02093-mod_watchdog-is-required/

[stijneikelboom]

Indeed, we also managed to complete the upgrade successfully after increasing LEAPP_OVL_SIZE. Afterwards, we ran into several other issues however, which kept me from reporting back here.

One of them was indeed the mod_watchdog issue. Another was an error about a too small DH key in Dovecot, which has already been addressed in https://github.com/plesk/centos2alma/issues/181, but does not seem to be fully fixed.

We are currently still having problems with Roundcube, but we might need to file another issue for that.

As far as this issue goes, LEAPP_OVL_SIZE=4096 seems to be the fix, so it may be worthwhile to incorporate it into the tool.

[Klaus-UR]

Indeed, we also managed to complete the upgrade successfully after increasing LEAPP_OVL_SIZE. Afterwards, we ran into several other issues however, which kept me from reporting back here.

One of them was indeed the mod_watchdog issue. Another was an error about a too small DH key in Dovecot, which has already been addressed in #181, but does not seem to be fully fixed.

We are currently still having problems with Roundcube, but we might need to file another issue for that.

As far as this issue goes, LEAPP_OVL_SIZE=4096 seems to be the fix, so it may be worthwhile to incorporate it into the tool.

Indeed, we also managed to complete the upgrade successfully after increasing LEAPP_OVL_SIZE. Afterwards, we ran into several other issues however, which kept me from reporting back here.

One of them was indeed the mod_watchdog issue. Another was an error about a too small DH key in Dovecot, which has already been addressed in #181, but does not seem to be fully fixed.

We are currently still having problems with Roundcube, but we might need to file another issue for that.

As far as this issue goes, LEAPP_OVL_SIZE=4096 seems to be the fix, so it may be worthwhile to incorporate it into the tool.

this was the trick! you save my day. After this the update works smooth and clean! Thank you very much!!

[SandakovMM]

Regarding LEAPP_OVL_SIZE, I will add a default value for the variable. It is still in use in the repositories from the newest leapp-data package for some reason, so the updated version requires setting the variable anyway. We should likely mention the variable in the documentation as well.