Early abort with "codec can't encode character"

[abien]

At the very start the script aborts with

2024-04-02 22:29:14,689 - ERROR - 'ascii' codec can't encode character '\u201c' in position 424: ordinal not in range(128)
2024-04-02 22:29:14,690 - ERROR - Please fix noted problems before proceed the conversion

centos2alma_feedback.zip

[SandakovMM]

Could you please call the script in verbose mode (--verbose) again and provide /var/log/plesk/centos2alma.log

[nilsringersma]

Hi @SandakovMM , I saw the OP did not response however I'm getting the same error. Below is the execution in verbose mode:

./centos2alma --verbose
Doing preparation checks...
Preparation checks failed: 'ascii' codec can't encode character '\u201c' in position 330: ordinal not in range(128)
Traceback (most recent call last):
  File "/root/centos2alma/pleskdistup/main.py", line 109, in required_conditions_satisfied
    writer.write(check)
  File "/root/centos2alma/pleskdistup/common/src/writers.py", line 22, in write
    sys.stdout.write(message)
UnicodeEncodeError: 'ascii' codec can't encode character '\u201c' in position 330: ordinal not in range(128)

Conversion can't be performed due to the problems noted above

[SandakovMM]

Hi @nilsringersma, Could you please provide the /var/log/plesk/centos2alma.log file?

[nilsringersma]

@SandakovMM sure thing, I've redacted the hostname but I guess thats not a problem.

2024-04-26 14:59:47,939 - INFO - Started with arguments ['./centos2alma', '--verbose']
2024-04-26 14:59:47,940 - DEBUG - Detected current OS distribution as CentOS 7
2024-04-26 14:59:47,940 - DEBUG - Current system: CentOS 7
2024-04-26 14:59:47,940 - DEBUG - Available upgraders: [Centos2AlmaConverterFactory(upgrader_name=Plesk::Centos2AlmaConverter)]
2024-04-26 14:59:47,940 - DEBUG - Looking for upgrader from CentOS 7
2024-04-26 14:59:47,940 - DEBUG - Found upgraders: [Centos2AlmaConverterFactory(upgrader_name=Plesk::Centos2AlmaConverter)]
2024-04-26 14:59:47,940 - DEBUG - Selected upgrader: Centos2AlmaConverter
2024-04-26 14:59:47,940 - DEBUG - Upgrader Centos2AlmaConverter support of your system: as source = True, as target = False
2024-04-26 14:59:47,940 - DEBUG - Start checks
2024-04-26 14:59:47,940 - DEBUG - Performing check: check if currently installed Plesk version is available
2024-04-26 14:59:47,961 - DEBUG - Received available versions of plesk are: [PleskVersion(major=18, minor=0, patch=60, hotfix=0), PleskVersion(major=18, minor=0, patch=59, hotfix=0), PleskVersion(major=18, minor=0, patch=58, hotfix=0), PleskVersion(major=18, minor=0, patch=57, hotfix=0), PleskVersion(major=18, minor=0, patch=56, hotfix=0), PleskVersion(major=18, minor=0, patch=55, hotfix=0)]
2024-04-26 14:59:47,961 - DEBUG - Plesk version installed on the host is '18.0.60.0'
2024-04-26 14:59:47,961 - DEBUG - Performing check: check if Plesk installer is in progress
2024-04-26 14:59:47,970 - DEBUG - Performing check: checking available space
2024-04-26 14:59:47,970 - DEBUG - Performing check: check for outdated PHP versions
2024-04-26 14:59:47,971 - DEBUG - Checking for minimum installed PHP version of PHP 7.1
2024-04-26 14:59:47,971 - DEBUG - Known PHP versions: [PHPVersion(major=5, minor=2), PHPVersion(major=5, minor=3), PHPVersion(major=5, minor=4), PHPVersion(major=5, minor=5), PHPVersion(major=5, minor=6), PHPVersion(major=7, minor=0), PHPVersion(major=7, minor=1), PHPVersion(major=7, minor=2), PHPVersion(major=7, minor=3), PHPVersion(major=7, minor=4), PHPVersion(major=8, minor=0), PHPVersion(major=8, minor=1), PHPVersion(major=8, minor=2), PHPVersion(major=8, minor=3)]
2024-04-26 14:59:47,971 - DEBUG - Outdated PHP versions: [PHPVersion(major=5, minor=2), PHPVersion(major=5, minor=3), PHPVersion(major=5, minor=4), PHPVersion(major=5, minor=5), PHPVersion(major=5, minor=6), PHPVersion(major=7, minor=0)]
2024-04-26 14:59:48,120 - DEBUG - Outdated PHP packages installed: []
2024-04-26 14:59:48,121 - DEBUG - No outdated PHP versions installed
2024-04-26 14:59:48,121 - DEBUG - Performing check: checking domains uses outdated PHP
2024-04-26 14:59:48,121 - DEBUG - Checking the minimum PHP version being used by the websites. The restriction is: PHP 7.1
2024-04-26 14:59:48,126 - DEBUG - Outdated PHP handlers: ["'plesk-php52-fastcgi'", "'plesk-php52-cgi'", "'plesk-php52-fpm-dedicated'", "'plesk-php52-fpm'", "'plesk-php53-fastcgi'", "'plesk-php53-cgi'", "'plesk-php53-fpm-dedicated'", "'plesk-php53-fpm'", "'plesk-php54-fastcgi'", "'plesk-php54-cgi'", "'plesk-php54-fpm-dedicated'", "'plesk-php54-fpm'", "'plesk-php55-fastcgi'", "'plesk-php55-cgi'", "'plesk-php55-fpm-dedicated'", "'plesk-php55-fpm'", "'plesk-php56-fastcgi'", "'plesk-php56-cgi'", "'plesk-php56-fpm-dedicated'", "'plesk-php56-fpm'", "'plesk-php70-fastcgi'", "'plesk-php70-cgi'", "'plesk-php70-fpm-dedicated'", "'plesk-php70-fpm'"]
2024-04-26 14:59:48,131 - DEBUG - Executing query ['/usr/sbin/plesk', 'db', '-B', '-N', '-e', "\n                SELECT d.name FROM domains d JOIN hosting h ON d.id = h.dom_id WHERE h.php_handler_id in ('plesk-php52-fastcgi', 'plesk-php52-cgi', 'plesk-php52-fpm-dedicated', 'plesk-php52-fpm', 'plesk-php53-fastcgi', 'plesk-php53-cgi', 'plesk-php53-fpm-dedicated', 'plesk-php53-fpm', 'plesk-php54-fastcgi', 'plesk-php54-cgi', 'plesk-php54-fpm-dedicated', 'plesk-php54-fpm', 'plesk-php55-fastcgi', 'plesk-php55-cgi', 'plesk-php55-fpm-dedicated', 'plesk-php55-fpm', 'plesk-php56-fastcgi', 'plesk-php56-cgi', 'plesk-php56-fpm-dedicated', 'plesk-php56-fpm', 'plesk-php70-fastcgi', 'plesk-php70-cgi', 'plesk-php70-fpm-dedicated', 'plesk-php70-fpm');\n            "]
2024-04-26 14:59:48,142 - DEBUG - Command ['/usr/sbin/plesk', 'db', '-B', '-N', '-e', "\n                SELECT d.name FROM domains d JOIN hosting h ON d.id = h.dom_id WHERE h.php_handler_id in ('plesk-php52-fastcgi', 'plesk-php52-cgi', 'plesk-php52-fpm-dedicated', 'plesk-php52-fpm', 'plesk-php53-fastcgi', 'plesk-php53-cgi', 'plesk-php53-fpm-dedicated', 'plesk-php53-fpm', 'plesk-php54-fastcgi', 'plesk-php54-cgi', 'plesk-php54-fpm-dedicated', 'plesk-php54-fpm', 'plesk-php55-fastcgi', 'plesk-php55-cgi', 'plesk-php55-fpm-dedicated', 'plesk-php55-fpm', 'plesk-php56-fastcgi', 'plesk-php56-cgi', 'plesk-php56-fpm-dedicated', 'plesk-php56-fpm', 'plesk-php70-fastcgi', 'plesk-php70-cgi', 'plesk-php70-fpm-dedicated', 'plesk-php70-fpm');\n            "] returned 0, stdout: '', stderr: ''
2024-04-26 14:59:48,142 - DEBUG - Performing check: checking cronjob uses outdated PHP
2024-04-26 14:59:48,142 - DEBUG - Checking the minimum PHP version used in cronjobs. Restriction is: PHP 7.1
2024-04-26 14:59:48,147 - DEBUG - Outdated PHP handlers: ["'plesk-php52-fastcgi'", "'plesk-php52-cgi'", "'plesk-php52-fpm-dedicated'", "'plesk-php52-fpm'", "'plesk-php53-fastcgi'", "'plesk-php53-cgi'", "'plesk-php53-fpm-dedicated'", "'plesk-php53-fpm'", "'plesk-php54-fastcgi'", "'plesk-php54-cgi'", "'plesk-php54-fpm-dedicated'", "'plesk-php54-fpm'", "'plesk-php55-fastcgi'", "'plesk-php55-cgi'", "'plesk-php55-fpm-dedicated'", "'plesk-php55-fpm'", "'plesk-php56-fastcgi'", "'plesk-php56-cgi'", "'plesk-php56-fpm-dedicated'", "'plesk-php56-fpm'", "'plesk-php70-fastcgi'", "'plesk-php70-cgi'", "'plesk-php70-fpm-dedicated'", "'plesk-php70-fpm'"]
2024-04-26 14:59:48,152 - DEBUG - Executing query ['/usr/sbin/plesk', 'db', '-B', '-N', '-e', '\n                SELECT command from ScheduledTasks WHERE type = "php" and phpHandlerId in (\'plesk-php52-fastcgi\', \'plesk-php52-cgi\', \'plesk-php52-fpm-dedicated\', \'plesk-php52-fpm\', \'plesk-php53-fastcgi\', \'plesk-php53-cgi\', \'plesk-php53-fpm-dedicated\', \'plesk-php53-fpm\', \'plesk-php54-fastcgi\', \'plesk-php54-cgi\', \'plesk-php54-fpm-dedicated\', \'plesk-php54-fpm\', \'plesk-php55-fastcgi\', \'plesk-php55-cgi\', \'plesk-php55-fpm-dedicated\', \'plesk-php55-fpm\', \'plesk-php56-fastcgi\', \'plesk-php56-cgi\', \'plesk-php56-fpm-dedicated\', \'plesk-php56-fpm\', \'plesk-php70-fastcgi\', \'plesk-php70-cgi\', \'plesk-php70-fpm-dedicated\', \'plesk-php70-fpm\');\n            ']
2024-04-26 14:59:48,162 - DEBUG - Command ['/usr/sbin/plesk', 'db', '-B', '-N', '-e', '\n                SELECT command from ScheduledTasks WHERE type = "php" and phpHandlerId in (\'plesk-php52-fastcgi\', \'plesk-php52-cgi\', \'plesk-php52-fpm-dedicated\', \'plesk-php52-fpm\', \'plesk-php53-fastcgi\', \'plesk-php53-cgi\', \'plesk-php53-fpm-dedicated\', \'plesk-php53-fpm\', \'plesk-php54-fastcgi\', \'plesk-php54-cgi\', \'plesk-php54-fpm-dedicated\', \'plesk-php54-fpm\', \'plesk-php55-fastcgi\', \'plesk-php55-cgi\', \'plesk-php55-fpm-dedicated\', \'plesk-php55-fpm\', \'plesk-php56-fastcgi\', \'plesk-php56-cgi\', \'plesk-php56-fpm-dedicated\', \'plesk-php56-fpm\', \'plesk-php70-fastcgi\', \'plesk-php70-cgi\', \'plesk-php70-fpm-dedicated\', \'plesk-php70-fpm\');\n            '] returned 0, stdout: 'httpdocs/wp-cron.php
', stderr: ''
2024-04-26 14:59:48,162 - DEBUG - Outdated PHP cronjobs: ['httpdocs/wp-cron.php']
2024-04-26 14:59:48,162 - DEBUG - Performing check: checking OS vendor PHP used by websites
2024-04-26 14:59:48,162 - DEBUG - Checking the OS vendor PHP version used by the websites
2024-04-26 14:59:48,167 - DEBUG - OS vendor PHP handlers: ["'fpm'", "'fastcgi'"]
2024-04-26 14:59:48,172 - DEBUG - Executing query ['/usr/sbin/plesk', 'db', '-B', '-N', '-e', "\n                SELECT d.name FROM domains d JOIN hosting h ON d.id = h.dom_id WHERE h.php_handler_id in ('fpm', 'fastcgi');\n            "]
2024-04-26 14:59:48,181 - DEBUG - Command ['/usr/sbin/plesk', 'db', '-B', '-N', '-e', "\n                SELECT d.name FROM domains d JOIN hosting h ON d.id = h.dom_id WHERE h.php_handler_id in ('fpm', 'fastcgi');\n            "] returned 0, stdout: 'REDACTED', stderr: ''
2024-04-26 14:59:48,181 - DEBUG - OS vendor PHP domains: ['REDACTED']
2024-04-26 14:59:48,182 - DEBUG - Performing check: checking if grub is installed
2024-04-26 14:59:48,182 - DEBUG - Performing check: checking if there is more than one NIC interface using ketnel-name
2024-04-26 14:59:48,182 - DEBUG - Performing check: checking if the last installed kernel is in use
2024-04-26 14:59:49,609 - DEBUG - Installed kernel versions: 3.10.0-1160.108.1.el7.x86_64, 3.10.0-1160.105.1.el7.x86_64, 3.10.0-1160.114.2.el7.x86_64, 3.10.0-1127.el7.x86_64, 3.10.0-1160.102.1.el7.x86_64
2024-04-26 14:59:49,612 - DEBUG - Current kernel version is '3.10.0-1127.el7.x86_64'
2024-04-26 14:59:49,612 - DEBUG - Performing check: checking if the local repository is present
2024-04-26 14:59:49,614 - DEBUG - Performing check: checking if there are duplicate repositories
2024-04-26 14:59:49,615 - DEBUG - Performing check: check mariadb repo available
2024-04-26 14:59:49,622 - DEBUG - Detected mariadb version is: 10.5.24
2024-04-26 14:59:49,623 - DEBUG - Repository file line: #http://downloads.mariadb.org/mariadb/repositories/
2024-04-26 14:59:49,623 - DEBUG - Repository file line: [mariadb]
2024-04-26 14:59:49,623 - DEBUG - Repository file line: name = MariaDB
2024-04-26 14:59:49,623 - DEBUG - Repository file line: baseurl = http://yum.mariadb.org/10.5/centos7-amd64
2024-04-26 14:59:49,623 - DEBUG - Repository file line: gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
2024-04-26 14:59:49,623 - DEBUG - Repository file line: gpgcheck=1
2024-04-26 14:59:49,694 - DEBUG - Performing check: check if the system not in a container
2024-04-26 14:59:49,694 - DEBUG - Checking if running in a container
2024-04-26 14:59:49,695 - DEBUG - Performing check: checking if all packages are up to date
2024-04-26 15:00:02,757 - DEBUG - Performing check: checking if outdated repository for letsencrypt extension is used
2024-04-26 15:00:02,757 - DEBUG - Performing check: checking postgres version 10 or later is installed
2024-04-26 15:00:02,757 - DEBUG - Performing check: checking if there are no unknown perl cpan modules
2024-04-26 15:00:02,757 - DEBUG - Performing check: check spamassassin additional plugins are disabled
2024-04-26 15:00:02,758 - ERROR - Preparation checks failed: 'ascii' codec can't encode character '\u201c' in position 330: ordinal not in range(128)
Traceback (most recent call last):
  File "/root/centos2alma/pleskdistup/main.py", line 109, in required_conditions_satisfied
    writer.write(check)
  File "/root/centos2alma/pleskdistup/common/src/writers.py", line 22, in write
    sys.stdout.write(message)
UnicodeEncodeError: 'ascii' codec can't encode character '\u201c' in position 330: ordinal not in range(128)

2024-04-26 15:00:02,758 - ERROR - Conversion can't be performed due to the problems noted above

[SandakovMM]

Ok, thank you. I will fix the problem with non-ascii symbol output into a logfile.

[abien]

Could you please call the script in verbose mode (--verbose) again and provide /var/log/plesk/centos2alma.log centos2alma.log

[SandakovMM]

Hello, I spent some time attempting to replicate the issue on my end and managed to do so only when the values of LC_ALL and LANG were changed to C. Could you please provide the current values of these environment variables from your server? As a temporary fix, I recommend invoking centos2alma with the specified variables: LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 ./cetos2alma...

[nilsringersma]

Thanks @SandakovMM for me LC_ALL is empty and LANG is set to C.UTF-8.

15:00:19 [root@xxx ~]# echo $LC_ALL

15:00:25 [root@xxx ~]# echo $LANG
C.UTF-8

From now on I will invoke the script with the specified variables.

[SandakovMM]

Thank you @nilsringersma, I can see $LANG=C.UTF-8 triggers the problem as well. :+1:

[SandakovMM]

Ok, we spend some time to find a solution to the problem, and we believe it is satisfactory. The solution provided as part of centos2alma v1.3.1

To begin with, we made the decision not to change the default encoding because we cannot guarantee the outcome of the conversion on systems with inconsistent locales. This means that although it may work for one wrongly encoded configuration file, it could cause issues with other files. Additionaly the inconsistancy could be unexpected by leapp, not only by centos2alma itself. As a default solution, we have implemented a different method for handling encoding errors when writing logs. In the new centos2alma version, you will encounter a similar error, but it should specify which pre-check has failed, allowing you to identify the problematic file.

We have also introduced a --locale option for the tool, which enables you to configure the locale that the script will use. However, we strongly advise investigating the root cause of the problem, such as identifying the specific file or installed package causing the issue. In the past, encoding problems have resulted in failures in the temporary OS used by leapp to reinstall packages, and this has led to instances becoming stuck. Therefore, caution is advised when using the --locale option.

Furthermore, it would be greatly appreciated if you could provide the results of your investigation into your encoding problems. This will enable us to add a pre-checker to identify local inconsistencies before commencing with the conversion.

[nilsringersma]

Great @SandakovMM, thank you for the addition in the new release. I will keep you updated on any encoding problems we encounter.

So far, I’ve been running the solution LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 ./centos2alma on approximately 245 hosts. About 85% of the hosts were successfully upgraded. The remaining hosts usually failed the pre-checks because GRUB was not installed. You saved me a lot of time; many thanks for that 🙏.

If you’re interested, I have some further feedback that is not directly related to this issue:

1. I also added a check to ensure the host has at least 2GB of RAM. I had over seven hosts fail with only 1GB of RAM and get stuck during the upgrade phase. When checking the output through the serial port, it showed the OOM killer had terminated processes, causing the upgrade to get stuck. If you want, I can open a new issue for it. Personally, it’s not a problem for me as I don’t prefer running Plesk on 1GB machines anyway. So now I upgrade the available memory pre-conversion.
2. I had a couple of hosts where mailboxes were used, and users could not log in to the IMAP server after the conversion. The error in /var/log/maillog was Failed to initialize SSL server context: Can't load DH parameters, as described in this Plesk support article. As a solution, I now run plesk sbin sslmng --service dovecot --strong-dh --dhparams-size=2048 post-conversion.

Thanks again for making this conversion script available 😄.

[SandakovMM]

@nilsringersma, thank you for the feedback! It is greatly appreciated and helps us to improve the tool