mnencia
commented
6 years ago It looks like that pg_basebackup output contains some non-unicode character. Could you please check what is the default locale of the barman server?
Closed mwuttke closed 6 years ago
mnencia
commented
6 years ago It looks like that pg_basebackup output contains some non-unicode character. Could you please check what is the default locale of the barman server?
mwuttke
commented
6 years ago de_DE.UTF-8 is the default locale
mnencia
commented
6 years ago Could you please search your barman logs to see if there is one or more stacktraces? It would be useful.
mwuttke
commented
6 years ago Hello Marco,
no I could'nt find any other stacktraces at the moment.
Shoud I change the default locale?
Thanks & Greetings, Michael
mnencia
commented
6 years ago Another small question: could you please run the following query on your PostgreSQL server?
SELECT name, setting FROM pg_settings WHERE name ~ '(encoding|lc_)';Sure:
postgres=# SELECT name, setting FROM pgsettings WHERE name ~ '(encoding|lc)';
name | setting
-----------------+-------------
client_encoding | UTF8
lc_collate | de_DE.UTF-8
lc_ctype | de_DE.UTF-8
lc_messages | de_DE.UTF-8
lc_monetary | de_DE.UTF-8
lc_numeric | de_DE.UTF-8
lc_time | de_DE.UTF-8
server_encoding | UTF8(8 Zeilen)
Thanks, Michael
mnencia
commented
6 years ago Ok, thanks. That's enough information. I'll look at the code.
Meantime as a workaround, you could try to set the shell locale to C before running Barman.
export LC_ALL=CIf that doesn't work you can try also setting
lc_messages = Cin the PostgreSQL configuration.
mwuttke
commented
6 years ago Hello Marco,
Sorry, but neither works for me. Neither the changes in the shell nor on the posgresql server.
Thanks, Michael
mnencia
commented
6 years ago ok, thanks
mlachal
commented
6 years ago Hello,
I have the same problem than moodlebeuth. Here is the information I can provide:
Installed Packages
barman.noarch 2.4-1.rhel6 @pgdg94which got updated yesterday.
barman diagnose
{
"global": {
"config": {
"archiver": "off",
"barman_home": "/data/barman",
"barman_user": "barman",
"basebackup_retry_sleep": "300",
"basebackup_retry_times": "2",
"compression": "gzip",
"configuration_files_directory": "/etc/barman.d",
"errors_list": [],
"immediate_checkpoint": "true",
"last_backup_maximum_age": "1 DAYS",
"log_file": "/var/log/barman/barman.log",
"minimum_redundancy": "1",
"retention_policy": "RECOVERY WINDOW OF 3 days",
"retention_policy_mode": "auto",
"reuse_backup": "off",
"wal_retention_policy": "main"
},
"system_info": {
"barman_ver": "2.4",
"kernel_ver": "Linux vmpostgre-pbarman1 2.6.32-696.30.1.el6.x86_64 #1 SMP Tue May 22 03:28:18 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux",
"python_ver": "",
"release": "LSB Version:\t:base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch\nDistributor ID:\tCentOS\nDescription:\tCentOS release 6.9 (Final)\nRelease:\t6.9\nCodename:\tFinal",
"rsync_ver": "rsync version 3.0.6 protocol version 30",
"ssh_ver": ""
}
},
"servers": {
"vmpostgre-pdsi1": {
"backups": {
[...]
"20180606T233001": {
"backup_id": "20180606T233001",
"backup_label": null,
"begin_offset": 40,
"begin_time": "Wed Jun 6 23:30:02 2018",
"begin_wal": "0000000100000039000000A4",
"begin_xlog": "39/A4000028",
"config_file": "/data/pgsql/9.6/postgresql.conf",
"copy_stats": {
"copy_time": 5.0719139999999996,
"total_time": 5.0719139999999996
},
"deduplicated_size": 528815511,
"end_offset": 0,
"end_time": "Wed Jun 6 23:30:07 2018",
"end_wal": "0000000100000039000000A4",
"end_xlog": "39/A5000000",
"error": null,
"hba_file": "/data/pgsql/9.6/pg_hba.conf",
"ident_file": "/data/pgsql/9.6/pg_ident.conf",
"included_files": null,
"mode": "postgres",
"pgdata": "/data/pgsql/9.6",
"server_name": "vmpostgre-pdsi1",
"size": 528815511,
"status": "DONE",
"tablespaces": null,
"timeline": 1,
"version": 90609,
"xlog_segment_size": 16777216
},
"20180607T233002": {
"backup_id": "20180607T233002",
"backup_label": null,
"begin_offset": 2316312,
"begin_time": "Thu Jun 7 23:30:02 2018",
"begin_wal": "0000000100000039000000DB",
"begin_xlog": "39/DB235818",
"config_file": "/data/pgsql/9.6/postgresql.conf",
"copy_stats": null,
"deduplicated_size": null,
"end_offset": null,
"end_time": null,
"end_wal": null,
"end_xlog": null,
"error": **"failure copying files ('ascii' codec can't encode character u'\\xe9' in position 175: ordinal not in range(128))"**,
"hba_file": "/data/pgsql/9.6/pg_hba.conf",
"ident_file": "/data/pgsql/9.6/pg_ident.conf",
"included_files": null,
"mode": "postgres",
"pgdata": "/data/pgsql/9.6",
"server_name": "vmpostgre-pdsi1",
"size": null,
"status": "FAILED",
"tablespaces": null,
"timeline": 1,
"version": 90609,
"xlog_segment_size": 16777216
}postgres=# SELECT name, setting FROM pg_settings WHERE name ~ '(encoding|lc_)';
name | setting
-----------------+-------------
client_encoding | UTF8
lc_collate | fr_FR.UTF-8
lc_ctype | fr_FR.UTF-8
lc_messages | fr_FR.UTF-8
lc_monetary | fr_FR.UTF-8
lc_numeric | fr_FR.UTF-8
lc_time | fr_FR.UTF-8
server_encoding | UTF8
(8 rows)export LC_ALL=C
export lc_messages=C
barman backup vmpostgre-tdsi1
Starting backup using postgres method for server vmpostgre-tdsi1 in /data/barman/vmpostgre-tdsi1/base/20180608T123003
Backup start at LSN: 1E/3E000060 (000000010000001E0000003E, 00000060)
Starting backup copy via pg_basebackup for 20180608T123003
ERROR: Backup failed copying files.
DETAILS: 'ascii' codec can't encode character u'\xe9' in position 175: ordinal not in range(128)Thank you for your help, Mikaël
dawncold
commented
6 years ago SELECT name, setting FROM pg_settings WHERE name ~ '(encoding|lc_)';
name | setting
-----------------+-------------
client_encoding | UTF8
lc_collate | C
lc_ctype | C
lc_messages | zh_CN.UTF-8
lc_monetary | zh_CN.UTF-8
lc_numeric | zh_CN.UTF-8
lc_time | zh_CN.UTF-8
server_encoding | UTF8
(8 rows)I modified backup.py and got a traceback
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/barman/backup.py", line 393, in backup
self.executor.backup(backup_info)
File "/usr/lib/python2.7/dist-packages/barman/backup_executor.py", line 324, in backup
self.backup_copy(backup_info)
File "/usr/lib/python2.7/dist-packages/barman/backup_executor.py", line 553, in backup_copy
pg_basebackup()
File "/usr/lib/python2.7/dist-packages/barman/command_wrappers.py", line 262, in __call__
self.get_output(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/barman/command_wrappers.py", line 294, in get_output
return self._get_output_once(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/barman/command_wrappers.py", line 355, in _get_output_once
self.err = _str(self.err)
File "/usr/lib/python2.7/dist-packages/barman/command_wrappers.py", line 51, in _str
return cmd_out.decode('utf-8', 'replace')
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 110-111: ordinal not in range(128)then I print cmd_out in _str, it is
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
警告: 跳过特殊文件 "./postgresql.conf"
警告: 跳过特殊文件 "./pg_ident.conf"
警告: 跳过特殊文件 "./pg_hba.conf"
注意: 没有启用WAL归档；您必须确保所有的WAL段已通过其他的方法拷贝，这样才能完成备份
pg_basebackup: base backup completedsome Chinese with English, I think set lc_message to C is a workaround method.
dawncold
commented
6 years ago _str argument cmd_out is str or unicode, if a unicode cmd_out with non-English chars execute cmd_out.decode('utf-8', 'replace'), it raises 'UnicodeEncodeError: 'ascii' codec can't encode characters in position...', so if cmd_out is unicode, just return it.
This is a modified _str, it works for me.
def _str(cmd_out):
"""
Make a string from the output of a CommandWrapper execution.
If input is None returns a literal 'None' string
:param cmd_out: String or ByteString to convert
:return str: a string
"""
if isinstance(cmd_out, unicode):
return cmd_out
if hasattr(cmd_out, 'decode') and callable(cmd_out.decode):
return cmd_out.decode('utf-8', 'replace')
else:
return str(cmd_out)P.S. this function return a str or unicode?
mwuttke
commented
6 years ago Hello,
this issue is known more than two month now. Are you still willing to solve the problem or should I look for a better solution to backup my postgresql databases?
Cheers, Michael
bsmr
commented
6 years ago I just hit the the problem. Server Environmenr is "de_DE.UTF-8". Also the DBs are using "de_DE.UTF-8".
I tried to use LANG=en_US.UTF-8 and PYTHONIOENCODING=UTF-8 (an internet search suggested this), but those did not help.
Is there any progress on this bug or an easy workaround, please? The customer doesn't want me to fiddle in the sources.
mgvel
commented
6 years ago same problem here
This is the first backup for server server-name
ERROR: The backup has failed starting backup
Asking PostgreSQL server to finalize the backup.
ERROR: Backup failed starting backup.
DETAILS: invalid literal for int() with base 10: 'content'
Processing xlog segments from file archival for server-name
00000001000000BE0000007A
00000001000000BE0000007B
00000001000000BE0000007B.00000028.backupIn postgres server
postgres=# SELECT name, setting FROM pg_settings WHERE name ~ '(encoding|lc_)';
name | setting
-----------------+-------------
client_encoding | UTF8
lc_collate | en_US.UTF-8
lc_ctype | en_US.UTF-8
lc_messages | en_US.UTF-8
lc_monetary | en_US.UTF-8
lc_numeric | en_US.UTF-8
lc_time | en_US.UTF-8
server_encoding | UTF8LOCALE in barman server
LANG=en_US.UTF-8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C
I've updated the barman version from 2.3 to 2.4 (debian strech package from the pgdg repository ) and now I've problems using the postgresql basebackup method. The basebackup process seems to completing the data transfer but the backup ends with an error:
and:
Any idea what is wrong? Or any hints? Thanks & Greetings, Michael