Problem of gather_vm_meta with unicode characters

[greldinard]

Hi NAUBackup,

we recently discovered that the script doesn't work if unicode characters are used within the vm metadata (in our case description). The following error output is shown:

2017-09-28-(12:20:17) - ************ vm-export= ***************
2017-09-28-(12:20:17) - *** vm-export begin PROD-Lars Test
2017-09-28-(12:20:17) - vm-export - vm_name: PROD-Lars Test max_backups: 3
2017-09-28-(12:20:17) - new backup_dir: /snapshots/xenserver2_backups/PROD-Lars Test/backup-2017-09-28-(12:20:17)
2017-09-28-(12:20:17) - Writing vm config file.
'ascii' codec can't encode character u'\xfc' in position 67: ordinal not in range(128)
2017-09-28-(12:20:17) - ***ERROR EXCEPTION - <type 'exceptions.UnicodeEncodeError'>
2017-09-28-(12:20:17) - ***ERROR NOTE: see VmBackup output for details
Traceback (most recent call last):
  File "./VmBackup.py", line 1464, in <module>
    main(session)
  File "./VmBackup.py", line 335, in main
    vm_meta_status = gather_vm_meta(vm_object, full_backup_dir)
  File "./VmBackup.py", line 562, in gather_vm_meta
    vm_out.write('name_description=%s\n' % vm_record['name_description'])
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 67: ordinal not in range(128)

Probably exchanging the use of str with .encode() (e.g. .encode('utf-8')) would already solve the problem. I hope this might be included in the next bugfix release!

Thanks in advance!

[NAUbackup]

I think the issue may be just that you have a space in your VM name "PROD-Lars Test" and it may be sufficient to just enclose it in quotes when you specify it in on the command line. Could you try that, please? If wildcarded in a configuration file, I tested that and it seems to work. We have run into all sorts of issues with VM names containing spaces and recommend against using anything except of letters, numbers, and the hyphen and underscore characters. Issue arose in XenDesktop that were related to spaces in SRs, as well! The documentation syas that the code does support VM names containing spaces, but if embedded in the configuration file, the name should also be quoted, so please try that and let me know the result.

[NAUbackup]

I just tested this out both with a general wildcard notation, as well as with an individual VM name surrounded by double quotes, so I am considering this issue closed.

[greldinard]

It definitvely doesn't depend on the spaces in the VM names. As soon as we remove our Umlauts (äöüß) from the description (!) of the vms the script runs fine. As it is allowed to use any UTF-8 character within XenServer for descriptions I still consider this being a bug of the script, as it doesn't support the same set of characters that XenServer supports. XenServer is a software that is in use internationally widespread, others will experience similar errors.

[NAUbackup]

Sorry, I misunderstood what the issue was. I will reopen and revisit this as time permits.

[NAUbackup]

If you run a standard vm-export from XenCenter or using the CLI, does the same issue take place if the VM description contains umlauts or other special characters?

[greldinard]

I will check it tomorrow in office. As I recall it's no problem using the CLI.

[greldinard]

Checked. It works very well using the CLI.

[NAUbackup]

I tried it, as well. The main issue then is how to get it back into a VM, which I've not fully been able to do if I do an "xe vm-export metadata=true ..." construct. I am looking into this further and it may be an easier way to get an isolated set of VM metadata, though that comes out in binary format.

[NAUbackup]

See info on this now in the thread: https://github.com/NAUbackup/VmBackup/issues/56