search

canonical / cloud-init

Official upstream for the cloud-init: cloud instance initialization
https://cloud-init.io/
Other
2.92k stars 871 forks source link

UnicodeEncodeError when creating user with non-ascii chars #3126

Closed ubuntu-server-builder closed 1 year ago

ubuntu-server-builder commented 1 year ago

This bug was originally filed in Launchpad as LP: #1751051

Launchpad details 
affected_projects = ['cloud-init (Ubuntu)', 'livecd-rootfs (Ubuntu)']
assignee = None
assignee_name = None
date_closed = 2018-03-27T20:55:33.465614+00:00
date_created = 2018-02-22T12:54:18.893369+00:00
date_fix_committed = 2018-03-08T07:53:36.232288+00:00
date_fix_released = 2018-03-27T20:55:33.465614+00:00
id = 1751051
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1751051
milestone = None
owner = ahasenack
owner_name = Andreas Hasenack
private = False
status = fix_released
submitter = ahasenack
submitter_name = Andreas Hasenack
tags = ['id-5a9fa29eda1dc1b22307ed30']
duplicates = []

Launchpad user Andreas Hasenack(ahasenack) wrote on 2018-02-22T12:54:18.893369+00:00

I was testing subiquity, and at the user creation prompt typed in "André D'Silva" for the username, and just "andre" for the login.

The installer finished fine, but upon first login I couldn't login. Booting into rescue mode showed me that the user had not been created.

Checking cloud-init logs, I find the UnicodeEncodeError. 2018-02-22 12:44:01,386 - init.py[DEBUG]: Adding user andre 2018-02-22 12:44:01,387 - util.py[WARNING]: Failed to create user andre 2018-02-22 12:44:01,387 - util.py[DEBUG]: Failed to create user andre Traceback (most recent call last): File "/usr/lib/python3/dist-packages/cloudinit/distros/init.py", line 463, in add_user util.subp(adduser_cmd, logstring=log_adduser_cmd) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1871, in subp env=env, shell=shell) File "/usr/lib/python3.6/subprocess.py", line 709, in init restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 4: ordinal not in range(128)

user-data contains this:

cloud-config

hostname: sbqt users:

cloud-init is 17.2-34-g644048e3-0ubuntu1 from bionic/main.

ubuntu-server-builder commented 1 year ago

Launchpad user Scott Moser(smoser) wrote on 2018-02-22T18:03:05.061558+00:00

I think the issue is: a.) there is no default locale set in the subiquity installed system. b.) python3 subprocess is doing a 'decode' for each argument in the command list. python2 default encoding is supposed to be based on the environment [1], but python3 default encoding is not. python3 is supposed to be utf-8. In the trace above we are down in C code where it is clearly doing 'ascii' encoding.

[1] https://docs.python.org/2/library/sys.html?highlight=getdefaultencoding#sys.getdefaultencoding [2] https://docs.python.org/3/library/stdtypes.html?highlight=decode#str.encode

You can see the problem generally below. I only use 'json' as a convienent way to pass in utf-8 characters. You can see that either unset LANG or LANG=C causes the issue.

I guess I never thought that subprocess would be converting an argument list of strings to bytes. That does make some sense.

So I think there are actually two changes: a.) subiquity (via either curtin or cloud-init) should be setting a utf-8 default locale (all ubuntu generally do that). I'm not sure why the image being installed didnt have one set.

b.) cloud-init's subp should probably just do the conversion to bytes of whatever it gets as an argument list for the command, and always assume that strings are to be encoded as utf-8.

$ cat go.py

!/usr/bin/python3

import json, subprocess, sys cmd = json.loads(sys.argv[1]) print("cmd=%s" % [x.encode("utf-8") for x in cmd]) subprocess.check_call(cmd)

my default lang is en_US.utf-8

$ ./go.py '["echo", "Andr\u00e9 DSilva"]' cmd=[b'echo', b'Andr\xc3\xa9 DSilva'] André DSilva

$ LANG=en_US.utf-8 ./go.py '["echo", "Andr\u00e9 DSilva"]' cmd=[b'echo', b'Andr\xc3\xa9 DSilva'] André DSilva

$ env -u LANG ./go.py '["echo", "Andr\u00e9 DSilva"]' cmd=[b'echo', b'Andr\xc3\xa9 DSilva'] Traceback (most recent call last): File "./go.py", line 5, in subprocess.check_call(cmd) File "/usr/lib/python3.6/subprocess.py", line 286, in check_call retcode = call(*popenargs, *kwargs) File "/usr/lib/python3.6/subprocess.py", line 267, in call with Popen(popenargs, **kwargs) as p: File "/usr/lib/python3.6/subprocess.py", line 709, in init restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 4: ordinal not in range(128)

$ LANG=C ./go.py '["echo", "Andr\u00e9 DSilva"]' cmd=[b'echo', b'Andr\xc3\xa9 DSilva'] Traceback (most recent call last): File "./go.py", line 5, in subprocess.check_call(cmd) File "/usr/lib/python3.6/subprocess.py", line 286, in check_call retcode = call(*popenargs, *kwargs) File "/usr/lib/python3.6/subprocess.py", line 267, in call with Popen(popenargs, **kwargs) as p: File "/usr/lib/python3.6/subprocess.py", line 709, in init restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 4: ordinal not in range(128)

ubuntu-server-builder commented 1 year ago

Launchpad user Launchpad Janitor(janitor) wrote on 2018-03-02T00:35:59.637769+00:00

This bug was fixed in the package cloud-init - 18.1-5-g40e77380-0ubuntu1

cloud-init (18.1-5-g40e77380-0ubuntu1) bionic; urgency=medium

ubuntu-server-builder commented 1 year ago

Launchpad user Steve Langasek(vorlon) wrote on 2018-03-07T08:27:52.748349+00:00

We need to make sure that the default locale when booting a subiquity image is C.UTF-8, not C. This probably needs fixing in livecd-rootfs and I don't think there are any code changes for subiquity.

ubuntu-server-builder commented 1 year ago

Launchpad user Launchpad Janitor(janitor) wrote on 2018-03-27T03:47:03.738660+00:00

This bug was fixed in the package livecd-rootfs - 2.515

livecd-rootfs (2.515) bionic; urgency=medium

ubuntu-server-builder commented 1 year ago

Launchpad user Chad Smith(chad.smith) wrote on 2018-03-27T20:55:35.461545+00:00

This bug is believed to be fixed in cloud-init in 18.2. If this is still a problem for you, please make a comment and set the state back to New

Thank you.