search

InstituteforDiseaseModeling / ckan

This is a fork of the official CKAN repository. This fork is being used for internal IDM purposes and not for public distribution. CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers datahub.io, catalog.data.gov and europeandataportal.eu/data/en/dataset among many other sites.
http://ckan.org/
Other
0 stars 1 forks source link

test_cli_user_add_unicode_fullname_system_exit test fails #5

Open MeWu-IDM opened 5 years ago

MeWu-IDM commented 5 years ago

this test has some unicode data:
self.user_cmd.args = ['add', 'berty', 'password=password123', 'fullname=Harold Müffintøp', 'email=berty@example.com'] It seems that docker locale may not be set correctly

debugging info: (Pdb) data_dict['fullname'] 'Harold M\xc3\xbcffint\xc3\xb8p'

(Pdb) sys.getfilesystemencoding() 'ANSI_X3.4-1968'

error: (Pdb) data_dict['fullname'] = data_dict['fullname'].decode( sys.getfilesystemencoding()) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

call stack /home/ckan/src/ckan/ckan/tests/lib/test_cli.py(124)test_cli_user_add_unicode_fullname_system_exit() -> self.user_cmd.add() /home/ckan/src/ckan/ckan/lib/cli.py(868)add() -> user_add(self.args[1:]) -> /home/ckan/src/ckan/ckan/lib/cli.py(129)user_add()

I also checked docker container locale, it's not appeared to be set

LANG= LANGUAGE= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL=

MeWu-IDM commented 5 years ago

related failures: TestUserAdd test_cli_user_add_unicode_fullname_system_exit TestUserAdd test_cli_user_add_unicode_fullname_unicode_decode_error TestDecodeEncodePath test_decode_encode_path History

MeWu-IDM commented 5 years ago

try unicode in UI to add new user to see if it repros

MeWu-IDM commented 5 years ago

after investigating this, I found that UI seems to be able to work with unicode (full name) image

the issue is in the cli repro: after deploy to a stage machine go to UI and create a user with full name "Harold Müffintøp" run docker exec -it ckan /bin/bash in the container run this command: paster user list -c /etc/ckan/production.ini error: UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 27: ordinal not in range(12 this will also give you error paster user add badbae email="bad2@gmail.com" name="bad23" fullname="Harold Müffintøp" password="12345678" -c /etc/ckan/production.ini UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

However, this works paster user berty -c /etc/ckan/production.ini it will return User: <User id=1b1d57c0-67e1-463a-b3be-f87a6123b941 name=berty password=$pbkdf2-sha512$25000$Q8gZ41zLOcc4J.Rci5HyXg$5ry0sZ5uQS9r9iaILU4uUhDCPiTFe/FnK2I.t3NnHOgmKLxhPCUMyNlQw/9iY0CyX/Yv4YNkX2RWPCdOtZnslw fullname=Harold Müffintøp email=berty@example.com apikey=c1e4a1e9-cc70-44e6-bf1d-1c92ed2ef68e created=2019-09-05 18:53:58.855307 reset_key=None about=None activity_streams_email_notifications=False sysadmin=False state=active>

move this to phase 2 as we will manually create user without unicode for phase 1