trcli sometimes defaults to CP1252 instead of unicode

[ch4h2s]

TestRail CLI Version

1.4.3

CLI Environment

Windows 10.0.18363.1256, Python 3.9.4

TestRail Version

TestRail v7.8.0 Default (1141)

TestRail Instance Type

Enterprise Cloud

Current behavior

When attempting to import an XML file with unicode characters in it that don't appear in cp1252, some test cases get created with titles that contain them, but then trcli cannot add them to a test run. It shows an error about CP1252. Apparently it's a known issue in Python, the solution would be here: https://intellipaat.com/community/19239/change-python-3-7-default-encoding-from-cp1252-to-cp65001-aka-utf-8 It may be that just the windows python defaults to that (unix usually defaults to iso-8859-1 which would make more sense since iso-8859-1 is the first 256 code points in unicode but in 8 bits).

Creating test run. Traceback (most recent call last): File "C:\PROGRAM Files\Python39\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\PROGRAM Files\Python39\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\PROGRAM Files\Python39\Scripts\trcli.exe__main.py", line 7, in File "C:\PROGRAM Files\Python39\lib\site-packages\click\core.py", line 1130, in call return self.main(*args, kwargs) File "C:\PROGRAM Files\Python39\lib\site-packages\click\core.py", line 1055, in main rv = self.invoke(ctx) File "C:\PROGRAM Files\Python39\lib\site-packages\click\core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\PROGRAM Files\Python39\lib\site-packages\click\core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "C:\PROGRAM Files\Python39\lib\site-packages\click\core.py", line 760, in invoke return callback(*args, kwargs) File "C:\PROGRAM Files\Python39\lib\site-packages\click\decorators.py", line 26, in new_func return f(get_current_context(), *args, *kwargs) File "C:\PROGRAM Files\Python39\lib\site-packages\click\decorators.py", line 84, in new_func return ctx.invoke(f, obj, args, kwargs) File "C:\PROGRAM Files\Python39\lib\site-packages\click\core.py", line 760, in invoke return __callback(*args, *kwargs) File "C:\PROGRAM Files\Python39\lib\site-packages\trcli\commands\cmd_parse_junit.py", line 92, in cli result_uploader.upload_results() File "C:\PROGRAM Files\Python39\lib\site-packages\trcli\api\results_uploader.py", line 109, in upload_results added_run, error_message = self.api_request_handler.add_run( File "C:\PROGRAM Files\Python39\lib\site-packages\trcli\api\api_request_handler.py", line 382, in add_run response = self.client.send_post(f"add_run/{project_id}", add_run_data) File "C:\PROGRAM Files\Python39\lib\site-packages\trcli\api\api_client.py", line 82, in send_post return self.send_request("POST", uri, payload, files) File "C:\PROGRAM Files\Python39\lib\site-packages\trcli\api\api_client.py", line 149, in send_request self.verbose_logging_function(verbose_log_message) File "C:\PROGRAM Files\Python39\lib\site-packages\trcli\cli.py", line 95, in vlog self.log(msg, args) File "C:\PROGRAM Files\Python39\lib\site-packages\trcli\cli.py", line 90, in log click.echo(msg, file=sys.stdout, nl=new_line) File "C:\PROGRAM Files\Python39\lib\site-packages\click\utils.py", line 299, in echo file.write(out) # type: ignore File "C:\PROGRAM Files\Python39\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u037e' in position 545: character maps to

Desired behavior

Given that TestRail itself uses utf8 and xml can use it too, trcli should use the encoding defined in the xml file to read it (or, if none is defined, attempt to guess between iso-8859-1 and utf8) and use utf8 for the output to testrail.

More Details

Another unrelated small issue, not very important since it can be easily fixed, is that when trcli installs, it accepts an outdated dependency on numpy, then throws an error at runtime. Upgrading numpy fixes it. Maybe add the api-breaking numpy version as a requirement ?

work-around: process the xml file to remove all non-ascii characters and replace them with ascii look-alikes or descriptions. Or adjust the scripts that generate that xml in the first place to not use unicode.

EDIT: replaced the progra-tilde-one with program files, so that the github mark-down editor doesn't mess it up

[d-rede]

Hello @ch4h2s

Thanks for reporting this issue. Unfortunately, I am not able to replicate it. Could you kindly provide a report file that hasn't worked for you?

Thanks

[Testinator-X]

According to the installation instructions python 3.10 or newer is recommended. Have you already checked if this also happens when you update your python Version?

[bitcoder]

Closing this in the next days unless there's more feedback on it

[ch4h2s]

Note: this may have to do with the fact that python was running on windows (Win 10) and not on linux when this happened. At any rate, trcli should force utf-8 when communicating with the testrail server and allow an option to read the encoding from the xml file and/or via an option on the command line and/or via environment.

[bitcoder]

As @Testinator-X mentioned, trcli requires at least Python 3.10; tests currently run on 3.10, 3.11, and 3.12. @ch4h2s , can you please test it using the latest trcli version (1.6.2) and Python 3.10 and let us know if the problem arises? If so, can you please share the JUnit XML report?

[ch4h2s]

Sorry, the old framework is not in use anymore, so cannot repro the issue here myself. Also that same old framework had been patched to use ASCII to go around this bug well before being deprecated (actually before this bug was even reported), so it would not have been able to reproduce the issue anyway.

If you are sure that trcli now uses UTF-8 for the XML input (or reads the encoding from the xml headers), you can close the issue.

If you need to test it on your machines/frameworks, try to get some non-ascii, non-CP1252 characters inside all of the xml strings and see what happens when run on windows.

[bitcoder]

As neither of us is unable to reproduce it, I'm closing this issue.