search

collective / i18ndude

i18ndude performs various tasks related to ZPT's, Python Scripts and i18n.
https://pypi.org/project/i18ndude
4 stars 9 forks source link

encoding problems on find untranslated strings #52

Closed gforcada closed 6 years ago

gforcada commented 6 years ago

On jenkins there is a job that searches for strings not marked for translation:

https://jenkins.plone.org/view/QA/job/qa-i18n-find-untranslated

Unfortunately, since the newer release of i18ndude it currently fails with:

Traceback (most recent call last):
  File "/home/jenkins/workspace/qa-i18n-find-untranslated/experimental/../bin/i18ndude", line 16, in <module>
    sys.exit(i18ndude.script.main())
  File "/home/jenkins/.buildout/eggs/i18ndude-5.0.1-py2.7.egg/i18ndude/script.py", line 715, in main
    errors = arguments.func(arguments)
  File "/home/jenkins/.buildout/eggs/i18ndude-5.0.1-py2.7.egg/i18ndude/script.py", line 159, in find_untranslated
    for content in common.present_file_contents(filename):
  File "/home/jenkins/.buildout/eggs/i18ndude-5.0.1-py2.7.egg/i18ndude/common.py", line 92, in present_file_contents
    yield prepare_xml(fh)
  File "/home/jenkins/.buildout/eggs/i18ndude-5.0.1-py2.7.egg/i18ndude/common.py", line 64, in prepare_xml
    return StringIO(unicode(content.strip()))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 6291: ordinal not in range(128)
mauritsvanrees commented 6 years ago

This happens at least for plone.app.viewletmanager/plone/app/viewletmanager/manage-viewletmanager.pt which contains something that looks like a space, but is actually \xc2\xa0.

I am preparing a PR for i18ndude to fix this.

Also, Vincent Fretin downgraded i18ndude in coredev because it would require a newer zope.tal 4.3.0 than we are using on 5.1. Ah, but this job runs on 5.2. Well, should be fixed with that PR.

mauritsvanrees commented 6 years ago

I have released the fix in 5.0.2 and have updated coredev 5.2.