search

collective / i18ndude

i18ndude performs various tasks related to ZPT's, Python Scripts and i18n.
https://pypi.org/project/i18ndude
4 stars 9 forks source link

po file encoding issue with i18ndude 5.0.1 #51

Closed vincentfretin closed 6 years ago

vincentfretin commented 6 years ago

See my comment https://github.com/plone/buildout.coredev/commit/d900d5417feb9fbcda2633c7460bd4aa3e13d2cc#commitcomment-28038520

With i18ndude 5.0.1, the po file encoding is completely messed up. It is ok with i18ndude 4.3.

vincentfretin commented 6 years ago

To reproduce, you can get the https://github.com/plone/buildout.coredev/ 5.1 branch and update in versions.cfg i18ndude = 5.0.1 and zope.tal = 4.3.0 and run the following commands to resync the po files and see the issue:

bin/buildout -c experimental/i18n.cfg
bin/i18n plone
cd src/plone.app.locales
git diff plone/app/locales/locales/fr/
vincentfretin commented 6 years ago

Instead of bin/i18n plone you can execute bin/i18ndude sync --pot src/plone.app.locales/plone/app/locales/locales/plone.pot src/plone.app.locales/plone/app/locales/locales/fr/LC_MESSAGES/plone.po which triggers the issue.

Ah it triggers the issue only if I have

import sys
sys.setdefaultencoding('utf-8')

in /usr/lib/python2.7/sitecustomize.py actually.

There is another encoding issue #52 may be related.

mauritsvanrees commented 6 years ago

Using sys.setdefaultencoding is really not recommended. It may fix something for you on your computer, but not for others. And apparently it can break something for you as well.

We can't really support Python 2 and 3 and sys.setdefaultencoding.

Strange and interesting that it works with previous versions though.

mauritsvanrees commented 6 years ago

Today I learned about export PYTHONIOENCODING=utf-8. See stackoverflow. That fixes UnicodeEncodeErrors when piping or redirecting output that contains non-ascii.

@vincentfretin I don't know why you use sys.setdefaultencoding, but I wonder if setting PYTHONIOENCODING would be an alternative for you.

vincentfretin commented 6 years ago

Yes I know it's not recommended. I use sys.setdefaultencoding because I have a customer project that use sqlalchemy with accents in table names :( and there is no other way to fix sqlalchmy table introspection. So if it works without setting sys.setdefaultencoding, I guess it's not a real issue, I'll make sure I comment it before syncing any translations :) We can close I think.