Closed vincentfretin closed 4 years ago
vincentfretin
commented
4 years ago @dataflake @d-maurer Before release Zope 4.4.5 can we make sure this is fixed? Maybe this will be fixed with your latest changes. Maybe I can test Zope 4.4.5 before this is released? Please tell me how, what version of packages or checkout I need.
dataflake
commented
4 years ago Run your tests with the Zope 4.x branch and report back.
vincentfretin
commented
4 years ago I have the error only on Python 2.7.17, not on Python 3.6.9
dataflake
commented
4 years ago Please confirm: Did you run your tests on the current 4.x branch?
vincentfretin
commented
4 years ago Is adding https://raw.githubusercontent.com/zopefoundation/Zope/4.x/versions.cfg to Plone buildout extends enough or is there unreleased packages?
dataflake
commented
4 years ago I do not know how the Plone tests are configured or run so I cannot really answer that question. I normally use mr.developer to check out Zope and add a versions stanza that removes the Zope version pin.
vincentfretin
commented
4 years ago Ok I found out how to modify https://github.com/plone/buildout.coredev branch 5.2.
Modify versions.cfg to use
extends = https://raw.githubusercontent.com/zopefoundation/Zope/4.x/versions.cfg
and add Zope to auto-checkout in checkouts.cfg
So I still get the error with a different traceback on Python 2.7:
2020-07-09 08:44:37,730 ERROR [Zope.SiteErrorLog:252][waitress-1] 1594277077.730.769589697737 http://0.0.0.0:8080/Plone/front-page/content_status_history
Traceback (innermost last):
Module ZPublisher.WSGIPublisher, line 162, in transaction_pubevents
Module ZPublisher.WSGIPublisher, line 359, in publish_module
Module ZPublisher.WSGIPublisher, line 262, in publish
Module ZPublisher.mapply, line 85, in mapply
Module ZPublisher.WSGIPublisher, line 63, in call_object
Module plone.app.content.browser.content_status_history, line 125, in __call__
Module Products.Five.browser.pagetemplatefile, line 126, in __call__
Module Products.Five.browser.pagetemplatefile, line 61, in __call__
Module zope.pagetemplate.pagetemplate, line 135, in pt_render
Module Products.PageTemplates.engine, line 332, in __call__
Module z3c.pt.pagetemplate, line 176, in render
Module chameleon.zpt.template, line 307, in render
Module chameleon.template, line 214, in render
Module chameleon.template, line 192, in render
Module f4fb66098a7c48e055f3330809be7ab6, line 3025, in render
Module 2eb701eda399d90ec726666ffa6c46f0, line 1255, in render_master
Module 2eb701eda399d90ec726666ffa6c46f0, line 430, in render_content
Module f4fb66098a7c48e055f3330809be7ab6, line 1358, in __fill_main
Module Products.PageTemplates.Expressions, line 445, in __call__
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)
- Stream: Félicitations ! Vous avez installé Plone avec succès.
^
- Expression: "context/main_template/macros/master"
- Filename: ... /app/content/browser/templates/content_status_history.pt
- Location: (line 6: col 23)
- Source: ... etal:use-macro="context/main_template/macros/master"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Arguments: repeat: <Products.PageTemplates.engine.RepeatDictWrapper object at 0x7faf70aa0320>
template: <Products.Five.browser.pagetemplatefile.ViewPageTemplateFile object at 0x7faf71cc1350>
views: <Products.Five.browser.pagetemplatefile.ViewMapper object at 0x7faf65e73690>
request: <WSGIRequest, URL=http://0.0.0.0:8080/Plone/front-page/content_status_history>
args: ()
here: <Document at /Plone/front-page>
user: <PropertiedUser 'admin'>
loop: {u'item': <Products.PageTemplates.engine.RepeatItem object at 0x7faf65e80ed0>}
nothing: None
root: <Application at >
container: <Document at /Plone/front-page>
modules: <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter object at 0x7faf76c357d0>
traverse_subpath: []
default: <DEFAULT>
context: <Document at /Plone/front-page>
view: <Products.Five.browser.metaconfigure.ContentStatusHistoryView object at 0x7faf71201d90>
translate: <function translate at 0x7faf65f489d0>
macroname: u'master'
options: {}
target_language: NoneI launched all Plone tests on jenkins too https://github.com/plone/buildout.coredev/pull/663 Let's see if we have any regression.
vincentfretin
commented
4 years ago I don't have the issue with Zope 4.3, I tested with https://github.com/plone/buildout.coredev/pull/662 So this is regression with Zope 4.4.x on Python 2.7.x
vincentfretin
commented
4 years ago src/plone.app.content/plone/app/content/browser/templates/content_status_history.pt
uses metal:use-macro="context/main_template/macros/master"
then execute the fill-slot="main" macro in the content_status_history.pt template
it defines the variable item_description item/Description; in https://github.com/plone/plone.app.content/blob/61c3b9918d79451aeceed63969cde930c70e2cef/plone/app/content/browser/templates/content_status_history.pt#L124
and the UnicodeDecodeError is triggered by
tal:attributes="title string:$item_type: $item_description;" in
https://github.com/plone/plone.app.content/blob/61c3b9918d79451aeceed63969cde930c70e2cef/plone/app/content/browser/templates/content_status_history.pt#L165
I have the error if I use:
tal:attributes="title string:$item_description;"
I don't have the error if I use:
tal:attributes="title item_description;"
or
tal:attributes="title python:item_type + ' : ' + item_description;"
So the issue is really an issue with string expression with a variable of type str utf-8
vincentfretin
commented
4 years ago As far as I understand it, in Zope 4.4.x on Python 2.7.x
tal:attributes="title string:$item_type: $item_description;"
tries to decode item_description ("Félicitations !") from ascii (this should be utf-8 I think by default, this would fix the issue) to unicode to concatenate with "Document:" (this one decode just fine from ascii to unicode)
vincentfretin
commented
4 years ago [25] > /home/vincentfretin/workspace/buildout.coredev-5.2/src/Zope/src/Products/PageTemplates/Expressions.py(449)__call__()
-> pass
(Pdb++) l
444 vvals.append(v)
445 try:
446 return self._expr % tuple(vvals)
447 except Exception:
448 import pdb; pdb.set_trace()
449 -> pass
450
451
452 def createZopeEngine(zpe=ZopePathExpr, untrusted=True):
453 e = ZopeEngine()
454 e.iteratorFactory = PathIterator
(Pdb++) self._expr
u'%s: %s'
(Pdb++) vvals
['Document', 'F\xc3\xa9licitations ! Vous avez install\xc3\xa9 Plone avec succ\xc3\xa8s.']
(Pdb++) 
d-maurer
commented
4 years ago Vincent Fretin wrote at 2020-7-9 00:29 -0700:
src/plone.app.content/plone/app/content/browser/templates/content_status_history.ptusesmetal:use-macro="context/main_template/macros/master"then execute the fill-slot="main" macro in the content_status_history.pt template it defines the variableitem_description item/Description;in https://github.com/plone/plone.app.content/blob/61c3b9918d79451aeceed63969cde930c70e2cef/plone/app/content/browser/templates/content_status_history.pt#L124 and the UnicodeDecodeError is triggered bytal:attributes="title string:$item_type: $item_description;" in https://github.com/plone/plone.app.content/blob/61c3b9918d79451aeceed63969cde930c70e2cef/plone/app/content/browser/templates/content_status_history.pt#L165 I have the error if I use:tal:attributes="title string:$item_description;" I don't have the error if I use:tal:attributes="title item_description;"`tal:attributes="title python:item_type + ' : ' + item_description;"So the issue is really an issue with string expression with a variable of type bytes utf-8
I will try to construct a minimal test and see what can be done.
However, UnicodeError is usually an application error - caused
by combining unicode and bytes. In principal, everything should
either be unicode or bytes - not a combination thereof.
vincentfretin
commented
4 years ago If I modify in src/Zope/src/Products/PageTemplates/Expressions.py
class UnicodeAwareStringExpr(StringExpr):
def __call__(self, econtext):
vvals = []
if isinstance(self._expr, text_type):
# coerce values through the Unicode Conflict Resolver
evaluate = econtext.evaluateText
else:
evaluate = econtext.evaluate
for var in self._vars:
v = evaluate(var)
vvals.append(v)
return self._expr % tuple(vvals)by (may not be the cleanest):
class UnicodeAwareStringExpr(StringExpr):
def __call__(self, econtext):
vvals = []
if isinstance(self._expr, text_type):
# coerce values through the Unicode Conflict Resolver
evaluate = econtext.evaluateText
else:
evaluate = econtext.evaluate
for var in self._vars:
v = evaluate(var)
if isinstance(self._expr, text_type) and isinstance(v, binary_type):
v = v.decode('utf-8')
vvals.append(v)
return self._expr % tuple(vvals)this fixes the issue.
vincentfretin
commented
4 years ago The description here comes from ZCatalog brain Description metadata. As far as I remember, the description has always been stored in utf-8 in ZCatalog on Plone.
d-maurer
commented
4 years ago A minimal example is:
from Products.PageTemplates.PageTemplate import PageTemplate
from Products.PageTemplates.tests.util import useChameleonEngine as u; u()
t=PageTemplate()
ts="""<div tal:define="u python:u'äöü'; b python:'ÄÖÜ'; c string:$u $b" tal:content="c"/>"""
t.write(ts)
t()The problem with a fix is that Zope unlike Plone wants to support non-utf-8 charsets. Therefore, I cannot assume that utf-8 is the correct charset for the "bytes -> text" conversion.
The comment coerce values through the Unicode Conflict Resolver indicates an expectation that evaluateText does something intelligent with unicode BUT in fact, there is no such intelligence (at least not here). I will look for something like "Unicode Conflict Resolver" and try to apply it at this place.
vincentfretin
commented
4 years ago Even a simpler example produce the error:
$ bin/zopepy
from Products.PageTemplates.PageTemplate import PageTemplate
from Products.PageTemplates.tests.util import useChameleonEngine as u; u()
t=PageTemplate()
ts="""<div tal:define="b python:'ÄÖÜ'; c string:$b" tal:content="c"/>"""
t.write(ts)
t()I have found tests related to "Unicode Conflict Resolver" -- only implemented for the "old template engine" (i.e. not for chameleon) and very doubtful: it "guesses" the charset from the request's HTTP_ACCEPT_CHARSET. With this header the browser indicates what charsets it can handle; it does not tell anything what charset is used by the server internally.
I will address the problem as follows:
ICharsets representing a sequence of potentail charsetsUnicodeError in UnicodeAwareStringExpr, it will adapt econtext to ICharsets with the default ("utf-8", zpublisher_default_encoding). The charsets are tried in turn to convert individual values to unicode.vincentfretin
commented
4 years ago Ok this seems to be a good solution.
d-maurer
commented
4 years ago The problem may be elsewhere: there is local support for "Unicode Conflict Resolution" but it is wrongly not used.
dataflake
commented
4 years ago @vincentfretin Please update to the latest Zope 4.x branch code, Dieter has added a possible fix.
vincentfretin
commented
4 years ago Wonderful! This fixed the issue.
vincentfretin
commented
4 years ago FYI the Plone 5.2 tests are green with Zope branch 4.x except for a plone.restapi failing test with a hard coded port (apparently we have a random port now when the tests are run), see https://github.com/plone/buildout.coredev/pull/663 So all is good I think for you @dataflake to make the Zope 4.4.5 release.
dataflake
commented
4 years ago Thank you for the feedback!
BUG/PROBLEM REPORT (OR OTHER COMMON ISSUE)
What I did:
Create a Plone site in French to have the front-page in French with some text with accent like "Félicitations !" or create Plone Site in English and create a document which contains "Félicitations !" in the description. Now in the toolbar go to State -> Advanced... This will open http://0.0.0.0:8080/Plone/front-page/content_status_history in a modal.
What I expect to happen:
Show the modal without error
What actually happened:
What version of Plone/ Addons I am using:
Plone 5.2.2rc1 / Zope 4.4.4 / Python 2.7