plone.app.referenceablebehavior.referenceable.IReferenceable - lower letter starting non ASCII title raises UnicodeDecodeError

[szakitibi]

BUG REPORT

First I want to apologize to report it here, but I could not create an issue on plone.app.referenceablebehavior repository, and contributing.rst then Guidelines for Contributing to Plone guided me here.

What I did:

We have some add-on packages using plone.app.referenceablebehavior.referenceable.IReferenceable so our Dexterity content types can be referenced by old 3rd party Archetype content types.

What actually happened:

When I add any Dexterity content type with enabled IReferenceable behaviour using the lower letter start then some non ASCII characters (e.g: lowerNonÄscii) pattern in it's title, I get:

Traceback (innermost last):

    Module ZPublisher.Publish, line 138, in publish
    Module ZPublisher.mapply, line 77, in mapply
    Module ZPublisher.Publish, line 48, in call_object
    Module plone.z3cform.layout, line 66, in __call__
    Module plone.z3cform.layout, line 50, in update
    Module plone.dexterity.browser.add, line 118, in update
    Module plone.z3cform.fieldsets.extensible, line 59, in update
    Module plone.z3cform.patch, line 30, in GroupForm_update
    Module z3c.form.group, line 145, in update
    Module plone.app.z3cform.csrf, line 21, in execute
    Module z3c.form.action, line 98, in execute
    Module z3c.form.button, line 315, in __call__
    Module z3c.form.button, line 170, in __call__
    Module plone.dexterity.browser.add, line 101, in handleAdd
    Module z3c.form.form, line 265, in createAndAdd
    Module plone.dexterity.browser.add, line 76, in add
    Module plone.dexterity.utils, line 185, in addContentToContainer
    Module Products.BTreeFolder2.BTreeFolder2, line 455, in _setObject
    Module zope.event, line 31, in notify
    Module zope.component.event, line 24, in dispatch
    Module zope.component._api, line 136, in subscribers
    Module zope.component.registry, line 321, in subscribers
    Module zope.interface.adapter, line 585, in subscribers
    Module zope.component.event, line 32, in objectEventNotify
    Module zope.component._api, line 136, in subscribers
    Module zope.component.registry, line 321, in subscribers
    Module zope.interface.adapter, line 585, in subscribers
    Module plone.app.referenceablebehavior.uidcatalog, line 34, in added_handler
    Module Products.Archetypes.UIDCatalog, line 226, in catalog_object
    Module Products.ZCatalog.ZCatalog, line 476, in catalog_object
    Module Products.ZCatalog.Catalog, line 353, in catalogObject
    Module Products.PluginIndexes.common.UnIndex, line 224, in index_object
    Module Products.PluginIndexes.common.UnIndex, line 251, in _index_object
    Module Products.PluginIndexes.common.UnIndex, line 199, in insertForwardIndexEntry

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128) 

Did a bit of debugging and the failing line contains self._index.get(entry, _marker), where self._index is a BTrees.OOBTree.OOBtree object, while the entry is the title <type 'str'> lowerNon\xc3\x84scii.

Error is not there if I use upper starting letter with non Ascii characters, e.g: UpperNonÄscii. Also spaces do not affect thing only the starting letter, so non Äscii gets the error while Non Äscii not.

Our packages are not involved in the error, since a dummy Dexterity type created through the web with enabled plone.app.referenceablebehavior,reference.IReferenceable behaviour fails, while without it there is no error.

What version of Plone/ Addons I am using:

We used them on a Debian GNU/Linux 9.3 demo server, with:

• Plone 4.3.15 (4317)
• CMF 2.2.10
• Zope 2.13.26
• Python 2.7.13 (default, Nov 24 2017, 17:33:09) [GCC 6.3.0 20170516]
• PIL 3.3.0 (Pillow)

And involved packages had versions:

• plone.app.referenceablebehavior- 0.7.6
• Products.Archetypes.UIDCatalog - 1.9.17
• Products.ZCatalog - 2.13.27
• ZODB3 - 3.10.7 (contains the BTrees.OOBTree.OOBtree object)

Since C seems to be involved with BTree objects, I checked it with ldd --version, says:

• ldd (Debian GLIBC 2.24-11+deb9u1) 2.24

What I expect to happen:

No UnicodeDecodeError raised.

Which actually happened when I tried to reproduce the error locally on Ubuntu 14.04, with:

• Plone 4.3.15 (4317)
• CMF 2.2.10
• Zope 2.13.26
• Python 2.7.6 (default, Nov 23 2017, 15:49:48) [GCC 4.8.4]
• PIL 3.3.0 (Pillow)

packages:

• plone.app.referenceablebehavior- 0.7.6
• Products.Archetypes.UIDCatalog - 1.9.17
• Products.ZCatalog - 2.13.27
• ZODB3 - 3.10.7

and C:

• ldd (Ubuntu EGLIBC 2.19-0ubuntu6.14) 2.19

[szakitibi]

After further testing and debugging it seems to relate to https://github.com/plonegovbr/brasil.gov.portal/issues/288. On PloneGov-BR they have encountered same traceback which they were able to fix with updating their indexer method to return utf8 encoded strings.

During our tests a pattern emerged, how we can reproduce the error from scratch and seemed like:

• it does not relate to python version and environment
• we had to create a relationship with one of our dexterity types first
• our content type did have:
  • a computed title attribute
  • a custom indexer method for title attribute returning unicode values
  • enabled IReferenceable behavior

Also following up on the failing entry of Products.PluginIndexes.comon.UnIndex.insertForwardIndexEntry method, we figured that plone.indexer.wrapper.IndexableObjectWrapper returned our custom indexer method.

Although we still don't have all the answers:

• Why did unrelated objects and types raise the error?
• Why did lower letter starting titles for dexterity types raise the error, while upper letter starting titles doesn't?

Applying the same fix as https://github.com/plonegovbr/brasil.gov.agenda/pull/73/commits/912e44c8a73c623fc64ffdf826a47560d3675b4b on our indexer method (plone.dexterity.utils.safe_utf8 value before return) seems to prevent the issue.