collective / collective.solr

Solr search engine integration for Plone
https://pypi.org/project/collective.solr/
22 stars 46 forks source link

Fix index subjects with non-ascii characters. #248

Closed maethu closed 4 years ago

maethu commented 4 years ago

Without any modification this test produces the following error:

Error in test test_umlauts (collective.solr.tests.test_subjects.TestDublincoreSubjectField)
Traceback (most recent call last):
  File "/path/to/python/parts/opt/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/path/to/pkgs/collective.solr/src/collective/solr/tests/test_subjects.py", line 27, in test_umlauts
    transaction.commit()
  File "/path/to/dotfiles/.buildout/eggs/transaction-2.4.0-py2.7.egg/transaction/_manager.py", line 252, in commit
    return self.manager.commit()
  File "/path/to/dotfiles/.buildout/eggs/transaction-2.4.0-py2.7.egg/transaction/_manager.py", line 131, in commit
    return self.get().commit()
  File "/path/to/dotfiles/.buildout/eggs/transaction-2.4.0-py2.7.egg/transaction/_transaction.py", line 296, in commit
    self._callBeforeCommitHooks()
  File "/path/to/dotfiles/.buildout/eggs/transaction-2.4.0-py2.7.egg/transaction/_transaction.py", line 367, in _callBeforeCommitHooks
    hook(*args, **kws)
  File "/path/to/dotfiles/.buildout/eggs/Products.CMFCore-2.4.0-py2.7.egg/Products/CMFCore/indexing.py", line 312, in before_commit
    self.queue.process()
  File "/path/to/dotfiles/.buildout/eggs/Products.CMFCore-2.4.0-py2.7.egg/Products/CMFCore/indexing.py", line 222, in process
    util.reindex(obj, attributes, update_metadata=metadata)
  File "/path/to/pkgs/collective.solr/src/collective/solr/indexer.py", line 261, in reindex
    self.index(obj, attributes)
  File "/path/to/pkgs/collective.solr/src/collective/solr/indexer.py", line 254, in index
    adder(conn, boost_values=boost_values(obj, data), **data)
  File "/path/to/pkgs/collective.solr/src/collective/solr/indexer.py", line 122, in __call__
    conn.add(**data)
  File "/path/to/pkgs/collective.solr/src/collective/solr/solr.py", line 284, in add
    lst.append(tmpl % self.escapeVal(value))
  File "/path/to/pkgs/collective.solr/src/collective/solr/solr.py", line 185, in escapeVal
    val = six.text_type(val)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Issue: Subject method defined in https://github.com/plone/plone.dexterity/blob/af8ce7925e2e24e0d7cb335413acd618ba919637/plone/dexterity/content.py#L513 returns alist of utf-8 strings not unicode, so it's not compatible anymore with python 2.7 deployments.

maethu commented 4 years ago

@tisto Do you mind if I merge? I don't know why some builds are failing...

tisto commented 4 years ago

@maethu we need to fix the build first, otherwise we will end up with a broken build forever. :(

tisto commented 4 years ago

@maethu this is a known issue with the Plone 4.3 buildout. I know how to fix it...

maethu commented 4 years ago

@maethu this is a known issue with the Plone 4.3 buildout. I know how to fix it...

Cool, any hint would be really appreciated. I'll fix it then.