Global navigation broken after check-in of dexterity folder

[mauritsvanrees]

Copied from this PR comment:

I tried it just now in Plone 5.2 Python 3 with your code. I do see a problem.

• I created a folder with one page and one link.
• I checked out the folder, changed the title and added one link, then checked in.
• Folder seems fine, contents as well: the page and two links.
• But main navigation is broken. The page shows: "error while rendering plone.global_sections"

Traceback

Traceback (most recent call last):
  File "/Users/maurits/shared-eggs/cp38/plone.app.viewletmanager-3.1.1-py3.8.egg/plone/app/viewletmanager/manager.py", line 110, in render
    html.append(viewlet.render())
  File "/Users/maurits/shared-eggs/cp38/plone.app.layout-3.4.6-py3.8.egg/plone/app/layout/viewlets/common.py", line 79, in render
    return self.index()
  File "/Users/maurits/shared-eggs/cp38/Zope-4.5.5-py3.8.egg/Products/Five/browser/pagetemplatefile.py", line 126, in __call__
    return self.__func__(__self__, *args, **kw)
  File "/Users/maurits/shared-eggs/cp38/Zope-4.5.5-py3.8.egg/Products/Five/browser/pagetemplatefile.py", line 58, in __call__
    s = self.pt_render(
  File "/Users/maurits/shared-eggs/cp38/zope.pagetemplate-4.5.0-py3.8.egg/zope/pagetemplate/pagetemplate.py", line 133, in pt_render
    return self._v_program(
  File "/Users/maurits/shared-eggs/cp38/Zope-4.5.5-py3.8.egg/Products/PageTemplates/engine.py", line 378, in __call__
    return template.render(**kwargs)
  File "/Users/maurits/shared-eggs/cp38/z3c.pt-3.3.0-py3.8.egg/z3c/pt/pagetemplate.py", line 176, in render
    return base_renderer(**context)
  File "/Users/maurits/shared-eggs/cp38/Chameleon-3.9.0-py3.8.egg/chameleon/zpt/template.py", line 302, in render
    return super(PageTemplate, self).render(**_kw)
  File "/Users/maurits/shared-eggs/cp38/Chameleon-3.9.0-py3.8.egg/chameleon/template.py", line 214, in render
    raise_with_traceback(exc, tb)
  File "/Users/maurits/shared-eggs/cp38/Chameleon-3.9.0-py3.8.egg/chameleon/utils.py", line 53, in raise_with_traceback
    raise exc
  File "/Users/maurits/shared-eggs/cp38/Chameleon-3.9.0-py3.8.egg/chameleon/template.py", line 192, in render
    self._render(stream, econtext, rcontext)
  File "/Users/maurits/community/plone-coredev/py3/var/cache/b74b3cb030d373673fef3a9ecd8fe459.py", line 224, in render
    __cache_4716882960 = _static_4657941232('python', 'view.render_globalnav()', econtext=econtext)(_static_4657941040(econtext, __zt_tmp))
  File "/Users/maurits/shared-eggs/cp38/zope.tales-5.1-py3.8.egg/zope/tales/pythonexpr.py", line 73, in __call__
    return eval(self._code, vars)
  File "<string>", line 1, in <module>
  File "/Users/maurits/shared-eggs/cp38/plone.app.layout-3.4.6-py3.8.egg/plone/app/layout/viewlets/common.py", line 390, in render_globalnav
    return self.build_tree(self.navtree_path)
  File "/Users/maurits/shared-eggs/cp38/plone.app.layout-3.4.6-py3.8.egg/plone/app/layout/viewlets/common.py", line 382, in build_tree
    for item in self.navtree.get(path, []):
  File "/Users/maurits/shared-eggs/cp38/plone.memoize-2.1.0-py3.8.egg/plone/memoize/view.py", line 59, in memogetter
    cache[key] = func(*args, **kwargs)
  File "/Users/maurits/shared-eggs/cp38/plone.app.layout-3.4.6-py3.8.egg/plone/app/layout/viewlets/common.py", line 323, in navtree
    brains = portal_catalog.searchResults(**query)
  File "/Users/maurits/community/plone-coredev/py3/src/Products.CMFPlone/Products/CMFPlone/CatalogTool.py", line 463, in searchResults
    return ZCatalog.searchResults(self, query, **kw)
  File "/Users/maurits/shared-eggs/cp38/Products.ZCatalog-5.2-py3.8.egg/Products/ZCatalog/ZCatalog.py", line 625, in searchResults
    return self._catalog.searchResults(query, **kw)
  File "/Users/maurits/shared-eggs/cp38/Products.ZCatalog-5.2-py3.8.egg/Products/ZCatalog/Catalog.py", line 1091, in searchResults
    return self.search(query, sort_indexes, reverse, sort_limit, _merge)
  File "/Users/maurits/shared-eggs/cp38/Products.ZCatalog-5.2-py3.8.egg/Products/ZCatalog/Catalog.py", line 702, in search
    result = self.sortResults(
  File "/Users/maurits/shared-eggs/cp38/Products.ZCatalog-5.2-py3.8.egg/Products/ZCatalog/Catalog.py", line 1007, in sortResults
    actual_result_count, length, result = sort_func(
  File "/Users/maurits/shared-eggs/cp38/Products.ZCatalog-5.2-py3.8.egg/Products/ZCatalog/Catalog.py", line 778, in _sort_iterate_resultset
    index_key_map = sort_index.documentToKeyMap()
  File "/Users/maurits/community/plone-coredev/py3/src/plone.folder/src/plone/folder/nogopip.py", line 121, in documentToKeyMap
    pos[rid] = container.getObjectPosition(id)
  File "/Users/maurits/community/plone-coredev/py3/src/plone.folder/src/plone/folder/ordered.py", line 110, in getObjectPosition
    return self.getOrdering().getObjectPosition(id)
  File "/Users/maurits/community/plone-coredev/py3/src/plone.folder/src/plone/folder/default.py", line 158, in getObjectPosition
    raise ValueError('No object with id "{0:s}" exists in "{1:s}".'.format(
ValueError: No object with id "pagina-1" exists in "/Plone2/map".

 - Expression: "python:view.render_globalnav()"
 - Filename:   ... ut-3.4.6-py3.8.egg/plone/app/layout/viewlets/sections.pt
 - Location:   (line 22: col 42)
 - Source:     ... eplace="structure python:view.render_globalnav()" />
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 - Arguments:  template: <Products.Five.browser.pagetemplatefile.ViewPageTemplateFile object at 0x10f68cc10>
               options: {}
               args: ()
               nothing: None
               modules: <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter object at 0x10c5fcdc0>
               request: <WSGIRequest, URL=http://localhost:8080/Plone2/front-page/document_view>
               view: <Products.Five.viewlet.metaconfigure.GlobalSectionsViewlet object at 0x112f62730>
               context: <Document at /Plone2/front-page>
               views: <Products.Five.browser.pagetemplatefile.ViewMapper object at 0x112f98550>
               here: <Document at /Plone2/front-page>
               container: <Document at /Plone2/front-page>
               root: <Application at >
               traverse_subpath: []
               user: <PropertiedUser 'admin'>
               default: <DEFAULT>
               repeat: <Products.PageTemplates.engine.RepeatDictWrapper object at 0x112fac900>
               loop: {}
               target_language: None
               translate: <function BaseTemplate.render.<locals>.translate at 0x112fa90d0>
               attrs: {}
               portal_tabs: [{'id': 'index_html', 'category': 'portal_tabs', 'title': 'Home', 'description': '', 'url': <Products.CMFCore.Expression.Expression object at 0x112768f90 oid 0x3300 in <Connection at 10e74caf0>>, 'link_target': None, 'icon': '', 'available': True, 'visible': True, 'allowed': True, 'name': 'Home'}, {'name': 'Nieuws', 'id': 'news', 'url': 'http://localhost:8080/Plone2/news', 'description': 'Nieuwsberichten op deze website.', 'review_state': 'published'}, {'name': 'Agenda', 'id': 'events', 'url': 'http://localhost:8080/Plone2/events', 'description': 'Agenda-items op deze website.', 'review_state': 'published'}, {'name': 'Gebruikers', 'id': 'Members', 'url': 'http://localhost:8080/Plone2/Members', 'description': 'Map voor de persoonlijke mappen van gebruikers.', 'review_state': 'private'}, {'name': 'Underscore', 'id': 'underscore-1', 'url': 'http://localhost:8080/Plone2/underscore-1', 'description': '', 'review_state': 'private'}, {'name': 'Plain text', 'id': 'all-plone-releases.txt', 'url': 'http://localhost:8080/Plone2/all-plone-releases.txt/view', 'description': '', 'review_state': Missing.Value}, {'name': 'file.pdf', 'id': 'file.pdf', 'url': 'http://localhost:8080/Plone2/file.pdf/view', 'description': '', 'review_state': Missing.Value}, {'name': 'whisky-as-we-get-it.jpg', 'id': 'whisky-as-we-get-it.jpg', 'url': 'http://localhost:8080/Plone2/whisky-as-we-get-it.jpg/view', 'description': '', 'review_state': Missing.Value}, {'name': '<script>console.error("hello from folder title")</script>', 'id': 'folder', 'url': 'http://localhost:8080/Plone2/folder', 'description': '<script>console.error("hello from folder description")</script>', 'review_state': 'published'}, {'name': 'Map 2', 'id': 'map', 'url': 'http://localhost:8080/Plone2/map', 'description': 'Testbeschrijving 2.', 'review_state': 'published'}]

A clear and rebuild of the catalog does not help.

[mauritsvanrees]

A quick pdb session:

[49] > /Users/maurits/community/plone-coredev/py3/src/plone.folder/src/plone/folder/default.py(160)getObjectPosition()
-> raise ValueError('No object with id "{0:s}" exists in "{1:s}".'.format(
(Pdb++) l
155             pos = self._pos()
156             if obj_id in pos:
157                 return pos[obj_id]
158             import pdb; pdb.set_trace()
159     
160  ->         raise ValueError('No object with id "{0:s}" exists in "{1:s}".'.format(
161                 obj_id, '/'.join(self.context.getPhysicalPath())))
162     
163         def idsInOrder(self):
164             """ see interfaces.py """
165             return list(self._order())
(Pdb++) self._pos()
<BTrees.OIBTree.OIBTree object at 0x11847e440 oid 0x38a8 in <Connection at 115187b50>>
(Pdb++) list(self._pos())
['link-2']
(Pdb++) self.context.objectIds()
['pagina-1', 'link-1', 'link-2']

The list of object ids is what I expect. The _pos (ordering) only contains the new link object I added. So the ordering seems to get overwritten on check-in.

[sneridagh]

I do remove the annotation on check-in: https://github.com/plone/plone.app.iterate/blob/656957d439f53d9d6f05668b964fe670a71cbaac/plone/app/iterate/dexterity/copier.py#L256 it seems that I missed to remove the pos one, but (right now testing it) and it breaks also...

[sneridagh]

No, it doesn't work either...

[sneridagh]

Breaks even without copying any annotation??? I don't get it...

[sneridagh]

well... ONLY moving the objects programatically causes that.

https://github.com/plone/plone.app.iterate/blob/656957d439f53d9d6f05668b964fe670a71cbaac/plone/app/iterate/dexterity/copier.py#L248

@mauritsvanrees @jensens I am missing something?

[mauritsvanrees]

Before cutting those objects, I get this:

(Pdb++) IAnnotations(self.context)["plone.folder.ordered.order"]
['link-2']
(Pdb++) IAnnotations(self.context)["plone.folder.ordered.pos"]
<BTrees.OIBTree.OIBTree object at 0x10dc5de40 oid 0x3c08 in <Connection at 10d8bd280>>
(Pdb++) list(IAnnotations(self.context)["plone.folder.ordered.pos"])
['link-1', 'link-2', 'page-1']

So the working copy has more items in pos than in order. After pasting, it becomes this:

(Pdb++) list(IAnnotations(self.context)["plone.folder.ordered.pos"])
['link-2']
(Pdb++) IAnnotations(self.context)["plone.folder.ordered.order"]
[]

So order is empty then, but pos still has content. A bit surprising, but I guess the plone.folder code gets confused when the two are not in sync. I don't really know why we have two of these annotations that seem very similar...

I thought this would help, but it does not:

        if "plone.folder.ordered.order" in wc_annotations:
            wc_annotations.pop("plone.folder.ordered.order")
        if "plone.folder.ordered.pos" in wc_annotations:
            wc_annotations.pop("plone.folder.ordered.pos")

Maybe it helps if we do not set the pos annotation when creating the initial check-out? We only remove the order now, not the pos.

[mauritsvanrees]

That seems to help. Then pos and order only contain the new object id before cutting. And after pasting, both are empty. So: in all places where we pop the order annotation, we should pop the pos annotation. Can you try that?

[sneridagh]

I will!

[sneridagh]

@mauritsvanrees works! Phew! Thanks for the help! I will push a fix!