ZEXP loses initids on export/import

agitator commented 5 years ago

I know export/import of zexp is unsupported. This issue should at least be a warning of it's limitations...

I got a contenttype that adds language independent files/images from the assets folder via relations. When I export/download a zexp from the live server and import it into a local instance, existing files and images lose their intid information.

After local import of the zexp shows as:

existing relations are still valid
new relations to existing (uploaded weeks ago) objects end up in an error like this

2019-07-12 10:35:24,593 ERROR   [Zope.SiteErrorLog:18][waitress] 1562920524.5926710.6317100839545621 http://localhost:11700/Plone/de/asdfasfd/@@edit
Traceback (innermost last):
  Module ZPublisher.WSGIPublisher, line 155, in transaction_pubevents
  Module ZPublisher.WSGIPublisher, line 337, in publish_module
  Module ZPublisher.WSGIPublisher, line 255, in publish
  Module ZPublisher.mapply, line 85, in mapply
  Module Products.PDBDebugMode.wsgi_runcall, line 60, in pdb_runcall
  Module plone.z3cform.layout, line 63, in __call__
  Module plone.z3cform.layout, line 47, in update
  Module plone.dexterity.browser.edit, line 58, in update
  Module plone.z3cform.fieldsets.extensible, line 65, in update
  Module plone.z3cform.patch, line 30, in GroupForm_update
  Module z3c.form.group, line 145, in update
  Module plone.app.z3cform.csrf, line 22, in execute
  Module z3c.form.action, line 98, in execute
  Module z3c.form.button, line 315, in __call__
  Module z3c.form.button, line 170, in __call__
  Module plone.dexterity.browser.edit, line 30, in handleApply
  Module z3c.form.group, line 114, in applyChanges
  Module z3c.form.form, line 51, in applyChanges
  Module plone.app.relationfield.widget, line 141, in set
  Module five.intid.intid, line 41, in getId
  Module zope.intid, line 108, in getId
zope.intid.interfaces.IntIdMissingError: <File at /Plone/de/Assets/testbilder/docgo-net-10-contracts-for-your-next-agile-software-project.pdf>
[20] > /Users/peter/workspace/gbd/eggs/zope.intid-4.3.0-py3.7.egg/zope/intid/__init__.py(108)getId()
-> raise IntIdMissingError(ob)

newly uploaded files/images can be referenced without a problem

The Plone site was exported as a whole therefore I would expect the intid utility to be exported as well. Anyone an idea?

agitator commented 5 years ago

probably not related with https://github.com/plone/Products.CMFPlone/issues/2866 in any way?

mauritsvanrees commented 5 years ago

I guess the zexp will only contain objects that are selectable in the ZMI, so no utilities. Or while importing a complete Plone Site, no local site manager is set up yet for this new Plone Site, so no local utilities can be registered during import. Seems hard to fix.

agitator commented 5 years ago

Besides the mentioned issue, I can't remember any issue as long as do Plone ;-) Furthermore a working zexp export/import always proofed to me, that I have a clean site with no dead utilities or addons that were no longer available.

cekk commented 4 years ago

I tried today exporting/importing a folder with two contents (one relates the other one) and on the import, the relation has gone.

I (probably by mistake) used zexp a lot in the past but i didn't checked relations status or if imported contents are correctly indexed into intids catalog. I think that we should write a HUGE warning in zexp page.

Is zexp really unspupported? Are there any other ways to do a similar thing in Plone? From time to time i need to recover some data from an old backup (mostly because some user delete some useful folders) and import into real db.

mauritsvanrees commented 4 years ago

I did create a script to register intids through the whole site. That may help avoid tracebacks. I think the objects themselves still have an attribute with their original intid, so no new intids are created this way. They are simply registered in the intid utility of the new site.

I guess plone.app.intid or some other package could register an event handler for this. There is probably a similar one already, but the zexp import case is missed.

I have a customer who is active in several countries. For each country we have a Plone Site in a shared ZODB, so one Zope with multiple Plones. Say we have a Dutch (nl) site and want to create a new Belgium (be) site based on it. IIRC, we do it like this:

Make a zexp of nl.
In the ZMI rename nl to be.
Import the original nl zexp.
After this, running the register_intids script from above may help. And do a clear and rebuild of the portal catalog, in the be site, or maybe both sites.

That has worked fine. But I can imagine that not everything completely works. At least for an export/import of a part of a site: if some information about the objects is not stored on the objects itself, but only in a central place, then this information will be lost.

agitator commented 3 years ago

running @@updateLinkIntegrityInformation also did help with some intid issues after a zexp import

mauritsvanrees commented 3 years ago

I did not know about the updateLinkIntegrityInformation view. Thanks for sharing. Looks like it should help yes, although it also looks like when searching for links it only finds 'implicit' relations in a rich text field, and it may lose explicit relations from a RelationList or RelationChoice, so those may get lost. But I have not tried this, so I could be wrong.

cc @fredvd

plone / Products.CMFPlone

ZEXP loses initids on export/import #2916