WrongType for logo when saving site control panel migrated to 5.2.

plone / Products.CMFPlone

The core of the Plone content management system

GNU General Public License v2.0

246 stars 186 forks source link

The site logo is missing a migration from ASCII to Bytes. In Plone 5.1, the site logo is of type ASCII, in 5.2 this was changed to Bytes, see commit 11482a2d37b9ded04d517bd1e9f50e234e3b3b46.

zope.schema.ASCII inherits from NativeString, which on Python 2 is the same as Bytes. On Python 3, ASCII is the same as String. Problem with a site started at 5.1 and migrated to 5.2 Py 3, is that the portal_registry reports that plone.site_logo is an ASCII type and its value is text (native Py3 string), but the ISiteSchema expects a Bytes field. Validation fails with:

zope.schema._bootstrapinterfaces.WrongType: (b'filenameb64:...', <class 'str'>, 'value')

When upgrading from Python 5.1 to 5.2 this goes fine, as long as you stay on Python 2.7: the registry record contains ASCII and the schema expects Bytes, but this is the same.
When you start with 5.2 on Python 2.7 and migrate the database to Python 3, it is also fine: record and schema are both Bytes.
When you migrate 5.1 to 5.2 and then to Python 3, it goes wrong: the registry record contains ASCII and the schema expects Bytes, and this is no longer the same.

What I did:

Using only buildout.coredev branches.
Create a standard Plone 5.1 site.
Go to portal_registry?q=plone.site_logo: field type is ASCII.
Change the site logo in @@site-controlpanel: this works.
Upgrade to Plone 5.2 on Python 2.7. (I copied the var dir to a checkout of 5.2), including @@plone-upgrade.
Go to portal_registry?q=plone.site_logo: field type is still ASCII.
Change the site logo in @@site-controlpanel: this still works.
Migrate the buildout and database to Python 3.7, restart.
Go to portal_registry?q=plone.site_logo: field type is still ASCII.
Go to @@site-controlpanel and save without making any change: this fails.

Alternatively:

Same as above, but on Python 2.7 never change the site logo, do not even visit the site control panel.
Then in Plone 5.2 Python 3 go to @@site-controlpanel and save without making any change: this should work.
Change the site logo in @@site-controlpanel and save: this fails

What I expect to happen:

The logo is saved, and used as logo.

What actually happened:

Traceback:

Traceback (innermost last):
  Module ZPublisher.WSGIPublisher, line 167, in transaction_pubevents
  Module ZPublisher.WSGIPublisher, line 364, in publish_module
  Module ZPublisher.WSGIPublisher, line 267, in publish
  Module ZPublisher.mapply, line 85, in mapply
  Module ZPublisher.WSGIPublisher, line 68, in call_object
  Module plone.z3cform.layout, line 63, in __call__
  Module plone.z3cform.layout, line 47, in update
  Module plone.z3cform.fieldsets.extensible, line 65, in update
  Module plone.z3cform.patch, line 30, in GroupForm_update
  Module z3c.form.group, line 145, in update
  Module plone.app.z3cform.csrf, line 22, in execute
  Module z3c.form.action, line 98, in execute
  Module z3c.form.button, line 315, in __call__
  Module z3c.form.button, line 170, in __call__
  Module plone.app.registry.browser.controlpanel, line 63, in handleSave
  Module z3c.form.group, line 114, in applyChanges
  Module z3c.form.form, line 51, in applyChanges
  Module z3c.form.datamanager, line 91, in set
  Module plone.registry.recordsproxy, line 60, in __setattr__
  Module plone.registry.registry, line 51, in __setitem__
  Module plone.registry.record, line 82, in _set_value
  Module zope.schema._bootstrapfields, line 292, in validate
  Module zope.schema._field, line 227, in _validate
  Module zope.schema._bootstrapfields, line 495, in _validate
  Module zope.schema._bootstrapfields, line 338, in _validate
zope.schema._bootstrapinterfaces.WrongType: (b'filenameb64:S2Vlb3BfRXhwbG9yaW5nLnBuZw==;datab64:iV...
...ORK5CYII=', <class 'str'>, 'value')

What version of Plone/ Addons I am using:

First: Plone 5.1, Python 2.7
Then: Plone 5.2 Python 2.7
Then: Plone 5.2 Python 3.7

Solution:

I have not tried solutions yet, but I can imagine:

An upgrade step in plone.app.upgrade to change the ASCII record to a Bytes record. In Python 2.7 this should be easy. It may be tricker when this is already Python 3.
bin/zodbupdate could maybe handle this when migrating to Py 3, but I guess not. Also, this should still be fixed even when the database has already been migrated to Py 3.

I do not expect that all ASCII records should be turned into Bytes. It seems this should be determined on a field-by-field basis. But I do wonder if a fix could be made more general: go through all registry records and check if the record type matches the interface.

Long comment. Summary is at the end.

The site for which I originally reported this problem, now has a related problem that is caused by my PR. After the upgrade, the site logo disappears.

After some tests, I see that this is specific for this site, because I copied the code of my PR to a custom package and ran it, and now it goes wrong when I update to Plone 5.2.3. But I see ways where others may get the same problem. Also: editing the logo via the portal_registry no longer works.

Let's see if I can reproduce this in core Plone.

In coredev 5.1 create a Plone Site and change the logo.
Copy the database to coredev 5.2 on Python 2 and run the plone-upgrade. This includes the upgrade step from my PR.
The changed logo still appears.
The site-controlpanel looks fine, and saving this without changes works fine too.
But go to portal_registry, lookup the logo, and try to edit it: the form shows no file.
Change the logo in site-controlpanel: works fine, but in portal_registry you still cannot edit it.
Uploading a new logo via portal_registry works, but leads to an ValueError in plone.formwidget.namedfile while rendering the logo. You also then get the same ValueError when you go to the site-controlpanel, so you can no longer fix it. So in my test I undo this.
Go to portal_setup, Upgrades, profile Products.CMFPlone:plone, show old upgrades, run "Migrate site logo from native string to bytes" again. No problem (this will be different in Python 3).

So 5.1 to 5.2 on Python 2 mostly works. The site logo is okay, as long as you only edit it in the site-controlpanel, not in portal_registry.

Now we try Python 3:

Copy the 5.2-py2 database to coredev 5.2 on Python 3.
Run ./bin/zodbupdate --convert-py3 --file=var/filestorage/Data.fs --encoding utf8 and start Plone.
site-controlpanel is still okay at this point. The custom logo is visible, and you can save the form with or without an updated logo.
portal_registry is still a no-go: the edit form for the logo shows no file, and uploading a file will cause the same ValueError as above. So don't do this.
Go to portal_setup, Upgrades, profile Products.CMFPlone:plone, show old upgrades, run "Migrate site logo from native string to bytes" again. Now the logo field is empty. This is basically what happened on my site: I had copied the upgrade step code to a site specific package, so in the end it ran twice.
The image is gone after this line.

For more fun, I notice an extra problem, where my PR is innocent:

In coredev 5.2 Python 3 create a fresh Plone Site.
In site-controlpanel update the logo.
In portal_registry open the edit form for the logo: this again shows no file.
Save this simple popup edit form without changes: the popup does not close. In the instance log you get: zope.schema._bootstrapinterfaces.WrongType: (<NOT_CHANGED>, <class 'bytes'>, 'value').
Instead save an actual file. The value is stored as filenameb64:....
Go to the site-controlpanel and you see a traceback for Module plone.formwidget.namedfile.converter, line 77, in b64decode_file: ValueError: not enough values to unpack (expected 2, got 1).
Trying the same in Python 2, I instead see ValueError: too many values to unpack. This is either a difference in Python 2 and 3, or because I first tried a .svg and later a .png, which gives a value starting with PNG IHDR.

Summary:

The popup edit form for plone.site_logo in portal_registry is broken.
In a site migrated from 5.1 to 5.2 the logo in the site-controlpanel works.
But when you run the upgrade step from my PR a second time in Python 3, the logo is removed.
The same would be true for any future upgrade that registers the ISiteSchema interface again when a field is added.
Root cause is that the field is changed to Bytes, but the value is still a string, despite my efforts in the PR.
The upgrade step should be fixed (I can try), but a quick bugfix release of plone.app.upgrade does not seem necessary.

plone / Products.CMFPlone