search

isawnyu / pleiades-gazetteer

This repository provides a home for tickets and other planning documents for the Pleiades gazetteer of ancient places. Code is kept in multiple other repositories.
https://pleiades.stoa.org
11 stars 0 forks source link

Upgrade to Plone 4.3.10 #215

Closed witekdev closed 7 years ago

witekdev commented 8 years ago

Besides the hotfix deployed yesterday https://github.com/isawnyu/pleiades-gazetteer/issues/212. In order for Pleiades to take advantage of the latest minor security fixes: https://plone.org/news/2016/minor-plone-security-fixes. It is best for us to upgrade to the latest Plone version ASAP.

paregorios commented 8 years ago

I am setting priority to high for this issue. I'd like to see the upgrade and any associated migration etc. tested on staging first (after copying up-to-date copies of filestorage and blobstorage) from production. Then we'll have some idea of what we're up against in terms of a production upgrade.

skleinfeldt commented 8 years ago

@paregorios OK, @witekdev will work on this.

witekdev commented 8 years ago

Production data has been copied to the staging site (blobs, filestorage and vaytrou data) Plone 4.3.10 has been deployed to staging, but the staging site is currently offline as I'm getting an error which I suspect is due the python that is on staging, and the instances keep restarting (so I shut them down for now)

/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py:706: UserWarning: The set_schema option of the <require /> directive is not supported in Zope 2. Ignored for <class 'pleiades.workspace.content.workspace.Workspace'>
  actions = self.handler(context, **args)
/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py:706: UserWarning: The set_schema option of the <require /> directive is not supported in Zope 2. Ignored for <class 'pleiades.workspace.content.workspace.WorkspaceCollection'>
  actions = self.handler(context, **args)
/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py:706: UserWarning: The set_schema option of the <require /> directive is not supported in Zope 2. Ignored for <class 'pleiades.workspace.content.folder.WorkspaceFolder'>
  actions = self.handler(context, **args)
Traceback (most recent call last):
  File "/srv/python27-apps/pleiades4/parts/instance2/bin/interpreter", line 339, in <module>
    execfile(__file__)
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/Zope2/Startup/run.py", line 76, in <module>
    run()
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/Zope2/Startup/run.py", line 22, in run
    starter.prepare()
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/Zope2/Startup/__init__.py", line 92, in prepare
    self.startZope()
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/Zope2/Startup/__init__.py", line 268, in startZope
    Zope2.startup()
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/Zope2/__init__.py", line 47, in startup
    _startup()
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/Zope2/App/startup.py", line 118, in startup
    load_zcml()
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/Zope2/App/startup.py", line 52, in load_zcml
    load_site()
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/Zope2/App/zcml.py", line 46, in load_site
    _context = xmlconfig.file(site_zcml)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 649, in file
    include(context, name, package)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include

      processxmlfile(f, context)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/OFS/metaconfigure.py", line 46, in loadProducts
    xmlconfig.include(_context, zcml, package=product)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()

    File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/srv/python27-apps/pleiades4/eggs/Zope2-2.13.24-py2.7.egg/OFS/metaconfigure.py", line 46, in loadProducts
    xmlconfig.include(_context, zcml, package=product)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/srv/python27-apps/pleiades4/eggs/z3c.autoinclude-0.3.5-py2.7.egg/z3c/autoinclude/zcml.py", line 104, in includePluginsDirective
   includeZCMLGroup(_context, info, filename)  
   File "/srv/python27-apps/pleiades4/eggs/z3c.autoinclude-0.3.5-py2.7.egg/z3c/autoinclude/zcml.py", line 30, in includeZCMLGroup
    include(_context, filename, includable_package)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/local/python/2.7.11/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 705, in finish
    args = toargs(context, *self.argdata)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 1397, in toargs
    args[str(name)] = field.fromUnicode(s)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/fields.py", line 137, in fromUnicode
    value = self.context.resolve(name)
  File "/srv/python27-apps/pleiades4/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 179, in resolve
    mod = __import__(mname, *_import_chickens)
  File "/srv/python27-apps/pleiades4/eggs/plone.namedfile-3.0.9-py2.7.egg/plone/namedfile/scaling.py", line 11, in <module>
    from plone.scale.scale import scaleImage
  File "build/bdist.linux-x86_64/egg/plone/scale/scale.py", line 2, in <module>
  File "build/bdist.linux-x86_64/egg/PIL/Image.py", line 66, in <module>
  File "build/bdist.linux-x86_64/egg/PIL/_imaging.py", line 7, in <module>
  File "build/bdist.linux-x86_64/egg/PIL/_imaging.py", line 6, in __bootstrap__
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/srv/python27-apps/pleiades4/parts/instance2/etc/site.zcml", line 16.2-16.23
    ZopeXMLConfigurationError: File "/srv/python27-apps/pleiades4/eggs/Products.CMFPlone-4.3.10-py2.7.egg/Products/CMFPlone/configure.zcml", line 98.4-102.10
    ZopeXMLConfigurationError: File "/srv/python27-apps/pleiades4/eggs/plone4.csrffixes-1.0.9-py2.7.egg/plone4/csrffixes/configure.zcml", line 46.2-53.6
    ImportError: /home/plone_daemon/.python-eggs/Pillow-3.2.0-py2.7-linux-x86_64.egg-tmp/PIL/_imaging.so: undefined symbol: PyUnicodeUCS4_AsLatin1String

It seems to me like the python that is on staging is not compiled with the _imaging module that the latest version of PIL needs.

@alecpm @cguardia @davisagli can you help out here?

davisagli commented 8 years ago

@witekdev try deleting the Pillow egg and rerun buildout. It looks like it was compiled for UCS-4 but the Python does not support it (Python can be compiled with or without support for this way of representing unicode)

witekdev commented 8 years ago

@davisagli that worked, the site is up and running. I'm getting an issue when doing a DRY run upgrade:

Dry run selected.
Starting the migration from version: 4311
Setting installed version of profile gomobile.mobile:default as unknown.
Setting installed version of profile Products.Poi:default as unknown.
Setting installed version of profile Products.AddRemoveWidget:default as unknown.
Setting installed version of profile gomobile.convergence:default as unknown.
Setting installed version of profile Products.DataGridField:default as unknown.
Setting installed version of profile Products.CMFSquidTool:default as unknown.
Setting installed version of profile plone.contentratings:default as unknown.
Ran upgrade step: Unmark installed profiles that are no longer available.
Ran upgrade step: Mark products as installed for installed uninstallable profiles
Ran upgrade step: Cleanup uninstalled products
Ran upgrade step: Cleanup the skins tool.
Removed fake kupu_library_tool
Removed kupustyles.css from portal_css.
Removed kupuplone.css from portal_css.
Removed kupudrawerstyles.css from portal_css.
Removed kupusourceedit.js from portal_javascripts.
Removed sarissa_ieemu_xpath.js from portal_javascripts.
Removed kupuhelpers.js from portal_javascripts.
Removed kupuploneui.js from portal_javascripts.
Removed kupubasetools.js from portal_javascripts.
Removed kupuloggers.js from portal_javascripts.
Removed kupuploneeditor.js from portal_javascripts.
Removed kupudrawers.js from portal_javascripts.
Removed kupueditor.js from portal_javascripts.
Removed kupunoi18n.js from portal_javascripts.
Removed kupuploneinit.js from portal_javascripts.
Removed kupucontentfilters.js from portal_javascripts.
Removed kupu control panel configlet.
Removed Kupu from available_editors.
Changed new member wysiwyg_editor to site default.
Ran upgrade step: Remove fake kupu tool and related settings and resources.
Ran upgrade step: Miscellaneous
End of upgrade path, main migration has finished.
Starting upgrade of core addons.
Done upgrading core addons.
Recatalog needed. This may take a while...
Exception was thrown while cataloging:
Traceback (most recent call last):
  File "/srv/python27-apps/pleiades4/eggs/Products.CMFPlone-4.3.10-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 295, in upgrade
    catalog.refreshCatalog(clear=1)
  File "/srv/python27-apps/pleiades4/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/ZCatalog.py", line 257, in refreshCatalog
    cat.clear()
  File "/srv/python27-apps/pleiades4/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/Catalog.py", line 105, in clear
    self.getIndex(index).clear()
  File "/srv/python27-apps/pleiades4/src/pleiades.vaytrouindex/pleiades/vaytrouindex/index.py", line 197, in clear
    response = cm.connection.clear()
  File "/srv/python27-apps/pleiades4/src/pleiades.vaytrouindex/pleiades/vaytrouindex/index.py", line 396, in clear
    raise VaytrouConnectionError(e)
VaytrouConnectionError: [Errno 111] Connection refused
Done with recatalog.
Dry run selected, transaction aborted

Thoughts?

davisagli commented 8 years ago

There is a geographic index which runs in a separate process and is communicated with via TCP. It wasn't running; I ran sudo supervisorctl start vtpleiades3 to start it, so let's see if that helps.

witekdev commented 8 years ago

thanks @davisagli, I stopped that to copy the vaytrou data over and forgot to restart it. I was clearly too zombified last night to remember. @paregorios I'm running the dry run upgrade now, will update progress here.

witekdev commented 8 years ago

I ran the upgrade steps on staging, no issues, currently running a catalog update.

witekdev commented 8 years ago

ok, maybe @alecpm @cguardia can advise here? When doing a catalog update via browser after a while I get:

Error 503 Backend fetch failed
Backend fetch failed
Guru Meditation:
XID: 820213
Varnish cache server

When ssh tunneling in directly after a longer period of time I get:

504 Gateway Time-out
The server didn't respond in time.

When that happens I also see that one of the plone instances has been restarted.

Is there a better strategy to do a catalog update on pleaides staging? And also how important is it for us to do a catalog update after a Plone upgrade?

witekdev commented 8 years ago

Perhaps I should recatalog each index separately?

witekdev commented 8 years ago

Something is not right with the re-indexing process. Even the individual re-indexing of the expires index which has only 5 entries, takes forever (at least a few minutes have gone by since I started and it was now timed out) @alecpm could you look into this please?

alecpm commented 8 years ago

It's normal for a catalog reindex (whole catalog or individual indexes) to take forever and it will almost always timeout the front end webserver/cache. The reindex needs to wake up every single object in the site and fetch data from them. As a rule, you should run those operations on the admin instance (instance1 on port 8080, you may have to ssh tunnel to access it or add your IP to the firewall list in the ansible config), which won't timeout. You can tail the instance1 log to see how the reindex is progressing, and whether it succeeds or eventually errors out. It's possible (especially on staging) that the instance could run out of RAM during the reindex and get killed by supervisor's memmon or the OS. For staging migrations and reindexes that require a lot of RAM, it sometimes helps to use supervisor to stop all the other instances and the memmon in order to free up RAM and prevent the auto-kill from happening.

Keep in mind though a front end timeout will not prevent the operation from continuing, so if it times out you should simply wait and check back later to see if it completed. Don't try to run another reindex after the timeout, because then you will have two instances performing the same long running operation and will inevitably end up with either a out of memory kill or conflict error followed by conflict retries repeating the whole super-long transaction.

davisagli commented 8 years ago

Why is the reindex necessary in this case? Is it part of the upgrade or something you're doing "just in case"?

David Glick (mobile)

On Sep 1, 2016, at 11:03 AM, Alec Mitchell notifications@github.com wrote:

It's normal for a catalog reindex (whole catalog or individual indexes) to take forever and it will almost always timeout the front end webserver/cache. The reindex needs to wake up every single object in the site and fetch data from them. As a rule, you should run those operations on the admin instance (instance1 on port 8080, you may have to ssh tunnel to access it or add your IP to the firewall list in the ansible config), which won't timeout. You can tail the instance1 log to see how the reindex is progressing, and whether it succeeds or eventually errors out. It's possible (especially on staging) that the instance could run out of RAM during the reindex and get killed by supervisor's memmon or the OS. For staging migrations and reindexes that require a lot of RAM, it sometimes helps to use supervisor to stop all the other instances and the memmon in order to free up RAM and prevent the auto-kill from happening.

Keep in mind though a front end timeout will not prevent the operation from continuing, so if it times out you should simply wait and check back later to see if it completed. Don't try to run another reindex after the timeout, because then you will have two instances performing the same long running operation and will inevitably end up with either a out of memory kill or conflict error followed by conflict retries repeating the whole super-long transaction.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

witekdev commented 8 years ago

@davisagli @alecpm as part of the plone upgrade process, there was a reindexing of the entire catalog. I don't know why, I assumed it was part of the upgrade. The entire dry run was taking forever and timing out so I took the approach of just applying the upgrade steps separately in portal_setup, and then doing the reindexing.

witekdev commented 8 years ago

@davisagli @alecpm You can see as part of my earlier comment that contains the dry run traceback, that it said:

End of upgrade path, main migration has finished.
Starting upgrade of core addons.
Done upgrading core addons.
Recatalog needed. This may take a while...

I'm not sure if that is just standard upgrade boiler plate or if in fact a recatalog was really needed?

witekdev commented 8 years ago

When comparing staging indexes against production indexes: There is only one difference: staging: geolocation VaytrouIndex 8351 production: geolocation VaytrouIndex 58704

witekdev commented 8 years ago

After recopying the vatrou data from production to staging, all the indexes in the catalog on staging are identical to those on production.

witekdev commented 8 years ago

@paregorios The staging site is now ready for you to test. Thanks! Since it is a copy of the production site data, code that was deployed to staging only such as: https://github.com/isawnyu/pleiades-gazetteer/issues/198 would now be MIA.

witekdev commented 8 years ago

@paregorios although, I'm actually not certain of that, maybe @alecpm can double check. I just blindly assumed that the changes in https://github.com/isawnyu/pleiades-gazetteer/issues/198 were to site data, but if they were to code only, they should be there on staging.

alecpm commented 8 years ago

There are migration steps that would need to be run to address the issue. Looks like everything is updated.

skleinfeldt commented 8 years ago

@alecpm just to verify: you are saying that not only is the correct code on staging and the correct data on staging, but all the needed migration steps have been run as well?

alecpm commented 7 years ago

I believe so @skleinfeldt

paregorios commented 7 years ago

So it sounds to me like staging has been running with the upgrade for a long time. I guess it's time to start planning to roll this out on production. @skleinfeldt do we need a story for this, or can we just use this ticket?

skleinfeldt commented 7 years ago

How about using this ticket @paregorios ? Although open to your suggestion.

paregorios commented 7 years ago

fine by me, but let's hold off doing anything with this until the dust settles from the https crossover

paregorios commented 7 years ago

dust is now settled from the https crossover; I assume this ticket now/still is ready to be addressed for production. Is that correct?

davisagli commented 7 years ago

Correct

witekdev commented 7 years ago

@paregorios Production has been patched finally with a lot of help from @davisagli tonight. Happy its done though :)

paregorios commented 7 years ago

thanks all!