search

isawnyu / pleiades-rdf

RDF from Pleiades
7 stars 2 forks source link

RDF dumps broken since before Christmas #1

Closed sgillies closed 11 years ago

sgillies commented 11 years ago

Noted by @paregorios. Looks like we might only be dumping the errata. Will investigate.

sgillies commented 11 years ago

Also noted by @steko.

sgillies commented 11 years ago

Fixed. Will watch this weekend's dump, but I expect it to be back to its full glory. I also reminded myself that since it's using our debug Plone instance, and not instance1 or instance2 (the ones serving the site), I don't need monit to avoid the ~3 hour dump window.

sgillies commented 11 years ago

We're running out of memory when making the graph.

Traceback (most recent call last): File "", line 1, in ? File "./dump-scripts/rdf.py", line 50, in ? places += grapher.place(obj) File "/home/zope/pleiades/p3-three/eggs/pleiades.rdf-0.8.1-py2.4.egg/pleiades/rdf/common.py", line 165, in place review_state='published') File "/home/zope/pleiades/p3-three/eggs/Plone-3.3.6-py2.4.egg/Products/CMFPlone/CatalogTool.py", line 480, in searchResults return ZCatalog.searchResults(self, REQUEST, _kw) File "/home/zope/pleiades/p3-three/parts/zope2/lib/python/Products/ZCatalog/ZCatalog.py", line 663, in searchResults return self._catalog.searchResults(REQUEST, used, _kw) File "/home/zope/pleiades/p3-three/parts/zope2/lib/python/Products/ZCatalog/Catalog.py", line 738, in searchResults return self.search(args, sort_index, reverse, sort_limit, _merge) File "/home/zope/pleiades/p3-three/parts/zope2/lib/python/Products/ZCatalog/Catalog.py", line 477, in search r = _apply_index(request) File "/home/zope/pleiades/p3-three/parts/zope2/lib/python/Products/PluginIndexes/common/UnIndex.py", line 398, in _apply_index r = set_func(r, set) MemoryError

sgillies commented 11 years ago

Fixed in 0.10, which goes into production tomorrow morning. Will watch this weekend's dumps.

sgillies commented 11 years ago

Results from last night's dump (0.11):

2605212 total triples (places, links, errata, authors, vocabs)

Links weren't serialized because their catalog records were stale (reporting as Places)

sgillies commented 11 years ago

Declaring this bug fixed. More little bugs to be found, certainly, but RDF is dumping.