Refactoring of openresearch datasource

[tholzheim]

Current state of affairs can be found at the branch orRefactor

Changes

• SMWEntity and SMWEntityList are obsolete and are removed
• SMW interactions capseled in OrSMW
  • api access to extract all entity records (for Event and Series)
  • Extraction from WikiMarkup
  • directly from the wiki, or
  • from backup files
• OREventManager and OREventSeriesManager usage simplified
  • configuration over new wikiId and via parameter instead of abstract configure function
• Removed api to change entites in the wiki
  • Since cc is primarily for accessing the data and not changing it modifying the data should be externalized
  • ormigrate i.e. provides with OrWikiPage an easy to use api to modify entities.

openreserach.py:

• https://github.com/WolfgangFahl/ConferenceCorpus/blob/bb995acb53794671208f6129d252be698a405098/corpus/datasources/openresearch.py#L699
• OREventManager and OREventSeriesManager switched from SMWEntityList to OrSMW
• OREvent and OREventSeries: removed binding to SMWEntity

Tests:

• https://github.com/WolfgangFahl/ConferenceCorpus/blob/bb995acb53794671208f6129d252be698a405098/tests/testOpenResearch.py#L16
• https://github.com/WolfgangFahl/ConferenceCorpus/blob/bb995acb53794671208f6129d252be698a405098/tests/testOpenResearch.py#L73
• https://github.com/WolfgangFahl/ConferenceCorpus/blob/bb995acb53794671208f6129d252be698a405098/tests/testOpenResearch.py#L140
• https://github.com/WolfgangFahl/ConferenceCorpus/blob/bb995acb53794671208f6129d252be698a405098/tests/testOpenResearch.py#L207

Open Questions

• OpenResearch now has three ways to initalize its entity managers api, WikiMarkup, WikiFiles
  • There already was the discussion that we should switch to only one openresearch datasource
  • Which one to use or should WikiMarkup just replace the WikiFiles method?

ccUpdate

$ ccUpdate --updateSource "orclone-backup"
Starting update of conference corpus database from orclone-backup cache ...
configureCorpusLookup callback called
Starting loading OPENRESEARCH (orclone-wikiMarkup) ...
(1/9984) Extracting Event record from AAAI HBM 2009 ... ✅
(2/9984) Extracting Event record from AASN 2008 ... ✅
(3/9984) Extracting Event record from AASNET 2008 ... ✅
(4/9984) Extracting Event record from ACE 2008 ... ✅
(5/9984) Extracting Event record from ACM SAC MCA 2009 ... ✅
(6/9984) Extracting Event record from ACM SAC OOPS 2010 ... ✅
(7/9984) Extracting Event record from ACM STC 2008 ... ✅
(8/9984) Extracting Event record from ACML 2009 ... ✅

...

(1125/1128) Extracting Event series record from UIST ... ✅
(1126/1128) Extracting Event series record from UbiComp ... ✅
(1127/1128) Extracting Event series record from VISAPP ... ✅
(1128/1128) Extracting Event series record from WPNC ... ✅
loading OPENRESEARCH (orclone-wikiMarkup) took 1371.8 s

:information_source: lookupId orclone-backup and or-backup perform the entity fetching with the WikiMarkup method to provide the latest data. The default backup path of the WikiFile (fromWikiFileManager) method is ~/or/wikibackup/<wikiId> which often resulted in loading an old backup. To avoid this the or-backup datasource should be loaded with the WikiMarkup method since it basically is the same procedure but the actuality of the records is ensured.

Downstream Problems

• ormigrate uses the SMWEntity to generate the fixed wikiFiles
  • can be switched to since the Entites are encapsulated in an EntityRating object wich can provide the functionality if needed

:information_source: The WikiMarkup method is intended to replace the WikiFile data fetching method to always provide the latest data. But the WikiFile method sould be keept if the data should be loaded from a backup.

ToDos

• [x] Add wikiMarkup property to the WikiMarkup extraction method
  • some fixers need this to access to the freetext
  • allows to rebuild the complete page

[tholzheim]

Added option to include wikiMarkup in getWikiSonFromPage