Change management

[dginev]

After the port is complete, we should put in place some change management in place.

NNexus 1.0 uses an invalidation index that allows it to instantly discover and report invalidation entries, at the price of a tight integration with Noosphere and a very heavy-weight data storage (it essentially carries a redundant copy of the PlanetMath sources).

Once we have the PULL API in place, a more loosely integrated change management approach should be considered, possibly based on caching term footprints of the scanned articles.

[dginev]

I have read some more on the invalidation index that the old NNexus has. In essence, each article is being cached in the backend, as well as a full term footprint of it (which I am not 100% clear on the exact specifics of). The claim in the papers is that only "likely" future terms are being cached, i.e. no phrase of great size will be recorded. The method hinted at has to do with frequency of occurrences in the indexed corpus.

Here are my thoughts on getting something on those lines in the NNexus Reloaded codebase:

• We only care to do management of change to auto-linked articles, not to the concept-indexed articles (before those used to be the same as NNexus was handling only PlanetMath, now this distinction is important).
• For each auto-linked article, we want to maintain a cache of "possible phrases" to be invalidated against.
• We only care to do invalidation once the index has changed, i.e. once an indexing request has completed. Thus:

Basic idea: make the response to an indexing request a list of URLs that should be considered for re-linking. As we are loosely integrated, the final call whether something should be re-linked should be in Planetary and should be done on-demand.

Tricky:

(re-)caching the terms of auto-linked articles, in an efficient and scalable manner.

I am wondering whether the "chained concept hash" should be kept from the original implementation as-is, whether it should hold only the indexed terms, or also the possible/dangling terms. Actually possible and dangling terms are yet another distinction. So we need three categories of phrase-like structures:

• indexed concepts (we are sure it's a concept and we know where it's defined)
• dangling concepts (we are sure it's a concept, but we don't have a definition yet)
• possible concept (it might be a concept, but we don't really know)

In any case, we should probably advise for a cron job that re-links the entire content of a site (PlanetMath or other) on a weekly/monthly basis, so that the just-in-time invalidation isn't the only refresh mechanism. But it should be reliable enough ... Maybe it already is?

[dginev]

One important Planetary use case would be re-indexing a freshly modified article. The workflow then would be:

• Fish out all concepts defined at that URL from the backend.
• Index the concepts defined at the new content at that URL.
• Compute a diff.
• For every missing concept, perform a deletion to the concept index.
• For every new concept, perform an addition to the concept index.

A tricky thing to keep in mind are synonyms. Maybe the most straightforward approach to dealing with them is to compile them down to independent concepts of their own. That should do the trick.

[dginev]

More thoughts on the invalidation and concept indices.

• Coming from auto-linking requests, that need to detect concepts, the chained hash index for concepts is an efficient idea that I will simply adopt as-is.
• Coming from the indexing side, where (re-)indexing URL resources needs to be reflected in all concepts defined in the affected resource, having a reverse index, connecting URLs to their concepts, seems to be the efficient way to go.
• Since the same URL resource can be used for both defining concepts, as well as for an auto-link target, it seems wisest to combine all resource logic in the same table.
• I am wondering if it is more efficient to precompute all MSC category distances or compute them on-the-fly for each auto-link request. There are 5500 MSC classes (63 top level, 400 second tier, 5000 leaf nodes) or about 15 million combinations of pairs. The question is what will be faster - table lookup in a 15 million row SQL table or computing distances over the MSC graph with Perl's Graph module. I suspect I'd have to benchmark to know ...
• This reminds me, I should update the MSC classification to the 2010 release. It also reminds me we will need maps from Wikipedia and DLMF to the MSC classification, so that we standardize all indices on the MSC.
• Concepts can have multiple MSC categories, and that should be respected when disambiguating auto-linked terms.

[dginev]

Oh, regarding whether to have a 15 million entry table or compute the MSC distance on the fly - the answer is probably in the middle. Once computed on-the-fly, the distances can be cached in the database. This way we don't create bloat between categories that will rarely be compared, we keep the table small and we still cache the most common comparisons.

[dginev]

This ticket ended up very close in essence to #16 , they should be co-requisites.

[dginev]

The plan of attack is now clear, and the invalidation (change management) of already indexed concepts is operational (see t/05 for an example). The rest is scheduled for the June release, see #27 .

Closing here.