Serendipity machine: from input text parsing to query terms Do list

[mialondon]

• [x] Get basic input form parsing working
• [x] Get text tokens from unstructured text
• [x] Choose entity recognition service
• [x] Document entity recognition service decision
• [ ] Hook recognised entities into API search fields
• keywords - unspecified search (i.e. all fields)
• people/organisations -> OR in creator, subject, description, title (?)
• dates -> DPLA temporal. Europeana?
• spatial -> DPLA spatial. Europeana?
• [x] Randomise which candidate terms go into the search query so search results will be slightly different each time
• [x] Optimise the query terms for optimum results - enough noise/randomness but also recognisably based on the queries submitted.
• [ ] Consider whether need to tweak based on the number of terms or size of resultset (if too large or too small)
• [x] Bring Zotero data input into the code flow

[moltude]

using dbpedia for recognition services.

[mialondon]

Discovered Open Calais wasn't as useful because a) pricing and b) based on terms used in newspaper reporting don't offer the same topic coverage as DBPedia.

Scott, do we need to mention the difference between annotated and spotlight terms anywhere in the documentation, or on a future wishlist?

[briancroxall]

Can we make it possible for a different Named Entity Recognition tool to be plugged in easily to the code, so if someone wants to fork and use a paid service that that wouldn't be a problem?

If we're not already in this state, it's clearly out of scope and someplace to think for refactoring the code for future updates.

[moltude]

Added a very simplistic _get_random that returns at most 15 keyword terms to search for

[moltude]

added support for early and late date ranges.

[mialondon]

I'll try to do a code review and check the state of this before November...