Expose search engine to external tools

[BjarniRunar]

This issue tracks the milestones defined in the "Mailpile Search Integration" Project which is supported by a grant from NLNet and NGI0 Discovery.

• [x] Port over / re-implement a working search engine in Python 3
• [x] Enhance the search engine to allow partitioning / compartmentalization of result sets
• [x] Give the user tools to selectively grant access to one or more partitions of results
• [x] Expose the search engine using a documented web-based API
• [x] Describe how to configure SearX for use with moggie, as a proof of concept
• [ ] Review and update documentation when #3 is done

[BjarniRunar]

The search engine and partitioning work is (modulo bugs and future refactoring) complete.

Some notable milestones along the way:

• bf91f625b47823962ec480532792d8b65128525f - Create numpy-based IntSet for tags and posting lists
• 444eecc2da90dfab43226f6a649eda26f88ef46f - This might be a search engine
• 68db14925e79113b2583cff13e253be547c9a531 - 1st draft of working partial word search
• 7a89cc17889c36b20d6a91daac32b7fbce7736e5 - Add support for All and subtraction in search
• a049a2d4b4fbc775ffa1e96faa895045bd1ef993 - Parse text into a search-op tree
• f546aa713c14e99887df20a82347f0321f2aee34 - WIP: Search worker
• 15fbc909081e3f6ce3d0af548032cc366295a8d3 - Move search engine logic to engine.py
• 355acab199f49732795efe62d9df00912cb831ae - Add namespace compartmentalisation support to the search engine
• bd3d6472a6b9c8cb477b4d7454e70b3ec33eb9e6 - Milestone: Adding to index, and searching for all:mail works!
• bd8ee3a5b91747a4af8086e29f6a2570105e0a4f - Importing mail and searching works
• 64439ab72f32870bcb1f664a13954343b3322861 - Search: Progress on tags, add short-word-pair binding
• b57575fadbefaae65412375c46abb93d76020a14 - Search engine enhancements, tagging progress
• a6309057734d207af601065c5e3545d5037fb4e7 - Fix some tag namespacing and search scoping buglets

[BjarniRunar]

There are now tools and docs explaining how to grant restricted access to the moggie search engine:

• https://github.com/BjarniRunar/moggie/blob/master/docs/search-integration.md

More relevant commits:

• 47ee0b2ce2a39aa8ffc717d2377e08afe5bd842f - Improve how contexts scope searches
• 2ee4f2f5afc39f3bfcefd0a644a04327944ed389 - Make app load port and kite details from config
• 55f5cdfcd041e13e05f1061ef504db2f16f0b9b3 - Added ConfigSet/ConfigGet API calls, moggie grant now works
• d3734267f3d87c6d015370d29277358ce1ef02bd - Added list edits to ConfigSet, moggie context now works
• 6ce8ed6ddc562763843d2a4605c2b5c5edd90847 - Select default context per access grant...
• 8594f5fb2294f7a7777a64a472ad122ef559cade - Created docker-based blackbox test script for context/grant...
• 2fde3ef618e683c89c5b4c8c9e3853af05bf8fb2 - Create docs/search-integration.md

[BjarniRunar]

There is now enough of a web interface exposed to allow 3rd party tools to search, display results, and display individual messages.

The search integration docs have been updated with a section aimed at developers: https://github.com/BjarniRunar/moggie/blob/master/docs/search-integration.md#6-notes-for-developers

Relevant commits:

• 8ec813150911dcba6afc55f6e156ba98320a301e - moggie search: Add signatures to result message IDs
• 48bbd69f71da17eb3f763ad36359a2cddda1bfed - moggie show mostly works now
• a25782c10eff76a024b831498930092272c75077 - Documented search/show API for integration

[BjarniRunar]

The proof-of-concept SearX plugin is here: https://github.com/BjarniRunar/moggie/blob/master/contrib/searx/moggie.py

It works! However there does seem to be a bug in moggie's web server causing it to fail when Firefox connects to it directly (when a user clicks a search result in SearX). Reloading the blank page seems to suffice? I'll have to track that down, but it's not directly related to this.

[BjarniRunar]

As a first draft, this is complete. Leaving open, because the API should be reviewed and documentation improved further once all the notmuch #3 work is complete.