Term accelerated searches using bloomfilter

[elliVM]

Allow search string pattern to be accelerated without using a global bloomfilter

[elliVM]

Added pattern table to bloomdb can be used to select specific filter. Pattern is stored a it's own bloom filter byte array, when a search term is included in the saved patterns (using bloommatch udf) it will activate bloom search using filter of that pattern.

For simplicity when a filter is created it will be assigned a single pattern that the search term is matched against. later this can be changed to multiple patterns per filter and vice versa.

[elliVM]

Changing to support multiple patterns per filter

[elliVM]

Implemented a schema with a pattern table and a junction table between patterns and filters,. Condition walker selects only filters with pattern match with search term and run UDF bloommatch for temp table filters generated from filter types and search term.

Next:

• Add testing
• Disable bloom if no filters found

[elliVM]

Changes to be made:

• [x] Only one pattern per filter is needed remove junction table. Move pattern to filtertype
• [x] Change to use regex for matching matching instead of UDF, start first without tokenization.
• [x] Update schema, move pattern to filtertype table as a datatype that can use regex.
• [x] Later tokenize search term before matching.

[elliVM]

Testing version with pattern matching against tokenized search terms

[elliVM]

New changes to be made

• [x] Create a database table for each stored regex pattern with a bloom filter
• [x] Join all filter tables that have a regex pattern match with incoming archive search term
• [x] Run bloommatch UDF for each logfile and select those that match any of the joined filters
• [x] To run bloommatch, create a temp table for each bloom filter table that has a pattern match

[elliVM]

Created a new walker that finds all dynamic bloomfilter tables that have a pattern match with the tokenized search term, will use this to select the tables for join with the main query. (Combined with Condition Walker)

[elliVM]

Created classes to hold dynamic tables and temp tables

[elliVM]

Internal PR

[elliVM]

updates to filtertype table: pattern varchar value increased to 2048 and pattern added to unique composite index

[elliVM]

Fixed issues with filter size selection in temp tables generated for bloommatch condition. Limited tokenizers to use only major tokens to match with dpf_03.

Working in QA with working filtering (pth-07 5.3.0-22-gbd5da88a)

Test example index=alert_examples earliest=-999d "c3468f80-4273-4867-9b66-3f470787c365"

without bloom took 16-18s with bloom 3-6s

[elliVM]

Fixing an issue where table pattern match filtering from meta data was fetching the whole table data to java memory, limited fetch to check only 1 row and only PK field.

[elliVM]

Duplicate rows on multiple pattern matches when multiple tables are joined, testing fix using group by logfile.id

update - group by too slow, false positives maybe caused by null on null bloommatch check if a pattern match table was joined that has no matching logfiles for index.

[elliVM]

null check after bloommatch condition for bloom filters fixed duplicate issues and speed up query with multiple joined tables.

[elliVM]

Fixed bug with multiple search terms, tested and working in QA.

[elliVM]

• Updated old tests to company standards

[elliVM]

refactoring: move all tokenization to PatternMatch class and move all bloommatch condition generation steps to BloomFilterTempTable class

[elliVM]

• QA testing showed good results with small number of matches performance gain fell down gradually as matches increased still all queries were faster with bloom enabled.
• A single large table and multiple tables had good performance.

[elliVM]

will split the refactoring into another PR and implement the changes requested in review