search

scoder / acora

Fast multi-keyword search engine for text strings
http://pypi.python.org/pypi/acora
BSD 3-Clause "New" or "Revised" License
247 stars 17 forks source link

readme

Acora

.. contents:: :local:

What is Acora?

Acora is 'fgrep' for Python, a fast multi-keyword text search engine.

Based on a set of keywords and the Aho-Corasick algorithm <https://en.wikipedia.org/wiki/Aho-Corasick_algorithm>_, it generates a search automaton and runs it over string input, either unicode or bytes.

Acora comes with both a pure Python implementation and a fast binary module written in Cython. However, note that the current construction algorithm is not suitable for really large sets of keywords (i.e. more than a couple of thousand).

You can find the latest source code <https://github.com/scoder/acora>_ on github.

To report a bug or request new features, use the github bug tracker <https://github.com/scoder/acora/issues>_. Please try to provide a short test case that reproduces the problem without requiring too much experimentation or large amounts of data. The easier it is to reproduce the problem, the easier it is to solve it.

Features

How do I use it?

Import the package::

>>> from acora import AcoraBuilder

Collect some keywords::

>>> builder = AcoraBuilder('ab', 'bc', 'de')
>>> builder.add('a', 'b')

Or::

>>> builder.update(['a', 'b'])  # new in version 2.0

Generate the Acora search engine for the current keyword set::

>>> ac = builder.build()

Search a string for all occurrences::

>>> ac.findall('abc')
[('a', 0), ('ab', 0), ('b', 1), ('bc', 1)]
>>> ac.findall('abde')
[('a', 0), ('ab', 0), ('b', 1), ('de', 2)]

Iterate over the search results as they come in::

>>> for kw, pos in ac.finditer('abde'):
...     print("%2s[%d]" % (kw, pos))
 a[0]
ab[0]
 b[1]
de[2]

Acora also has direct support for parsing files (in binary mode)::

>>> keywords = ['Import', 'FAQ', 'Acora', 'NotHere'.upper()]

>>> builder = AcoraBuilder([s.encode('ascii') for s in keywords])
>>> ac = builder.build()

>>> found = set(kw for kw, pos in ac.filefind('README.rst'))
>>> len(found)
3

>>> sorted(str(s.decode('ascii')) for s in found)
['Acora', 'FAQ', 'Import']

FAQs and recipes

) How do I run a greedy search for the longest matching keywords?

::

   >>> builder = AcoraBuilder('a', 'ab', 'abc')
   >>> ac = builder.build()

   >>> for kw, pos in ac.finditer('abbabc'):
   ...     print(kw)
   a
   ab
   a
   ab
   abc

   >>> from itertools import groupby
   >>> from operator import itemgetter

   >>> def longest_match(matches):
   ...     for pos, match_set in groupby(matches, itemgetter(1)):
   ...         yield max(match_set)

   >>> for kw, pos in longest_match(ac.finditer('abbabc')):
   ...     print(kw)
   ab
   abc

Note that this recipe assumes search terms that do not have inner overlaps apart from their prefix.

) How do I parse line-by-line with arbitrary line endings?

::

   >>> def group_by_lines(s, *keywords):
   ...     builder = AcoraBuilder('\r', '\n', *keywords)
   ...     ac = builder.build()
   ...
   ...     current_line_matches = []
   ...     last_ending = None
   ...
   ...     for kw, pos in ac.finditer(s):
   ...         if kw in '\r\n':
   ...             if last_ending == '\r' and kw == '\n':
   ...                 continue # combined CRLF
   ...             yield tuple(current_line_matches)
   ...             del current_line_matches[:]
   ...             last_ending = kw
   ...         else:
   ...             last_ending = None
   ...             current_line_matches.append(kw)
   ...     yield tuple(current_line_matches)

   >>> kwds = ['ab', 'bc', 'de']
   >>> for matches in group_by_lines('a\r\r\nbc\r\ndede\n\nab', *kwds):
   ...     print(matches)
   ()
   ()
   ('bc',)
   ('de', 'de')
   ()
   ('ab',)

) How do I find whole lines that contain keywords, as fgrep does?

::

   >>> def match_lines(s, *keywords):
   ...     builder = AcoraBuilder('\r', '\n', *keywords)
   ...     ac = builder.build()
   ...
   ...     line_start = 0
   ...     matches = False
   ...     for kw, pos in ac.finditer(s):
   ...         if kw in '\r\n':
   ...             if matches:
   ...                  yield s[line_start:pos]
   ...                  matches = False
   ...             line_start = pos + 1
   ...         else:
   ...             matches = True
   ...     if matches:
   ...         yield s[line_start:]

   >>> kwds = ['x', 'de', '\nstart']
   >>> text = 'a line with\r\r\nsome text\r\ndede\n\nab\n start 1\nstart\n'
   >>> for line in match_lines(text, *kwds):
   ...     print(line)
   some text
   dede
   start

Changelog