Description of changes:
This series of commits simplifies implementations of lcs_substr and remove_common_tokens as well as greatly reducing amount of memory allocations. Functionality is preserved, all tests pass.
Note: in lcs_substr the common prefix is .trim()ed. However, if this common prefix happens to start with whitespace characters, this will give wrong results later in remove_common_tokens. It seems like it actually should be .trim_end().
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Description of changes: This series of commits simplifies implementations of
lcs_substr
andremove_common_tokens
as well as greatly reducing amount of memory allocations. Functionality is preserved, all tests pass.Note: in
lcs_substr
the common prefix is.trim()
ed. However, if this common prefix happens to start with whitespace characters, this will give wrong results later inremove_common_tokens
. It seems like it actually should be.trim_end()
.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.