search

mattgodbolt / zindex

Create an index on a compressed text file
BSD 2-Clause "Simplified" License
622 stars 37 forks source link

Support asynchronous external indexing #15

Open mattgodbolt opened 9 years ago

mattgodbolt commented 9 years ago

At the moment line buffering is required of the --pipe command, as the whole thing is synchronous. Ideally the external indexer could run in parallel with zindex itself, which would make it impossible to deadlock, and can probably speed up the operation too.

mattgodbolt commented 9 years ago

Will probably have to implement this as some large jq queries are sometimes hitting some buffering limit (e.g. a query I have which has 567481 byte lines, and 6000 byte outputs)

mattgodbolt commented 9 years ago

That turned out to be a bug, so this is just a "would be nice"

mattgodbolt commented 9 years ago

This still seems to happen when (for example) tests run under an IDE (eg CLion)

mattgodbolt commented 9 years ago

(/home/mgodbolt/dev/zindex/tests/ExternalIndexerTest.cpp:21)