Closed alfred-landrum closed 4 years ago
This was done in #647
Verified in Brim commit e9b0840
talking to zqd
commit 48ed30a
.
I'd ingested the 12 GB wrccdc "year1" data set and let it do the initial full scan through the data, which I understand has the side-effect of populating the index in memory in zqd
. When zoomed all the way out, full scans were taking on the order of minutes. I then was able to repeatedly brush to zoom in to a tight time range in the data, and the histogram redraws now complete in mere seconds, thanks to the time index seeking into the the ZNG and showing me just the correct subset of records. This is shown in the attached video.
Thanks @aswan!
We've added support for an in-memory time index in zqd; we need to ensure that Brim's api usage (ingest & search) uses it.