search

brimdata / super

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.39k stars 64 forks source link

brim searches should use time index #622

Closed alfred-landrum closed 4 years ago

alfred-landrum commented 4 years ago

We've added support for an in-memory time index in zqd; we need to ensure that Brim's api usage (ingest & search) uses it.

aswan commented 4 years ago

This was done in #647

philrz commented 4 years ago

Verified in Brim commit e9b0840 talking to zqd commit 48ed30a.

I'd ingested the 12 GB wrccdc "year1" data set and let it do the initial full scan through the data, which I understand has the side-effect of populating the index in memory in zqd. When zoomed all the way out, full scans were taking on the order of minutes. I then was able to repeatedly brush to zoom in to a tight time range in the data, and the histogram redraws now complete in mere seconds, thanks to the time index seeking into the the ZNG and showing me just the correct subset of records. This is shown in the attached video.

Verify.zip

Thanks @aswan!