Add "hit"/"miss" statistics for queries

adulau / hashlookup-server

Fast lookup server for NSRL and other hash database used in digital forensic

https://adulau.github.io/hashlookup-server/

GNU Affero General Public License v3.0

41 stars 7 forks source link

Open cudeso opened 3 years ago

cudeso commented 3 years ago

Track the queries that returned a hit or did not return a hit. Track per hash and have a "namespace".

handy when importing sets from other sources
run over all files in a disk, track which files have been seen, which one haven't been seen
per "dfir"-session
get statistics afterwards
option to query and reset statistics, per session
store on db side

adulau commented 3 years ago

Thanks a lot for the ideas.

For tracking queries, I'll add table of hashes with hits and no hits. It will also add a pub-sub channel for it.
For the session, that's a clever idea. Something like Ubuntu Bionic default session and we could even save the set and make it available if other parties are interested.

adulau commented 3 years ago

[X] Pub-sub functionality for hit-missed added in 31ac93e764278d40e99e1e95534850ad65a850af
[X] Add a hit/miss zrank option to add statistics in 72b462b5ea858f7973fa17dc1cd575c5433df277
[ ] Support statistics for bulk search too
[ ] Add a new API endpoints to paginate over the statistics
[X] Add a new option for session creation with TTL