PoC to better streamline the import of NSRL data.

adulau / hashlookup-server

Fast lookup server for NSRL and other hash database used in digital forensic

https://adulau.github.io/hashlookup-server/

GNU Affero General Public License v3.0

41 stars 7 forks source link

PoC to better streamline the import of NSRL data. #4

Closed cudeso closed 3 years ago

cudeso commented 3 years ago

Still requires some work but basic concept works. Currently only tested with Android

adulau commented 3 years ago

Thanks a lot for the contribution. I'll do some tests.

cudeso commented 3 years ago

Thanks. There are also three additional elements added

drecords['source'] = "NSRL"
drecords['db'] = rdbkey
drecords['insert-timestamp'] = time.time()

rdbkey comes from the dataset identifier <- defined in config.json ; provided as argument to __process_nsrl_base

adulau commented 3 years ago

Thanks, it makes sense and I think this can be easily compressed on the RocksDB size (as it's the often the same data). I'll add a doc with the database structure.