search

fkie-cad / FACT_core

Firmware Analysis and Comparison Tool
https://fkie-cad.github.io/FACT_core
GNU General Public License v3.0
1.22k stars 224 forks source link

Integrating hashlookup.circl.lu in FACT #702

Closed adulau closed 2 years ago

adulau commented 2 years ago

Hi All,

We (CIRCL) are maintaining the hashlookup.circl.lu. It would be interesting to lookup hashes found in firmware against hashlookup to see existing and known binaries from other distributions. The API is publicly and freely accessible.

Cheers

dorpvom commented 2 years ago

Hi Alexandre! I've tried out the API and it seems straight forward to develop a plug in to check with the API. I will POC one in the coming days. Do you have any experience as to how likely it is to find a match on arbitrary buildroot firmware? Cheers!

dorpvom commented 2 years ago

Branch 702-hashlookup contains a POC for this. My run on a COTS Home Router with ~ 1700 files produced 15 matches (14 config files and a font).

dorpvom commented 2 years ago

@adulau The plugin seems to work fine. We were discussing if some file types should be blacklisted or only a select few should be whitelisted to not cause unnecessary load on the API. Do you have ideas towards that based on your experience with the tool?

adulau commented 2 years ago

That's awesome! Thank you very much.

There is no specific limitation. My experience for the API, I would recommend to implement a local cache to avoid querying multiple type the same hash.

adulau commented 2 years ago

We recently introduced a Bloom filter to allow users to do local lookup without the need of an Internet connectivity. If you want to use the Bloom filter in FACT, you can use the flor library from DCSO which we use for hashlookup-forensic-analyser.

rhelmke commented 2 years ago

That's awesome! Thank you very much.

There is no specific limitation. My experience for the API, I would recommend to implement a local cache to avoid querying multiple type the same hash.

Hey,

just wanted to let you know that FACT uses hash-based file object deduplication across the whole database. Thus, analysis results are also shared. So basically there shouldn't be a need for dedicated lookup caching: it is supported by design :-).

Multiple lookups of the same hash should only occur when the user explicitly issues an analysis update. But then, I think one would expect that FACT refetches the dataset :-)

rhelmke commented 2 years ago

The hashlookup plugin has been integrated into FACT for a while now :-)

Closing!