private-search-set support

[be-a-panther]

Please Describe The Feature You Want

1. extending the threat intelligence module to check against the circl.lu provided bloomfilter https://circl.lu/services/hashlookup/#querying-hashlookup-without-online-queries
2. add private search set support https://github.com/hashlookup/private-search-set

(Optional): Suggest A Solution

1. extend the folder StratosphereLinuxIPS/config/local_ti_files with a folder pss and put there two folders with naming blockand allow
2. allow the monthly download for the provided bloomfilter from circl.lu
3. extend the the threat intelligence module to check the bloom filter before query the api
4. extend the overall query behavior to be confidential and only based on local ti files

• I guess that the native bloomfilter from circl.lu will be shared as private search set soon
• AFAIK there is a migration from version 1 to version 2 due a bug in version 1 @qjerome

[eldraco]

Thanks @be-a-panther . Can you tell us more why you would need PSS? I'm not sure if I get the use case.

Remember that there are two ways of quering data:

1. Slips downloads all the blacklists and aggregates them and updates them.
2. Slips query services like Virustotal if you have an API, URLhaus, Spamhaus, Circl.lu.

The bloomfilter can be included as a threat intelligence feed directly probably. If the format is good, try to just include it here https://github.com/stratosphereips/StratosphereLinuxIPS/blob/master/config/TI_feeds.csv

It should be downloaded, aggregated, used and updated every 24hs automatically.

[be-a-panther]

The PSS allows me to share and to include Indicators of Attacks (IoA) without knowing the clear data in beforehand.

Taking the case that I get a PSS. I would query any Indicator against it. As soon as I get a positive result I can assume that I'm under attack. But with the PSS I can not leak or share the clear text without possessing it.

---

Using the bloomfilter instead of the full list makes it faster to check against it and keeping the dataset relatively small. But it is a binary format, so I can not directly put it into the the feeds.csv If you use the bloomfilter for a small dataset (e.g. sharing IoA), you have a plausible deniability integrated.

Just context: In the case you can not rely on public known good api services because you have special systems. In that case I would create a bloom filter over all files of e.g. my embedded systems and serve those information with a forked hashlookup server. But to reduce the network traffic and resource load I would distribute this knowledge as a bloomfilter down to each system and support this to the HIDS.

---

The bloomfilter and Private-Search-Sets are related, but maybe two different sides of the same medal. The private-search-set can be included as long as the provided data is the full pss set. As soon the private-search-set is distributed with a bloomfilter, we need the software support for this binary format.

I hope it is getting clearer.

[eldraco]

Hi @be-a-panther, we have some thoughts about this

Bloom filters alone

• We can not have false positives in Slips, so using a Blooom filter alone is not good for us. Imaging searching for google.com and receiving a Positive. And in bloom filters the prob of positive goes up with larger DBs (now our TI DB is in the order of two hundred thousand entries per day)
• If the DB is local, there is no need for privacy. The problem goes when querying on the internet.
• If the DB is on the internet, then yes

Private set search

• This is better and we like the idea of private search and watermarking. We can see two ways of using this
  1. We connect to some MISP instance and we use their PSS if they have one. This is now a new task for us to implement. (We will also implement to share TI to MISP)
  2. We would like to implement PSS in our global P2P TI system that is being develop right now. (code is called Fides and Iris in this repo)
• Issues:
  • As far as I can tell MISP does not have it implemented yet, right? we can wait for this.
  • We will check if PSS also means a prob of false positives, if this is the case, we need to think about it again.

[be-a-panther]

AFAIK MISP does not have the feature implemented yet.

The Private-Search-Set can contain a full list or a bloomfilter. It depends what you get shared. If you can not accept any FPR, then it is getting complicated. As long as you are in the possession of the PSS, you can check if it is the full version or the bloomfilter. If you want to use the MISP in the background with providing PSS data, then you will get a FPR included.

If someone would put "google.com" into the bloomfilter, it would be bad. But the same applies to a (e.g. yara) rule. IMHO the PSS is useful to share confidential indicators. the PSS gets obsolete in that moment, someone makes the containing indicators public. After this moment you are capable of testing all your PSS against those freshly released indicators. The bloomfilter gives you better options to share indicators into leaky environments.

I guess that the bloomfilter itself would fit into your P2P concept. You can reduce the api-query load for your TI db (>200000entries/day) and distribute a bloomfilter to each IDS. Maybe it can be used to store the older dumps from the TI db, that you generate a history lookup without wasting too much space and resources. But I didn't look into your P2P code.

But yes, a bloomfilter only with malicious IPs would not make any sense.