Concrete Data Storage / Refresh Proposal w/ Architectural Diagrams

[dm03514]

• The component view of the application
  • ipsets database
  • application
  • system/os
  • database refresher etc
• The flows of the application
  • transactional flow (end user request -> application -> db -> end user response)
  • database refresh

[dm03514]

Since latency is critical requirement for the application, I feel like the updating/refresh logic should go on in the background in order to keep latency in the app as low as possible. This means that the app will just reach into the datastore trusting that it is up to date.

The next thing I'm thinking is whether there should be a centralized store that is consistent between all instance of the application at the cost of network latency overhead, vs individual data stores per application which may result in inconsistencies, ie if there are 2 instances of our service one may have the most up to date lists, while the other is refreshing. If we had a single IP and made a request to each during this time, it could be that one instances returns malicious and the other which is in the process of updating returns not malicious.

Have to benchmark these in order to present a proposal ie speed vs consistency

[dm03514]

Going to try postgres' cidr type and their index.

Needs to benchmark:

• querying full datasets
• insert times
• update times

[dm03514]

$ cat conf/firehol.conf
FIREHOL_LOAD_KERNEL_MODULES=0

ipv4 ipset create dshield hash:net
ipv4 ipset addfile dshield /etc/firehol/ipsets/dshield.netset
sudo ipset-apply /etc/firehol/ipsets/dshield.netset