This is more of a question

[ghost]

Hello, how can I connect this to a shadowserver database which has been built on prem. Can you kindly guide me on that please

[giga-a]

Hey @iowuraku! what kinda database do you have? and what type of data it stores?

[ghost]

Hello @giga-a it is a simple Mysql database that has just a table. It stores source IP, destination ip, src port, destination country and source country, malware type, source city and destination city. There is also country code

[t3chn0m4g3]

My thoughts are going in a similar direction, but would require an adjusted design of Raven. If it had a middleware feeding data from i.e. DBs, ES, JSON to a Redis which in turn would provide the data for a WS which could be accessed by Raven for plotting. This would solve not needing a direct access to the log source to feed Raven on the endpoint and caching data on Redis for multiple clients. This Map Service as an example what I mean, however Raven looks better 😎.

[giga-a]

@t3chn0m4g3! like this

😎

[giga-a]

@t3chn0m4g3 This wasElastic -> publisher -> Redis -> subscriber -> Websocket -> Raven based on the example you provided. Raven already has websocket and I only needed to implement the rest

Raven does multiple things while processing the attack event, it analyzes it before plotting, it gives info regarding the IP, port, phone number, or coordinates. Then, it plots the info by IP, country, city,country, city,state,country, or coordinates if any of them are valid.

What I did was adding an option called custom, this option disables the analyzing step, and moves to plot the info by IP, port, phone number, or coordinates if any is correct. This allows you to use any metadata fields (like geoip_ext fields) and show them to the end user

tpot (E.g. record trimmed)

{
  "_index": "logstash-2022.02.12",
  "_type": "_doc",
  "_id": "xxx",
  "_score": 1,
  "_source": {
    "t-pot_ip_int": "10.0.2.15",
    "in_iface": "enp0s3",
    "flow_id": "test",
    "src_ip": "xxx",
    "payload": "",
    "host": "test",
    "t-pot_ip_ext": "xxx",
    "tags": [
      "_geoip_lookup_failure"
    ],
    "path": "/data/suricata/log/eve.json",
    "proto": "UDP",
    "geoip_ext": {
      "location": {
        "lon": -1.6541,
        "lat": 1.6952
      },
      "ip": "xxx",
      "postal_code": "xxx",
      "country_code2": "US",
      "region_name": "Washington",
      "country_code3": "US",
      "city_name": "Vancouver",
      "longitude": 1.6541,
    },
    "src_port": 5353,
    "t-pot_hostname": "panickyskunk",
  }
}

Example (Dummy + real values from elastic, adjust as needed!)

redis.publish('alerts',dumps([{'function':'table','method':'coordinates','object':{'from':from_,'to':to_},'color':{'line':{'from':'#123456','to':'#123456'}},'timeout':1000,'options':['line','multi-output','country-by-coordinate'],'custom':{'from':{'srcip':dummy_ip(),'srcport':randint(10000,50000)},'to':{'destip':dummy_ip(),'destport':dest_port}},}]))

{
'function': 'table'                   #plot it and add it to the table (You can also plot without adding it)
'method': 'coordinates'               #we are using valid coordinates to plot (You can change this to name, or ip)
'object': {'from':['1.0','10.0'],     #[lat,lon] or 'us' or 'wa,us' or '8.8.8.8'
            'to': ['1.0','20.0']},    #[lat,lon] or 'us' or 'wa,us' or '8.8.8.8'
'timeout': 1000,                      #this is the attack duration on the map
'options': ['line',                   #draw line
            'multi-output',           #draw line
            'country-by-coordinate'], #find country by coordinates
custom':                              #`custom` will skip the analysis step and the follow feilds if above is valid
        {'from':                      #the following will be added to the `from` box
            {'example':'Nothing',     #box `from` will have `example: Nothing`
             'srcport':55443},        #box `from` will have `srcport: 55443`
         'to':                        #the following will be added to the `to` box
            {'destip':'1.0.0.0',      #box `to` will have `destip: 1.0.0.0`
             'destport':8080}}        #box `to` will have `destport: 8080`
}

Here are files (You will need to clone the entire project into tpot and run raven_tpot_simulation.py & raven_server.py, then 0.0.0.0:4751/raven_tpot.html - also you need to install redis)

• raven_tpot_simulation.py, this is the publisher
• raven_server.py this is the server and has a subscriber
• raven_tpot.html

Let me know what you think!?

[t3chn0m4g3]

@giga-a Yes, this is exactly what I meant! Perfect, thank you!

[giga-a]

@t3chn0m4g3 I guess we work really well tougher!

[giga-a]

@iowuraku I will provide you with an example in the next a few days.

[t3chn0m4g3]

@giga-a This is the way 😄 I am planning to use this for T-Pots which use a distributed installation, so you can see attacks on all the T-Pots, not just a single one.

[giga-a]

@iowuraku apologize for the delay, I had to refactor a few things and made the current way of plotting a lot easier! Now, you can choose any combination for from and to, and Raven will parse that, you can use ip with a coordinate, name with ip etc..

raven.add_to_data_to_table({'from':'8.8.8.8','to':'delhi,in'},{'line':{'from':null,'to':null}},2000,['line','multi-output','single-output'])

raven.add_marker_by_gussing({'from':'-11.074920,-51.648929','to':'us'},{'line':{'from':null,'to':'#FFFF00'}},1000,['line'])

So, I made you an example for mysql, and you need to have a mysql running and then run the mysql_simulation.py. It should create a db called testdb, it creates a table called attacks with src_ip , src_port, dest_ip, dest_port, and time. Then, it inserts dummy records into attacks - Meanwhile, the server connects to the testdb and fetches the new records based on the time.

Here are the file: docker-compose file and server file - Remember that these files are examples, and you need to adjust them based on your needs

Please take a look and let me know what you think!

[ghost]

Hi there we first tested the examples to see if theyll populate on load... the database socket works but it didnt load the data. We created our database too and tested ..the socket worked but it didnt load the data on the map. The generate side still picks from the js previously built

[giga-a]

@iowuraku which html file are you using? my guess that you were on the index.html, it should be simulation.html because of this option window['raven'].fetch_data_from_server() which activates polling data from the server

Here is a new testing example:

git clone https://github.com/qeeqbox/raven
cd raven
sudo docker-compose -f docker-compose-mysql.yml up --build --force-recreate --no-deps --remove-orphan

And, go to http://localhost:8080/simulation.html

Try this and let me know!

[giga-a]

@t3chn0m4g3

I am planning to use this for T-Pots which use a distributed installation, so you can see attacks on all the T-Pots, not just a single one.

Great, when you start working on that let me know, I can help!

[t3chn0m4g3]

@giga-a Thank you! Appreciate it.