import www.botvrij.eu json with python

[truckydev]

Hi team,

Here a little python script to import json elements from http://www.botvrij.eu/data/feed-osint/

It's not an issue, just an easy way to populate an instance.

from pymisp import PyMISP
import requests, json, re

botvrij = requests.get('http://www.botvrij.eu/data/feed-osint/')
reg = re.compile(r'href="(\w){8}-.+(json)"')
listJson = [r.group() for r in reg.finditer(botvrij.text)]
listJson = [x.replace('"','') for x in listJson]
listJson = [x.replace('href=','') for x in listJson]

misp = PyMISP('http://misp.test/', 'key', False, 'json')
for uri in listJson:
    requests.get('http://www.botvrij.eu/data/feed-osint/'+uri)
    misp.add_event(requests.get('http://www.botvrij.eu/data/feed-osint/'+uri).text)

Hope will be useful

[FloatingGhost]

reg = re.compile(r'href="(\w){8}-.+(json)"')

Reminder: http://stackoverflow.com/a/1732454

[truckydev]

use BeautifullSoup is maybe overkill no ?

[FloatingGhost]

no!

from pymisp import PyMISP
import requests
import bs4

botvrij = requests.get('http://www.botvrij.eu/data/feed-osint/')
soup = bs4.BeautifulSoup(botvrij.text, "lxml")
listJson = [x.attrs["href"] for x in soup.find_all("a") if "json" in x.attrs["href"] ][:-1]

misp = PyMISP('http://misp.test/', 'key', False, 'json')
for uri in listJson:
    requests.get('http://www.botvrij.eu/data/feed-osint/{}'.format(uri))
    misp.add_event(requests.get('http://www.botvrij.eu/data/feed-osint/{}'.format(uri)).text)

[iglocska]

Why not just get the manifest.json, loop through the events and load each event json by uuid? Sounds cleaner.

On Nov 30, 2016 4:11 PM, "Hannah Ward" notifications@github.com wrote:

no!

from pymisp import PyMISPimport requestsimport bs4

botvrij = requests.get('http://www.botvrij.eu/data/feed-osint/') soup = bs4.BeautifulSoup(botvrij.text, "lxml") listJson = [x.attrs["href"] for x in soup.find_all("a") if "json" in x.attrs["href"] ][:-1]

misp = PyMISP('http://misp.test/', 'key', False, 'json')for uri in listJson: requests.get('http://www.botvrij.eu/data/feed-osint/{}'.format(uri)) misp.add_event(requests.get('http://www.botvrij.eu/data/feed-osint/{}'.format(uri)).text)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/MISP/MISP/issues/1701#issuecomment-263897533, or mute the thread https://github.com/notifications/unsubscribe-auth/ADf6wACU1WTCOAPwYUVEJIAfx7dajEbBks5rDZIfgaJpZM4LAAFW .

[FloatingGhost]

from pymisp import PyMISP
import requests

botvrij = requests.get('http://www.botvrij.eu/data/feed-osint/manifest.json')

misp = PyMISP('http://misp.test/', 'key', False, 'json')
for uri in botvrij.json():
    req = requests.get('http://www.botvrij.eu/data/feed-osint/{}.json'.format(uri))
    misp.add_event(req.json())

yeah you're right that is nicer

[iglocska]

Excellent B-)

[truckydev]

Yes, it's an easy way

[adulau]

Nice example. I used it to update the PyMISP section in misp-book.

https://github.com/MISP/misp-book/tree/master/pymisp#consuming-feed

Thank you very much.