search

yampelo / beagle

Beagle is an incident response and digital forensics tool which transforms security logs and data into graphs.
MIT License
1.27k stars 145 forks source link

sending data to neo4j #25

Closed garanews closed 5 years ago

garanews commented 5 years ago

Following the guide in order to use Neo4J I eported these variables (passing through -e in docker run...) : BEAGLEGENERALLOG_LEVEL=DEBUG BEAGLENEO4JHOST=bolt://my_neo4j_ip:7687 BEAGLENEO4JUSERNAME=myuser BEAGLENEO4JPASSWORD=mypwd

I am able to see the debug messages, but there is no communication with Neo4j. I see INFO | Initialized NetworkX Backend Am I missing something? I tried also to use docker-compose with same variables under "environment" but result is the same. Thanks

garanews commented 5 years ago

I dont find in the code a place to change the backend...

yampelo commented 5 years ago

You need to use the python bindings to send to neo4j:

from beagle.backends import Neo4J
from beagle.datasources import HXTriage

HXTriage(...).to_graph(backend=Neo4J)

https://github.com/yampelo/beagle#python-library

Fair warning, my cypher is awful and there may be a way way way better to set it up to send to neo4j, if you guys have more experience you should totally contribute a better way to transform the node classes to batch cypher upload queries 😄

garanews commented 5 years ago

I installed (not so easy) beagle outside docker with pipenv in order to modify code faster. I'm looking the base_transformer.py should I set there Neo4j instead networkx? I tried this from beagle.backends.neo4j import Neo4J def to_graph(self, backend: "Backend" = Neo4J, *args, **kwargs) -> Any:

but system is still using networkx instead neo4j

yampelo commented 5 years ago

Are you trying to use the web interface to upload to Neo4J? That won't work, you need to use the bindings like above. The web interface is meant to be a self-contained thing, where you upload a triage, it transforms it to a NetworkX object, saves it in the database, then redirects you to the web interface for it.

Try calling the python library directly like in my example above.

garanews commented 5 years ago

having some issue passing db parameters: Traceback (most recent call last): File "", line 1, in File "/root/.local/share/virtualenvs/beagle-ZloiBEyB/lib/python3.6/site-packages/beagle/datasources/base_datasource.py", line 111, in to_graph return self.to_transformer(self.transformers[0]).to_graph(*args, *kwargs) # type: ignore File "/root/.local/share/virtualenvs/beagle-ZloiBEyB/lib/python3.6/site-packages/beagle/transformers/base_transformer.py", line 60, in to_graph backend = backend(nodes=nodes, metadata=self.datasource.metadata(), args, **kwargs) File "/root/.local/share/virtualenvs/beagle-ZloiBEyB/lib/python3.6/site-packages/beagle/backends/neo4j.py", line 38, in init self.neo4j = GraphDatabase(uri, auth=(username, password)) TypeError: object() takes no parameters

passing manually someghing changes: backend = HXTriage('/home/users/analyst/dsadadm15jMy.mans').to_graph(backend=Neo4J, uri="bolt://12.3.4:7687", auth=("neo4j","password"))

2019-04-19T15:00:42 | beagle.transformers.base_transformer.run:111 | INFO | Finished processing of events, created 8838 nodes. 2019-04-19T15:00:42 | beagle.backends.neo4j.init:36 | INFO | Connecting to neo4j server at bolt://1.2.3.4:7687 Traceback (most recent call last): File "", line 1, in File "/root/.local/share/virtualenvs/beagle-ZloiBEyB/lib/python3.6/site-packages/beagle/datasources/base_datasource.py", line 111, in to_graph return self.to_transformer(self.transformers[0]).to_graph(*args, *kwargs) # type: ignore File "/root/.local/share/virtualenvs/beagle-ZloiBEyB/lib/python3.6/site-packages/beagle/transformers/base_transformer.py", line 60, in to_graph backend = backend(nodes=nodes, metadata=self.datasource.metadata(), args, **kwargs) File "/root/.local/share/virtualenvs/beagle-ZloiBEyB/lib/python3.6/site-packages/beagle/backends/neo4j.py", line 38, in init self.neo4j = GraphDatabase(uri, auth=(username, password)) TypeError: object() takes no parameters

yampelo commented 5 years ago

Seems like the Neo4J has a couple of bugs with it since I had to move over from py2neo to the official neo4j driver. I might not be able to debug until this weekend due to travel, do you wanna chat about it on slack? : https://join.slack.com/t/beaglegraph/shared_invite/enQtNjE1MzQyNTE3NTI3LTUzMGNhMzcwN2M5ODg3NGRmNzVmM2ZjMmU0NDk3ODEwMmRkZjEyZWIxYzU5NmM3MWMxOWU4MjVhNWEyNWVkNTI

garanews commented 5 years ago

Sure, I joined in slack (I'm also on gitter, same user) on other projects I am running py2neo (4.x). Take your time, now Easter times :) Thanks

yampelo commented 5 years ago

Hey @garanews , i finally finished my finals week and had time to check this out. I got it working!

image

yampelo commented 5 years ago

The version on pypi (pip install pybeagle --upgrade) will allow it to work

garanews commented 5 years ago

Hello, thanks, going to try it immediately! Now it would be useful sending data from ui directly to neo4j :)

yampelo commented 5 years ago

Opened #41 to track it, it should be pretty straight forward to do.