This is an open implementation of the SCISSOR project.
The goal of this project is to have a fully functional virtualized environment up and running with the minimal amount of manual steps.
docker-compose up
from the docker
directoryEach architectural component is deployed in it's own Docker container.
Here is a list of all the managed components with a brief description:
kafka
: provides an Apache Kafka message broker (the SMI component of SCISSOR) as well as Apache Zookeeper (needed by the Edge Agent Controller (see flume VM) and the Control and Coordination Agent (in this kafka VM)).flume
: provides parse, filter and output components of the Command and Control Layer (CCL) processing chain.semantics
: provides enrich and transform components of the CCL processing chain.logstash
: provides endpoints for the data handover between Monitoring Layer (ML) entities and the CCL (e.g. via filebeat and ZeroMQ)logstash24
: provides endpoints for the data handover between Monitoring Layer (ML) entities and the CCL (e.g. via filebeat and ZeroMQ)datasource24
: This machine is used exclusively for testing, generating artificial loads and replaying previously recorded data.d-streamon-master
: D-Streaon is a distributed framework for network monitoring, this is the Master machined-streamon-slave
: Slave machine of D-Streamonprelude-manager
: IDMEF database to store IDMEF objects. Can be accessed through libprelude or via mysql direct access. A service (prelude-registrator
) has been added to enable automatic registration.prelude-manager-db
: MariaDB instance supporting prelude-manager
and prewikka
event-correlator
: Connects to the prelude database (prelude-manager) and applies correlation rules to incoming events to detect advanced attacks.prewikka
: Web interface that connects to the prelude-manager
(via direct mysql access) and displays alerts via http.kafka-idmef-converter
: Gets messages from kafka coming from the CCL, applies a filter, converts them to IDMEF, serializes them and sends them back to kafka on a dedicated topic (currently named IDMEF).kafka-prelude-connector
: Gets messages from kafka on the IDMEF topic, unserializes them and sends them to the prelude-manager
instance (using libprelude).The Docker containers are managed with Docker Compose (see docker-compose.yml
), that takes care of:
The test suite runs through the following steps:
You can find what we consider the unmatained legacy version of this project (Virtual Machines managed by Vagrant) by checking out the 1.0.0
git tag.
We adopted a Test Driven Development process to ensure the correctness of the "dockerization" process we started as part of the 2.0.0
release.
We configured CI builds for each commit and pull request. On each build we run the full verification and test suite.
If you have issue or want to contribute to the project, please create a new GitHub issue or pull request.
Note that collaborators cannot push directly to development
and master
branches but should open a pull request against these branches and wait for the automated checks to complete and for a final manual review by other contributors.
If you want to setup a development environment for this project, you may follow one of the following paths:
We used a virtual machine managed with Vagrant to bootstrap the development environment because we wanted to standardize our tooling and to automate the setup of each development workstation.
Run vagrant up
from the root of the project. Vagrant will download and run a VirtualBox VM with all the needed development tools configured and ready to be used. See https://github.com/ferrarimarco/open-development-environment-devbox for more info.
If you prefer a manual setup to the development box described above, here are the necessary dependencies:
Run test/test-docker-images.sh --only=integration --docker-context-path=docker --skip-build --skip-pull --skip-start
from the root of the project. This script will:
--only=lint-dockerfile
switch)--only=lint-shell
switch)--only=integration
switch)
--skip-build
switch) OR pull each image from Docker Hub (useful for CI, skippable with --skip-pull
switch)--skip-start
switch)