Configure a testing framework for the infrastructure

[ferrarimarco]

Let's start by finding information about each component.

VMs to describe:

• [x] kafka
• [x] flume
• [x] semantics
• [x] logstash24
• [x] datasource24
• [x] d-streamon-master
• [x] d-streamon-slave
• [x] prelude-manager-oss
• [x] prelude-correlator-oss
• [x] prewikka-oss
• [x] kafka-idmef-converter
• [x] kafka-prelude-connector
• [x] gateway
• [x] event-correlator

From Pad | Nov 22:

D-streamon-master: Functional requirements: git, cmake, g++, libboost-dev, libpcap-dev, libzmq3-dev, libpugixml-dev, sshpass, curl, build-essential, libssl-dev, mongodb-server, redis-server, ansible, Expected state: OS: Ubuntu trusty 64 Listening ports: 9999 (Master GUI), 5570 (Zmq Proxy), 3000 (swagger) Memory: At least 1024 Mb

D-streamon-slave: Functional requirements: libboost-dev, libpcap-dev, libzmq3-dev, unzip

Expected state: OS: Ubuntu trusty 64 Memory: Al least 1024 Mb

From @agil3b3ast | Nov 22:

event-correlator:

• Functional requirements: Requires python2.7, libprelude (with python bindings), prelude-tools (to register with the manager), kafka-python
• Expected state:
  • OS: CentOS 7.2
  • Listening ports: None
  • CPU, memory: Unknown
• Where to get:
  • Source at https://github.com/scissor-project/prelude-correlator.git

From @ferrarimarco | Nov 22:

gateway:

• Functional requirements: See package dependencies
• Expected state:
  • OS: Ubuntu 16.04
  • Listening ports: UDP/53 (DNS), UDP/67 (DHCP, BOOTP)
  • CPU, memory: Unknown, depends on the number of hosts to manage

From Grégory | Nov 7:

Hi Marco and everyone. [...] Here are the answers for all of my components:

prelude-manager-oss:

• Functional requirements: See package dependencies
• Expected state:
  • OS: CentOS 7.2
  • Listening ports: 3306 (mysql), 5553 (registrator), 4690 (prelude-manager)
  • CPU, memory: Unknown, depends on event stream rate.
• Where to get: Package on EPEL

prewikka-oss:

• Functional requirements: See package dependencies
• Expected state:
  • OS: CentOS 7.2
  • Listening ports: 80
  • CPU, memory: Unknown
• Where to get: Package on EPEL

kafka-idmef-converter:

• Functional requirements: Requires python2.7, libprelude (with python bindings), kafka-python
• Expected state:
  • OS: CentOS 7.2
  • Listening ports: None
  • CPU, memory: Unknown
• Note: The filtering feature has been added recently (v0.2.2)
• Where to get:
  • Source at https://gitlab.scissor-project.com/glandais/kafka-idmef-converter
  • Package at https://yum.sixsq.com/scissor/yum/

kafka-prelude-connector:

• Functional requirements: Requires python2.7, libprelude (with python bindings), prelude-tools (to register with the manager), kafka-python
• Expected state:
  • OS: CentOS 7.2
  • Listening ports: None
  • CPU, memory: Unknown
• Where to get:
  • Source at https://gitlab.scissor-project.com/glandais/kafka-prelude-connector
  • Package at https://yum.sixsq.com/scissor/yum/

prelude-correlator-oss:

• Functional requirements: Unknown
• Expected state:
  • OS: CentOS 7.2
  • Listening ports: None
  • CPU, memory: Unknown
• Where to get: Package on EPEL

From @devbrand | Nov 21:

kafka:

• Functional Requirements see package dependencies
• Expected state
  • OS: Ubuntu 16.04
  • listening ports: 9092 (kafka), 2181 (zookeeper)

flume:

• Functional Requirements see package dependencies
• Expected state
  • OS: CentOS 7.4
  • listening ports: 44444 (Parse agent), 10000 (Filter agent), 15222 (Filter agent), 20000 (Output agent)

semantics:

• Functional Requirements see package dependencies
• Expected state
  • OS: CentOS 7.4
  • listening ports: 20000 (Enrich agent), 30000 (Transform agent)

logstash:

• Functional Requirements see package dependencies
• Expected state
  • OS: Ubuntu 16.04
  • listening ports: 1(+) port for each ML integrated

logstash24:

• Functional Requirements see package dependencies
• Expected state
  • OS: Ubuntu 16.04
  • listening ports: 1(+) port for each ML integrated

datasource24:

• Functional Requirements see package dependencies
• Expected state
  • OS: Centos 7.4

[ferrarimarco]

@agil3b3ast what are the functional requirements and the expected state of event-correlator? See the examples in the first post of this issue, thanks

[ferrarimarco]

InSpec has been configured to run on each Travis build. It currently checks all the dockerfiles in docker/.

See:

• https://github.com/scissor-project/open-scissor/commit/2417995908d57e91059d5f098260214ab0469fcb
• https://github.com/scissor-project/open-scissor/commit/dd39553d567d7377be491e3c0ce7c45f8d19519f
• https://github.com/scissor-project/open-scissor/commit/b5b2c6c7008837771297021e401f52f6740934ea
• https://github.com/scissor-project/open-scissor/commit/69e969eb70babccbe9019eb1b931b93107bb656c
• https://github.com/scissor-project/open-scissor/commit/ba69526fedb6fe202343762034e0381815f00105
• https://github.com/scissor-project/open-scissor/commit/f7f6f7eb351492084f4f85b25085f15092d5fee3

Here is an example: https://travis-ci.org/scissor-project/open-scissor/builds/309482821