maheshbabugorantla commented 4 years ago

ElasticSearch Emitter

Resolves Issue #392

Summary of Changes

Added Elasticsearch Emitter
Added UnitTest to test formatting of crawler frames into elasticsearch document format

Results

Environment Setup

$ docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
e96b77434155        kibana:7.4.2          "/usr/local/bin/dumb…"   21 hours ago        Up 21 hours         0.0.0.0:5601->5601/tcp                           tender_dijkstra
a658b6357af5        elasticsearch:7.4.2   "/usr/local/bin/dock…"   6 months ago        Up 22 hours         0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   es_emitter_test

Crawling and Indexing frames in INVM Crawl Mode

$ sudo venv/bin/python crawler/crawler.py --url elastic://localhost:9200 --features os,disk,process,package --extraMetadata '{"iteration_number": 1, "hostname": "my_ubuntu_1804"}' --format json

$ sudo venv/bin/python crawler/crawler.py --url elastic://localhost:9200 --features os,disk,process,package --extraMetadata '{"iteration_number": 2, "hostname": "my_ubuntu_1804"}' --format json

Kibana Query (Filtering by extraMetadata fields)

hostname : "my_ubuntu_1804" and iteration_number : "1" elastic_emitter_iteration_1

hostname : "my_ubuntu_1804" and iteration_number : "2" elastic_emitter_iteration_2

sahilsuneja1 commented 4 years ago

Test environment setup failing. Maybe need to update https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/ to https://download.docker.com/linux/ubuntu/dists/trusty/stable/binary-amd64/ in https://github.com/cloudviz/agentless-system-crawler/blob/master/.travis.yml#L22 as per https://github.com/moby/moby/issues/16941 Will look into it later.