cloudviz / agentless-system-crawler

A tool to crawl systems like crawlers for the web
Apache License 2.0
117 stars 44 forks source link

ctprobe: upgrade to conntrackprobe v0.2 #310

Closed stefanberger closed 7 years ago

stefanberger commented 7 years ago

Conntrackprobe v0.2 implements the socket-datacollector functionality of writing received data into a file in JSON format. So we don't need to start socket-datacollector anymore when using conntrackprobe, so we have a single process monitoring network connection creations using netlink. Using conntrackprobe v0.2 has some consequences on the ctprobe plugin since it doesn't need to start socket-datacollector anymore, which allows us to remove some code. I made the code a bit more robust against killling of conntrackprobe so that it gets restarted on next crawl().

Stefan

codecov-io commented 7 years ago

Codecov Report

Merging #310 into master will increase coverage by 0.02%. The diff coverage is 90.62%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #310      +/-   ##
==========================================
+ Coverage   88.91%   88.93%   +0.02%     
==========================================
  Files         115      115              
  Lines        4581     4582       +1     
==========================================
+ Hits         4073     4075       +2     
+ Misses        508      507       -1
Impacted Files Coverage Δ
...awler/plugins/systems/ctprobe_container_crawler.py 88.31% <90.62%> (-0.39%) :arrow_down:
crawler/utils/dockerutils.py 96.68% <0%> (+0.82%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 70d5ff7...796d6b4. Read the comment docs.

sahilsuneja1 commented 7 years ago

Looks good!

  1. Do you still need original solution of softflowd + socker-datacollector, or can we remove these dependencies?
  2. Which language is Conntrackprobe written in? If python, can we make its code part of the plugin itself and avoid the .deb route?
  3. Please squash the commits together
stefanberger commented 7 years ago

@sahilsuneja1 1) I want to still keep softflowd in the case of overlapping IP addresses, which wouldn't be handled by this plugin. 2) conntrackprobe is here: https://github.ibm.com/stefanb/conntrackprobe It's a daemon itself and would be much too complicated to put into the core crawler. 3) Done.

stefanberger commented 7 years ago

@sahilsuneja1 Thanks.