search

sonata-nfv / tng-industrial-pilot

5GTANGO Smart Manufacturing Pilot
13 stars 17 forks source link

VNF Intrusion detection system #25

Closed danielbehnke closed 5 years ago

danielbehnke commented 6 years ago

Detect all attacks on machine network, created once per machine park

stefanbschneider commented 5 years ago

@arocha7 @miguelmesquita Let's use this issue to share and align updates on the development and integration of the IDS into NS2.

Wiki entry for documentation of IDS deployment: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/IDS-deployment

Descriptors for native k8s deployment: https://github.com/sonata-nfv/tng-industrial-pilot/tree/master/k8s

stefanbschneider commented 5 years ago

@arocha7 @miguelmesquita

The logstash container is crashing in between:

eu-5gtango-lh-vnf2-0-4-6929a5c3-69b679697-mqd9m        2/2     Running             3          4m40s

After a few initial crashes, it now seems to run quite stable.

Logs give me this error message:

$ kubectl logs eu-5gtango-lh-vnf2-0-4-6929a5c3-69b679697-mqd9m cdu01-6929a5c3-1758-45ac-ae11-7a5401b83c9d
10.104.147.220 10.107.209.98
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-09-18T06:46:53,877][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2019-09-18T06:46:53,898][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2019-09-18T06:46:54,863][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.3.1"}
[2019-09-18T06:46:54,929][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"21a4aa16-f747-4f43-8c2d-abc97467855d", :path=>"/usr/share/logstash/data/uuid"}
[2019-09-18T06:46:57,393][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Illegal character in authority at index 7: http://10.104.147.220\n10.107.209.98:9200"}
[2019-09-18T06:46:57,452][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2019-09-18T06:47:03,529][INFO ][org.reflections.Reflections] Reflections took 803 ms to scan 1 urls, producing 19 keys and 39 values
[2019-09-18T06:47:10,395][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaNet::URISyntaxException", :message=>"Illegal character in authority at index 7: http://10.104.147.220\n10.107.209.98:9200", :backtrace=>["java.net.URI$Parser.fail(java/net/URI.java:2915)", "java.net.URI$Parser.parseAuthority(java/net/URI.java:3249)", "java.net.URI$Parser.parseHierarchical(java/net/URI.java:3160)", "java.net.URI$Parser.parse(java/net/URI.java:3116)", "java.net.URI.<init>(java/net/URI.java:600)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(jdk/internal/reflect/NativeConstructorAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:490)", "org.jruby.javasupport.JavaConstructor.newInstanceDirect(org/jruby/javasupport/JavaConstructor.java:247)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.util.safe_uri.initialize(/usr/share/logstash/logstash-core/lib/logstash/util/safe_uri.rb:21)", "usr.share.logstash.logstash_minus_core.lib.logstash.util.safe_uri.RUBY$method$initialize$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/util//usr/share/logstash/logstash-core/lib/logstash/util/safe_uri.rb)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate_value(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:514)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$validate_value$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/config//usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.process_parameter_value(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:329)", "org.jruby.RubyArray.collect(org/jruby/RubyArray.java:2563)", "org.jruby.RubyArray.map(org/jruby/RubyArray.java:2577)", "org.jruby.RubyArray$INVOKER$i$0$0$map19.call(org/jruby/RubyArray$INVOKER$i$0$0$map19.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.process_parameter_value(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:329)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate_check_parameter_values(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:352)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate_check_parameter_values(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:346)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1438)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1435)", "org.jruby.RubyHash.visitLimited(org/jruby/RubyHash.java:690)", "org.jruby.RubyHash.visitAll(org/jruby/RubyHash.java:675)", "org.jruby.RubyHash.iteratorVisitAll(org/jruby/RubyHash.java:1395)", "org.jruby.RubyHash.each_pairCommon(org/jruby/RubyHash.java:1430)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1419)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate_check_parameter_values(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:345)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$validate_check_parameter_values$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/config//usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.validate(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:235)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$validate$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/config//usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.config_init(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:86)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$config_init$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/config//usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.outputs.base.initialize(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:60)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:894)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:798)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:363)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.initialize(org/logstash/config/ir/compiler/OutputStrategyExt.java:232)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt$INVOKER$i$1$0$initialize.call(org/logstash/config/ir/compiler/OutputStrategyExt$SimpleAbstractOutputStrategyExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.logstash.config.ir.compiler.OutputDelegatorExt.initialize(org/logstash/config/ir/compiler/OutputDelegatorExt.java:48)", "org.logstash.config.ir.compiler.OutputDelegatorExt.initialize(org/logstash/config/ir/compiler/OutputDelegatorExt.java:30)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:242)", "org.logstash.plugins.PluginFactoryExt$Plugins.buildOutput(org/logstash/plugins/PluginFactoryExt.java:140)", "org.logstash.config.ir.CompiledPipeline.lambda$setupOutputs$0(org/logstash/config/ir/CompiledPipeline.java:115)", "java.util.ArrayList.forEach(java/util/ArrayList.java:1540)", "org.logstash.config.ir.CompiledPipeline.setupOutputs(org/logstash/config/ir/CompiledPipeline.java:112)", "org.logstash.config.ir.CompiledPipeline.<init>(org/logstash/config/ir/CompiledPipeline.java:82)", "org.logstash.execution.JavaBasePipelineExt.initialize(org/logstash/execution/JavaBasePipelineExt.java:50)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(org/logstash/execution/JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.initialize(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.execute(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/pipeline_action//usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.converge_state(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"]}
[2019-09-18T06:47:10,425][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaNet::URISyntaxException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:338:in `block in converge_state'"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaNet::URISyntaxException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:109
             add at org/logstash/execution/ConvergeResultExt.java:37
  converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:338
[2019-09-18T06:47:10,518][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaNet::URISyntaxException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:338:in `block in converge_state'"]}
[2019-09-18T06:47:10,751][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Can't access Kibana:

Kibana pods and service are running fine after instantiation, but I can't access the dashboard. If I try to curl, I get connection refused:

# kubectl get svc
eu-5gtango-k-vnf4-0-4-bcb033cb        LoadBalancer   10.102.132.91    10.200.16.61   5601:31552/TCP                               6m7s
# trying to access
$ curl 10.200.16.61:5601
curl: (7) Failed to connect to 10.200.16.61 port 5601: Connection refused
stefanbschneider commented 5 years ago

Is an issue due to the connection of the CNFs. The wrong env vars, with capital letters, are used. These inculde the IP and port of all running instances, so the CNF can't distinguish which env vars belong to the current instance.

Instead, use the lower case env vars, which are injected by the MANO individually for each deployment. That should resolve the issue with the 2 instances.

stefanbschneider commented 5 years ago

Another issue is that the alarm is always triggered - no matter how the env vars for the filter are set.

stefanbschneider commented 5 years ago

This is done