Weekly testing [do not close]

[mpeuster]

This issue tracks the results of the weekly pilot tests performed by UPB.

[mpeuster]

2019-12-18

Status

• Tests have been executed up to step 7

Issues

• cannot delete existing policy because it is enforced
• sometimes IMMS seems to "hang" after a reconfiguration has happened (worked in second try)
• Automatic reconfiguration: ip0 metric is not appearing in prometheus

[mpeuster]

2020-01-08

Status

• Instantiation and manual re-configuration using the FMP works

Issues

• cannot delete existing policy because it is enforced
• sometimes IMMS seems to "hang" after a reconfiguration has happened (worked in second try)
• Automatic reconfiguration: ip0 metric is not appearing in prometheus
  • Update: ip0 metric now appears, but is not set to > 0 if intrusion happens
  • There seems to be some issue with the IDS: Kibana is not starting

[mpeuster]

2020-01-15

New issues

FMP deployed at UPB as some small glitches:

• 338

• 339

NS2 IDS:

Kibana server doesn't start: https://github.com/sonata-nfv/tng-industrial-pilot/issues/340

Automated triggering still broken

• tng-monitor is configured to use 10.200.16.2:30090 but there is nothing running in the int-ath K8s cluster. See: https://jenkins.sonata-nfv.eu/job/tng-mon-endpoints/configure

Fixes

• Updated version numbers in policies that may have caused the policy not to work: https://github.com/sonata-nfv/tng-industrial-pilot/pull/341

[stefanbschneider]

2020-01-22

Issues

Same as last week...

NS2 IDS

Kibana server doesn't start: https://github.com/sonata-nfv/tng-industrial-pilot/issues/340

Automated triggering still broken

• tng-monitor is configured to use 10.200.16.2:30090 but there is nothing running in the int-ath K8s cluster. See: https://jenkins.sonata-nfv.eu/job/tng-mon-endpoints/configure
• Thus, no ip0 metric in int-sp-ath: http://int-sp-ath.5gtango.eu:9090/graph?g0.range_input=1h&g0.expr=ip0&g0.tab=0

IMMS Traffic doesn't arrive in Aveiro platform

• If deploying NS1 and NS2 on http://sta-sp-ave-2.5gtango.eu , the ip0 metric is shown.
• When starting the IMMS and connecting to NS2 running in Aveiro, the manufacturing data/traffic only reaches NS1's Grafana/EAE with huge delay. There are over 1min timespans without any change and then jumps of over 100 parts at once (see below)
• The IMMS' logs suggest that the connection to NS2 works. I don't have access to the Aveiro kubernetes and can't check the other logs...

[stefanbschneider]

2020-01-29

Tested: (success = check)

• [x] NS1, NS2, NS3 packaging, on-boarding
• [x] NS1 instantiation with Tango Portal, NS2 with FMP, connection of IMMS
• [x] Manual isolation and deisolation with FMP (not on 1st try, see below)
• [x] NS3 instantiation with FMP
• [x] Re-instantiation of NS1, NS22 with instantitation params. Creation of Prometheus alert with Policy (after reuploading policy)
• [x] Intrusion and automatic isolation with IDS and Policies (after fix; not super stable)

New issues

• Manual isolation with FMP worked fine, but deisolation didn't. Traffic stopped arriving in quarantine, but didn't arrive in normal NS1 either.
  • Seems like it was an issue with the IMMS. The connection thread inside the IMMS was hanging/not printing any logs anymore (just the production thread). Not sure why.
  • After restarting the IMMS, it worked fine again - including isolation and deisolation. Should still try to make IMMS more robust to avoid this in the live review.
• Again, the policy didn't work and did not create the Prometheus alert when instantiating NS2
  • Works after uploading a new version of the identical policy. Apparently, this is necessary after removing and onboarding NS2. I added this to the documentation.
  • Would be useful if we could at least remove old policies in the Tango portal such that we can simply replace them without having to increment the policy version every time.
• Intrusion not detected! After trying to trigger an intrusion with smbclient, nothing happens. No intrusion is registered in Kibana or by the monitoring.
  • Fixed the VNFDs, which seems to solve the issue: https://github.com/sonata-nfv/tng-industrial-pilot/pull/347
  • New issue: IDS works, reconfiguration works, traffic stops arriving at normal NS1 and starts arriving at quarantine NS1. But: The traffic/number of parts doesn't increase anymore when arriving at quarantine NS1. Not sure if the production thread in the IMMS somehow stopped.
  • Resolved and works fine when IMMS is restarted
  • But: 1st intrusion wasn't detected by IDS. 2nd intrusion everything worked fine
• Also the monitoring in Prometheus shows the ip0 for some random, non-existing container to be greater than 0:

[stefanbschneider]

2020-02-05

Tested: (success = check)

• [x] NS1, NS2, NS3 packaging, on-boarding
• [x] NS1 instantiation with Tango Portal, NS2 with FMP, connection of IMMS
• [x] Manual isolation and deisolation with FMP
• [x] NS3 instantiation with FMP
• [x] Re-instantiation of NS1, NS22 with instantitation params. Creation of Prometheus alert with Policy
• [x] Intrusion and automatic isolation with IDS and Policies

New issues

• No new major issues
• Intrusion detection and reconfiguration of NS2 in UC2 is pretty slow (minutes after intrusion), but it works
• Kibana logs show more entries that intrusion attempts

[stefanbschneider]

2020-03-04

• UC1 with the Docker IMMS worked fine
• Also manual isolation and deisolation
• Didn't test UC2 and UC3