search

sonata-nfv / tng-industrial-pilot

5GTANGO Smart Manufacturing Pilot
13 stars 17 forks source link

SP-using-K8s deployment of pilot #76

Closed mpeuster closed 5 years ago

mpeuster commented 5 years ago

Based on the native K8s services, and the emulator services, create a version of the pilot that can be deployed by the 5GTANGO SP on K8s.

See Felipe's initial work #75

See K8s-native issue #72

See updated Dockerfiles and descriptors: #77

stefanbschneider commented 5 years ago

@felipevicens If the SP starts all deployments, pods, services in K8s with vendor-name-version-UUID, then we need some mechanism (through environmental variables), where the SP tells the CDUs of the service during instantiation what their service UUID will be. We need this for example such that the CC processor (MQTT client) can connect to the CC broker (MQTT broker), based on its k8s service name.

See https://github.com/sonata-nfv/tng-industrial-pilot/issues/89#issuecomment-466917855

stefanbschneider commented 5 years ago

Just updated the concept to make it even easier. See wiki.

Basically, the SP creates separate k8s services for each pod/CDU and chooses these service names however it likes (e.g., including UUIDs). It then passes the complete k8s service names of all CDUs as environmental variables when starting a service. Each container/CDU/pod should get a list of environmental variables that map the vendor.name.version.cdu_id of each CDU in the NS to the corresponding k8s service name.

felipevicens commented 5 years ago

Hello Stefan, I've included the feature in k8s wrapper #19. I've noticed that instance_uuid is just for the VNF that we are deploying and not for the rest of VNF. Since wrapper doesn't have NS knowledge, we need to think in a way on how to pass the information. Also ENV vars for CNFs is ready here: wiki

stefanbschneider commented 5 years ago

Perfect, that was quick! Maybe we can use the env vars for CNFs when starting NS2 after NS1 to refer to NS1.

But we still need a solution for passing the k8s service names of all CDUs within a NS that are started together. I think the best and currently only way I can think of is to adjust the SP to provide info about NS to the IA and the K8s wrapper. Let's discuss this in gitter :)

Probably need to involve @tsoenen for this.

stefanbschneider commented 5 years ago

Updated wiki page according to call: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/Integration-with-SP

Main agreements:

Todos:

stefanbschneider commented 5 years ago
stefanbschneider commented 5 years ago

I just updated the NSDs and VNFDs according to Felipe's example.

We still need to somehow pass information about the k8s service names to the CNFs. This is needed, for example, in the EAE to connect to the CC's Prometheus DB: https://github.com/sonata-nfv/tng-industrial-pilot/blob/master/sdk-projects/tng-smpilot-ns1-k8s/Definitions/vnfd/vnfd_eae.yml#L43

felipevicens commented 5 years ago

@stefanbschneider I've found an issue in the k8s wrapper deploying ns1. I let you know when bug is fixed.

https://github.com/sonata-nfv/tng-sp-ia-k8s/issues/23

stefanbschneider commented 5 years ago

So, deployment with the SP on k8s works! But it still requires some manual configuration. We still need to connect the CNFs automatically using environmental variables as described.

@tsoenen already provided the functionality in the SP: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/Integration-with-SP#update-250319-implementation-in-the-sp

We now just have to adjust the CNFs to make use of these new environmental variables and test if it works.

stefanbschneider commented 5 years ago

@tsoenen @felipevicens

I just checked the available env vars inside the pods started by the SP on Kubernetes:

root@eu-5gtango-smpilot-eae-0-1-e74bd4c4-b797f8677-9t5lh:/usr/share/grafana# printenv | grep EU_
EU_5GTANGO_SMPILOT_EAE_0_1_E74BD4C4_SERVICE_HOST=10.107.98.244
EU_5GTANGO_SMPILOT_EAE_0_1_E74BD4C4_PORT_3000_TCP_PORT=3000
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_PORT_1883_TCP=tcp://10.100.137.145:1883
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_SERVICE_HOST=10.100.137.145
EU_5GTANGO_SMPILOT_EAE_0_1_E74BD4C4_SERVICE_PORT=3000
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_PORT_9090_TCP_PORT=9090
EU_5GTANGO_SMPILOT_EAE_0_1_E74BD4C4_PORT_3000_TCP_ADDR=10.107.98.244
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_SERVICE_PORT_PROMETHEUS=9090
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_SERVICE_PORT=1883
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_PORT_9090_TCP_PROTO=tcp
EU_5GTANGO_SMPILOT_EAE_0_1_E74BD4C4_SERVICE_PORT_GRAFANA=3000
EU_5GTANGO_SMPILOT_EAE_0_1_E74BD4C4_PORT=tcp://10.107.98.244:3000
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_PORT_1883_TCP_PROTO=tcp
EU_5GTANGO_SMPILOT_EAE_0_1_E74BD4C4_PORT_3000_TCP=tcp://10.107.98.244:3000
EU_5GTANGO_SMPILOT_EAE_0_1_E74BD4C4_PORT_3000_TCP_PROTO=tcp
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_SERVICE_PORT_MQTT=1883
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_PORT_9090_TCP_ADDR=10.100.137.145
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_PORT_1883_TCP_ADDR=10.100.137.145
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_PORT=tcp://10.100.137.145:1883
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_PORT_1883_TCP_PORT=1883
EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_PORT_9090_TCP=tcp://10.100.137.145:9090

This is a bit different than what I expected. Most importantly, the name of the env vars isn't static but seems to contain dynamic information that is unknown to the developer. The idea was to have predefined names for the env vars, which are fixed and which can be accessed by the CNF in a consistent way.

For example, to connect the EAE to the CC, we need to know it's IP address. Currently this is available under the env var EU_5GTANGO_SMPILOT_CC_0_1_D8E33112_SERVICE_HOST=10.100.137.145. Where does the D8E33112 come from and how would the EAE know this up front? I think the key of the env var should be sth like EU_5GTANGO_SMPILOT_CC_0.1_SERVICE_HOST, i.e., without any dynamic stuff in there.

Nothing urgent, just future work :)

tsoenen commented 5 years ago

Hi @stefanbschneider

The env vars need a dynamic part, i.e. in this case the first 8 chars of the cnf instance uuid, because there can be multiple such cnfs. If we have two instances of the same cnf, their env vars would be identical if they only contained a static part and it would not be possible to pass along both.

anyway, the way the env vars are build is deterministic. So if you do a .split('_') in python, the third element in the resulting list will always contain the pilot name, the 4th the cnf name, etc.

stefanbschneider commented 5 years ago

Ah true, I forgot about multiple instances of the same CNF. We'll work with the prefix/split parts then :+1: