search

openfaas-incubator / vcenter-connector

Extend vCenter with OpenFaaS
MIT License
28 stars 8 forks source link

Standardize on event spec incl. event payload #30

Closed embano1 closed 2 years ago

embano1 commented 4 years ago

Problem

The current event structure used is rather simple, does not conform to any specification (e.g. CloudEvents) and misses the vCenter event payload (body).

type OutboundEvent struct {
    Topic    string `json:"topic,omitempty"`
    Category string `json:"category,omitempty"`
    Source   string `json:"source"`

    UserName               string                         `json:"userName,omitempty"`
    CreatedTime            time.Time                      `json:"createdTime,omitempty"`
    ObjectName             string                         `json:"objectName,omitempty"`
    ManagedObjectReference *vtypes.ManagedObjectReference `json:"managedObjectReference,omitempty"`
}

Solution Proposal

In order to support richer events a breaking change is required. This issue explores a potential solution using the CloudEvents v1 spec. Following the spec, data would carry the vCenter event payload (encoded as JSON via datacontenttype) as received (i.e. unmodified) from vCenter. type is a combination of com.openfaas.vcenter-connector and the corresponding event, e.g. vm.created. id is a UUID which can be used for idempotency semantics (detect duplicates) by combining source + id.

Source would map to the vCenter instance which created the event. Subject is the canonical name of the event (CamelCase syntax). time is the event timestamp created by vcenter-connector. data.CreatedTime is the event timestamp created by the corresponding vCenter.

{
  "specversion": "1.0",
  "type": "com.openfaas.vcenter-connector.vm.created",
  "source": "vcenter1.org.example.com",
  "subject": "VmCreatedEvent",
  "id": "A234-1234-1234",
  "time": "2019-11-22T19:26:48.123456Z",
  "datacontenttype": "application/json",
  "data": {
    "Key": 20877,
    "ChainId": 20873,
    "CreatedTime": "2019-11-22T19:26:47.781672Z",
    "UserName": "VSPHERE.LOCAL\\Administrator",
    "Datacenter": {
      "Name": "vcqaDC",
      "Datacenter": {
        "Type": "Datacenter",
        "Value": "datacenter-2"
      }
    },
    "ComputeResource": {
      "Name": "cls",
      "ComputeResource": {
        "Type": "ClusterComputeResource",
        "Value": "domain-c7"
      }
    },
    "Host": {
      "Name": "10.10.10.1",
      "Host": {
        "Type": "HostSystem",
        "Value": "host-21"
      }
    },
    "Vm": {
      "Name": "test-01",
      "Vm": {
        "Type": "VirtualMachine",
        "Value": "vm-69"
      }
    }
  }
}

Benefits

Impact to existing Users

cc/ @martindekov @alexellis

martindekov commented 4 years ago

So as I understand we should create a struct by the specification provided by CloudEvents? Is there a library which we can vendor, or we just have to augment the already existing OutboundEvent structure so it obeys the specification?

In general I like the idea of specifications This potentially can be made as a library to be used by other people for Go, if such does not exist yet and(saw sdk exists) also we can think of following specifications for the other connectors.

If it is breaking change, is the CloudEvents the most popular out there? I suppose we want to go with the most popular/used specification for events there is.

martindekov commented 4 years ago

Me and @embano1 had a fast chat and reached a conclusion that

augment the already existing OutboundEvent structure so it obeys the specification

would be the better way to go as it won't vendor tons of code for something as simple as following a specification for a struct.

embano1 commented 4 years ago

Here's my proposal for the new event structure:

// outboundEvent is the JSON object sent to subscribed functions
// We follow CloudEvents v1.0 event specification
type outboundEvent struct {
    // ID of the event; must be non-empty and unique within the scope of the producer.
    ID string `json:"id"`
    // Source - Hostname of the event producer (vcenter).
    Source string `json:"source"`
    // SpecVersion - The version of the CloudEvents specification used by the event.
    SpecVersion string `json:"specversion"`
    // Type - canonicalType + vcenter event category (dot notation)
    Type string `json:"type"`
    // Subject - vcenter event category (CamelCase)
    Subject string `json:"subject"`
    // Time - Timestamp set by vcenter connector when this message was created.
    Time time.Time `json:"time"`
    // Data - Event payload as received from vcenter (includes event creation timestamp set by vcenter)
    Data vtypes.BaseEvent `json:"data,omitempty"`
}

Note: The related v1.0 CloudEvents spec can be found here (with some minor deviations): https://github.com/cloudevents/sdk-go/blob/241be38f57d14ef1906fd5ad3c47bbee3851ae91/pkg/cloudevents/event.go#L11

Using our canonical vm.powered.on example this would be the outboundMessage received by any subscribed function:

{
  "id": "583b2d1b-d59f-4c40-8152-30a8cbd9726a",
  "source": "vcenter-01.example.local",
  "specversion": "1.0",
  "type": "com.github.openfaas-incubator.vcenter-connector.vm.powered.on",
  "subject": "VmPoweredOnEvent",
  "time": "2019-11-26T19:52:51.04247Z",
  "data": {
    "Key": 1020,
    "ChainId": 1017,
    "CreatedTime": "2019-11-26T19:52:49.813879Z",
    "UserName": "VSPHERE.LOCAL\\Administrator",
    "Datacenter": {
      "Name": "vcqaDC",
      "Datacenter": {
        "Type": "Datacenter",
        "Value": "datacenter-2"
      }
    },
    "ComputeResource": {
      "Name": "cls",
      "ComputeResource": {
        "Type": "ClusterComputeResource",
        "Value": "domain-c7"
      }
    },
    "Host": {
      "Name": "10.10.10.1",
      "Host": {
        "Type": "HostSystem",
        "Value": "host-21"
      }
    },
    "Vm": {
      "Name": "vm-01",
      "Vm": {
        "Type": "VirtualMachine",
        "Value": "vm-56"
      }
    },
    "Ds": null,
    "Net": null,
    "Dvs": null,
    "FullFormattedMessage": "vm-01 on  10.10.10.1 in vcqaDC is powered on",
    "ChangeTag": "",
    "Template": false
  }
}

If we have consensus on this structure I'll file a PR where we can discuss the code changes and types used in the message body.

Regarding backwards compatibility and breaking changes: I'd rather make this breaking change now where users can decide when to upgrade. I suspect after the announcements around the vCenter Event Broker Appliance (VEBA) the adoption rate is growing quickly.

cc/ @lamw please also take a look

lamw commented 4 years ago

LGTM

martindekov commented 4 years ago

LGTM++

alexellis commented 4 years ago

This is the first time I'm seeing this request which Michael mentioned on Slack and seemed to have the impression had been agreed upon. That's not the case, but I'll take another look after the holidays.