search

LiveRamp / reslang

A language for describing resource-oriented APIs & turning them into Swagger or resource diagrams. Oriented around the concepts we want to expose in the APIs.
Apache License 2.0
23 stars 7 forks source link

Events Update Discussions #51

Closed davinchia closed 4 years ago

davinchia commented 4 years ago

~Reslang currently generates the following:~

channels:
  rest-segment-management.v1-segment:
    description: >-

~as part of the channel name.~

~Two problems:~ ~1) Josh's script expects the channel name to begin with topics/ in order to generate terraform code. I think it's a fair design choice that simplifies validation.~ ~2) Google PubSub topic names have the following constraint:~

A name must start with a letter or underscore and may contain only letters,
digits, underscores, and dashes.

~Reslang should spit out something like this:~

channels:
  topics/rest-segment-management_v1-segment: (an underscore to separate the namespace from the resource name?)

Update: This is done by https://github.com/LiveRamp/reslang/pull/55.

davinchia commented 4 years ago

Event naming ~There can be slightly better naming for adhoc topics.~

~I generally like the namespace pattern. What about getting rid of adhoc entirely? Since the rest topic is per-resource, I'd imagine topics will be an equal mixture of rest & adhoc, making adhoc less useful.~

~Something else that came to mind - I think one big use case will be emitting events after a long-running operation is completed. Should the request-resource reslang type result in different topics?~

Update: This is done by https://github.com/LiveRamp/reslang/pull/55.

davinchia commented 4 years ago

Permissioning Josh's generation currently follows the following pattern for permissioning

channels:
  topics/datastore-segment-created:
    x-liveramp-publishers:
      - serviceAccount:dmfs-pubsub-editor@liveramp-eng-dmfs.iam.gserviceaccount.com
    x-liveramp-subscribers:
      - serviceAccount:dms-reporting-api@liveramp-eng-dmfs.iam.gserviceaccount.com
    subscribe:
      summary: Sent when a new segment is created in the Connect Data Store.
      message:
        payload:
          $ref: '#/components/schemas/segmentCreated'

It doesn't strictly follow the asyncpi channelItem spec per se, but I think it's alright.

We'd need some sort of config file reslang reads within the directory to allow easy permissioning. Something called events-permissions.yaml containing a map of topic to permissions should be sufficient.

davinchia commented 4 years ago

Global reslang file We need some sort of global reslang config file that configures the url in the generated asyncapi spec

servers:
  production:
    url: https://pubsub.googleapis.com/v1/projects/liveramp-events-prod
    protocol: http
    description: Google PubSub API for liveramp-events-prod

and the generated openapi spec

openapi: 3.0.1
info:
  title: Segment API - ALPHA
  description: ''
  version: 0.0.8
servers:
  - url: 'https://api.liveramp.com/segment-management'

This will allow us to isolate prod/staging via git branching as per the api-spec workflow.

davinchia commented 4 years ago

Select Events to emit Right now, the events keyword results in all the supported http operations being included in the generated asyncapi header in the messageTraits e.g.

  messageTraits:
    RestHeader:
      headers:
        type: object
        properties:
          verb:
            description: ''
            $ref: '#/components/schemas/VerbEnum'
          location:
            description: ''
            type: string
            format: url
            example: 'https://www.domain.com (url)'
          ids:
            description: ''
            items:
              type: string
            type: array
        required:
          - verb
          - location
          - ids

Providing a way to specify the exact http operations events are emitted for will be less confusing.

davinchia commented 4 years ago

General AsyncApi thoughts As of yesterday, confirmed that isn't any generation for google pub/sub on any of the languages we want. So we probably have to go quicktype -> serialisable models -> native pub/sub libraries (maybe, an internal pub sub library for ease of use at a later time).

Because of this, I think we should skip using the messages and the messageTraits portion of the schema and stick with just the schemas object, ala OpenAPI 3.0. This will simplify extracting the relevant fields to generate the right models.

There are also some additional metadata (the variable block) at the top we can remove:

    variables:
      port:
        description: Secure connection (TLS) is available through port 8883
        default: '1883'

Right now, this is what we generate:

# generated by Reslang v1.4.2
asyncapi: 2.0.0
info:
  title: Segment API - ALPHA
  description: ''
  version: 0.0.8
servers:
  production:
    url: 'https://pubsub.googleapis.com/v1/projects/liveramp-events-prod'
    protocol: Google Cloud Pub/Sub
    description: LiveRamp Production pubsub instance
    variables:
      port:
        description: Secure connection (TLS) is available through port 8883
        default: '1883'
defaultContentType: application/json
channels:
  topics/rest-segment-management-v1-segment:
    x-liveramp-publishers:
      - group:eng-squad-segment@liveramp.com
    x-liveramp-subscribers:
      - group:eng-squad-segment@liveramp.com
    description: >-
      A Segment represents a group of people that share some common attributes,
      for example 'people who love dogs', which can be used for various
      purposes, most commonly, for displaying ads on an Ad platform.

      Members of a segment can be represented by any type of anonymized
      identifiers supported by LiveRamp such as LiveRamp IdentityLinks, Mobile
      Ids (Apple and Android), Cookie Ids or any client-specific (Custom) Ids.

      However, a segment can only contain one type of identifiers at this time.
    subscribe:
      summary: 'REST: Segment'
      operationId: Segment
      message:
        $ref: '#/components/messages/Segment'
  topics/adhoc-segment-management-v1-segment-ingestion-completed:
    x-liveramp-publishers:
      - group:eng-squad-segment@liveramp.com
    x-liveramp-subscribers:
      - group:eng-squad-segment@liveramp.com
    description: Published whenever a segment ingestion request completes.
    subscribe:
      summary: 'Adhoc: SegmentIngestionCompleted'
      operationId: SegmentIngestionCompleted
      message:
        $ref: '#/components/messages/SegmentIngestionCompleted'
components:
  messages:
    Segment:
      name: Segment
      title: Segment
      summary: >-
        A Segment represents a group of people that share some common
        attributes, for example 'people who love dogs', which can be used for
        various purposes, most commonly, for displaying ads on an Ad platform.

        Members of a segment can be represented by any type of anonymized
        identifiers supported by LiveRamp such as LiveRamp IdentityLinks, Mobile
        Ids (Apple and Android), Cookie Ids or any client-specific (Custom) Ids.

        However, a segment can only contain one type of identifiers at this
        time.
      contentType: application/json
      traits:
        - $ref: '#/components/messageTraits/RestHeader'
      payload:
        $ref: '#/components/schemas/SegmentOutput'
    SegmentIngestionCompleted:
      name: SegmentIngestionCompleted
      title: SegmentIngestionCompleted
      contentType: application/json
      traits:
        - $ref: '#/components/messageTraits/SegmentIngestionCompletedHeader'
      payload:
        $ref: '#/components/schemas/SegmentIngestionCompleted'
  schemas:
    SegmentOutput:
      type: object
      properties:
        id:
          description: >-
            A persistent UUID, generated automatically, used to reference the
            segment.
          type: string
          format: uuid
          example: 123e4567-e89b-12d3-a456-426655440000
        name:
          description: >-
            A pretty string supplied by the client, used to reference the
            segment.
          type: string
          minLength: 1
          maxLength: 255
        identifierType:
          description: Anonymous Identifier type to identify members of the segment.
          $ref: '#/components/schemas/IdentifierTypeEnum'
        legacyAudienceID:
          description: |-
            Audience ID to maintain backwards compatibility.
                This is meant to be temporary and will be removed in a future version of this resource.
          type: integer
          format: int64
        legacyFieldID:
          description: |-
            Field ID to maintain backwards compatibility.
                This represents the Field within the legacy Audience associated with this Segment.
                This is temporary and will be removed in a future version of this resource.
          type: integer
          format: int64
        legacyValueID:
          description: |-
            Value ID to maintain backwards compatibility.
                This represents the value within the legacy Field associated with this Segment.
                This is temporary and will be removed in a future version of this resource.
          type: integer
          format: int64
        createdAt:
          description: >-
            Datetime in UTC, formatted as ISO-8601, when the segment was
            created.
          type: string
          format: ISO8601 UTC date-time
          example: '2019-04-13T03:35:34Z'
        updatedAt:
          description: >-
            Datetime in UTC, formatted as ISO-8601, when the segment's metadata
            (for example, name) or member list is updated.
          type: string
          format: ISO8601 UTC date-time
          example: '2019-04-13T03:35:34Z'
      required:
        - id
        - name
        - identifierType
        - legacyAudienceID
        - legacyFieldID
        - legacyValueID
        - createdAt
        - updatedAt
      description: >-
        A Segment represents a group of people that share some common
        attributes, for example 'people who love dogs', which can be used for
        various purposes, most commonly, for displaying ads on an Ad platform.

        Members of a segment can be represented by any type of anonymized
        identifiers supported by LiveRamp such as LiveRamp IdentityLinks, Mobile
        Ids (Apple and Android), Cookie Ids or any client-specific (Custom) Ids.

        However, a segment can only contain one type of identifiers at this
        time.
    IdentifierTypeEnum:
      type: string
      enum:
        - LIVERAMP_IDENTITY_LINK
    SegmentIngestionCompleted:
      type: object
      properties:
        segmentIngestionRequest:
          description: ''
          type: string
          format: uuid
          example: Link to SegmentIngestionRequest resource via its id
      required:
        - segmentIngestionRequest
    VerbEnum:
      type: string
      enum:
        - POST
        - PUT
        - PATCH
        - GET
        - MULTIGET
        - DELETE
  messageTraits:
    RestHeader:
      headers:
        type: object
        properties:
          verb:
            description: ''
            $ref: '#/components/schemas/VerbEnum'
          location:
            description: ''
            type: string
            format: url
            example: 'https://www.domain.com (url)'
          ids:
            description: ''
            items:
              type: string
            type: array
        required:
          - verb
          - location
          - ids
    SegmentIngestionCompletedHeader: {}

The channel reference objects in the messages block, that in turn reference objects in messageItems and schema, making it tricker to parse, combine and pipe to another tool.

I think we should combine all the schema information into the schema block and have something like this:

channels:
  topics/datastore-segment-created:
    x-liveramp-publishers:
      - serviceAccount:dmfs-pubsub-editor@liveramp-eng-dmfs.iam.gserviceaccount.com
    x-liveramp-subscribers:
      - serviceAccount:dms-reporting-api@liveramp-eng-dmfs.iam.gserviceaccount.com
    subscribe:
      summary: Sent when a new segment is created in the Connect Data Store.
      message:
        payload:
          $ref: '#/components/schemas/segmentCreated'

  topics/datastore-segment-purchased:
    x-liveramp-publishers:
      - serviceAccount:dmfs-pubsub-editor@liveramp-eng-dmfs.iam.gserviceaccount.com
    x-liveramp-subscribers:
      - serviceAccount:dms-reporting-api@liveramp-eng-dmfs.iam.gserviceaccount.com
    subscribe:
      summary: Sent when any segment is purchased in the Connect Data Store.
      message:
        payload:
          $ref: '#/components/schemas/segmentPurchased'

  topics/datastore-segment-updated:
    x-liveramp-publishers:
      - serviceAccount:dmfs-pubsub-editor@liveramp-eng-dmfs.iam.gserviceaccount.com
    x-liveramp-subscribers:
      - serviceAccount:dms-reporting-api@liveramp-eng-dmfs.iam.gserviceaccount.com
    subscribe:
      summary: Sent when any segment is updated in the Connect Data Store.
      message:
        payload:
          $ref: '#/components/schemas/segmentUpdated'

components:
  schemas:
    segmentCreated:
      type: object
      required:
        - createdAt
        - dmsSegmentId
        - fieldId
        - fullName
        - sellerCustomerId
        - valueId
      example:
        createdAt: 2020-03-16T18:25:25Z
        dmsSegmentId: 1234
        fieldId: 5678
        fullName: Nexus Megacorp, Inc.
        sellerCustomerId: 5533
        valueId: 9944
      properties:
        createdAt:
          type: string
          format: date-time
          description: Time the segment was created
        dmsSegmentId:
          $ref: '#/components/schemas/dmsSegmentId'
        fieldId:
          $ref: '#/components/schemas/fieldId'
        fullName:
          $ref: '#/components/schemas/fullName'
        sellerCustomerId:
          $ref: '#/components/schemas/sellerCustomerId'
        valueId:
          $ref: '#/components/schemas/valueId'

    segmentPurchased:
      type: object
      required:
        - buyerCustomerId
        - dmsSegmentId
        - fieldId
        - purchasedAt
        - sellerCustomerId
        - valueId
      example:
        buyerCustomerId: 4422
        dmsSegmentId: 1234
        fieldId: 5678
        purchasedAt: 2020-03-16T18:25:25Z
        sellerCustomerId: 5533
        valueId: 9944
      properties:
        buyerCustomerId:
          type: number
          description: Customer ID of Buyer from RLDB. Note this is the Customer ID, not the DmsBuyer ID
        dmsSegmentId:
          $ref: '#/components/schemas/dmsSegmentId'
        fieldId:
          $ref: '#/components/schemas/fieldId'
        purchasedAt:
          type: string
          format: date-time
          description: Time the segment was purchased
        sellerCustomerId:
          $ref: '#/components/schemas/sellerCustomerId'
        valueId:
          $ref: '#/components/schemas/valueId'

    segmentUpdated:
      type: object
      required:
        - dmsSegmentId
        - fieldId
        - fullName
        - sellerCustomerId
        - updatedAt
        - valueId
      example:
        dmsSegmentId: 1234
        fieldId: 5678
        fullName: Nexus Megacorp, Inc.
        sellerCustomerId: 5533
        updatedAt: 2020-03-16T18:25:25Z
        valueId: 9944
      properties:
        dmsSegmentId:
          $ref: '#/components/schemas/dmsSegmentId'
        fieldId:
          $ref: '#/components/schemas/fieldId'
        fullName:
          $ref: '#/components/schemas/fullName'
        sellerCustomerId:
          $ref: '#/components/schemas/sellerCustomerId'
        updatedAt:
          type: string
          format: date-time
          description: Time the segment was updated
        valueId:
          $ref: '#/components/schemas/valueId'

    dmsSegmentId:
      type: number
      description: Dms Segment ID from RLDB

    fieldId:
      type: number
      description: Field ID of DmsSegment from RLDB

    fullName:
      type: string
      description: The full name of the segment

    sellerCustomerId:
      type: number
      description: Customer ID of Seller from RLDB. Note this is the Customer ID, not the DmsProvider ID

    valueId:
      type: number
      description: Value ID of DmsSegment from RLDB
liveandrew commented 4 years ago

all fixed here: https://github.com/LiveRamp/reslang/pull/94