Tiles for supported webhook integations

[johanstokking]

Summary

Tiles for supported webhook integrations.

Why do we need this?

To show recognizable logos for users of upstream platforms and make configuration easier.

What is already there? What do you see now?

Manual webhook configuration, i.e. URL, headers, format.

What is missing? What do you want to see?

Tiles that provide URL, headers and format.

How do you propose to implement this?

1. AS file based configuration of webhook templates and API - @adriansmares
   • id
   • name, description, logo, info url, documentation url
   • format to use
   • url with path variables (for app ID, device ID, DevEUI)
   • field for user security token that can be used in various ways (e.g. Authorization header with optional prefix). Also the name of the user security token (i.e. "Subscription ID", "API Key")
2. AS rpcs to serve webhook templates (part of pkg/applicationserver/io/web) - @adriansmares
3. Console support for fetching templates, rendering tiles and asking user security token - @kschiffer

We should keep the template ID used in the application webhook so that we can render the logo of the webhook. Apart from that, for editing and deleting the webhook, I would suggest using the existing edit/delete functionality, so that the template is only used for creating the webhook.

Can you do this yourself and submit a Pull Request?

Can review

[adriansmares]

As mentioned before, the Application Server part of the implementation has been finished in #1011.

The templates are served by the ApplicationWebhookRegistry service and follow the structure of the ApplicationWebhookTemplate proto. The main takeaways from the templates are the following:

• They represent standalone integrations such as TTN-Mapper, Collos etc.
• They provide the base_url, headers, format and paths for the webhook.
• In order to use a template, one must fill the values of its fields. After the fields have been filled, they are replaced in the base_url, headers and format and the webhook gets created normally.

Fields represents a custom field that needs to be filled by the user in order to use the template. A field can be an API key, an username or password, or any custom platform specific field (such as region). They have to be replaced inside the webhook using RFC 6570 format.

Taking the following example template:

applicationwebhooktemplateidentifiers:
  templateid: example
name: Example
description: Example template
logourl: example.svg
infourl: https://example.com/
documentationurl: https://www.thethingsnetwork.org/docs/applications/example/
baseurl: https://thethingsnetwork.example.com/v3{?env,create,access_token}
headers: {}
format: json
fields:
- id: env
  name: Region
  description: The platform region
  secret: false
  defaultvalue: ""
- id: create
  name: Create device flag
  description: If true, the device will be automatically created, otherwise false
  secret: false
  defaultvalue: "true"
- id: access_token
  name: Trusted Application API key
  description: Get this key from the application page on Example.com
  secret: true
  defaultvalue: ""
uplinkmessage:
  path: ""
joinaccept: null
downlinkack: null
downlinknack: null
downlinksent: null
downlinkfailed: null
downlinkqueued: null
locationsolved: null

The user needs to fill the 3 fields ( env, create and access_token ). Let's consider that he entered env as us, create as true and access_token as real-token-here. Then the console needs to create an webhook with base_url = https://thethingsnetwork.example.com/v3?env=us&create=true&access_token=real-token-here, empty headers, json format and with an empty uplink message path.

[johanstokking]

Pushing this to December

[johanstokking]

@adriansmares can you point to the gRPC services and messages and provide info to populate a registry locally via file system? If that is not ready yet, please set blocked with reference and potentially push to Next Up

[adriansmares]

• Create a repository folder on your file system. I've named mine webhook-templates in the current directory.
• Create the templates index file called templates.yml in the repository folder with the following contents:

• example

• Create a template file called example.yml in the repository folder with the following contents:

  template-id: example
  name: Example Template
  description: A simple webhook template
  logo-url: https://upload.wikimedia.org/wikipedia/commons/3/30/Vector-based_example.svg
  info-url: https://www.example.com
  documentation-url: https://www.iana.org/domains/reserved
  fields:
  - id: region
  name: Server region
  description: On which region should the service endpoint land on
  secret: false
  default-value: eu
  format: json
  headers:
  Authorization: Bearer xyz
  create-downlink-api-key: true
  baseurl: https://www.example.com{/region}
  paths:
  uplink-message: /uplink

• Update your stack configuration to point to your local repository. YAML configuration example:

  as:
  webhooks:
  templates:
    directory: ./webhook-templates

  The above setting can also be set using the command line switch --as.webhooks.templates.directory.

After starting the TTS, you should be able to retrieve both (all) the templates and specific templates in the following manner:

# Listing templates with pretty printing
$ curl http://localhost:1885/api/v3/as/webhook-templates?field_mask=description,logo_url,info_url,documentation_url,fields,format,headers,create_downlink_api_key,base_url,uplink_message | json_pp
{
   "templates" : [
      {
         "description" : "A simple webhook template",
         "format" : "json",
         "logo_url" : "https://upload.wikimedia.org/wikipedia/commons/3/30/Vector-based_example.svg",
         "create_downlink_api_key" : true,
         "uplink_message" : {
            "path" : "/uplink"
         },
         "name" : "Example Template",
         "fields" : [
            {
               "default_value" : "eu",
               "id" : "region",
               "name" : "Server region",
               "description" : "On which region should the service endpoint land on"
            }
         ],
         "info_url" : "https://www.example.com",
         "documentation_url" : "https://www.iana.org/domains/reserved",
         "headers" : {
            "Authorization" : "Bearer xyz"
         },
         "ids" : {
            "template_id" : "example"
         }
      }
   ]
}
# Getting individual templates with pretty printing
$ curl http://localhost:1885/api/v3/as/webhook-templates/example?field_mask=description,logo_url,info_url,documentation_url,fields,format,headers,create_downlink_api_key,base_url,uplink_message | json_pp
{
   "documentation_url" : "https://www.iana.org/domains/reserved",
   "description" : "A simple webhook template",
   "logo_url" : "https://upload.wikimedia.org/wikipedia/commons/3/30/Vector-based_example.svg",
   "create_downlink_api_key" : true,
   "uplink_message" : {
      "path" : "/uplink"
   },
   "headers" : {
      "Authorization" : "Bearer xyz"
   },
   "name" : "Example Template",
   "format" : "json",
   "ids" : {
      "template_id" : "example"
   },
   "info_url" : "https://www.example.com",
   "fields" : [
      {
         "default_value" : "eu",
         "name" : "Server region",
         "description" : "On which region should the service endpoint land on",
         "id" : "region"
      }
   ]
}

[kschiffer]

I'm currently thinking about where to display the tiles in the current menu structure. I see three options:

1. Add another + Add Webhook from template button next to the + Add Webhook button
2. Add a subview after clicking + Add Webhook, where the user will see the tiles and an additional tile, like Add blank webhook

Would currently lean towards 1, but not sure. Input welcome.

[adriansmares]

I think 2 would provide a more clear UX (it removes the confusion between template/no template).

[kschiffer]

Regarding the field prop of the webhook template: I need additional information revealing the type of the field. E.g. there are flags that should be rendered as checkboxes in the console. I need to be able to detect that:

- id: env
  name: Region
  description: The platform region
  secret: false
  defaultvalue: ""
  type: "string"
- id: create
  name: Create device flag
  description: If true, the device will be automatically created, otherwise false
  secret: false
  defaultvalue: "true"
  type: "boolean"

There might be other types necessary (enums, numbers, etc)

Can we add that @adriansmares ?

Also note that with the current implementation, we're not able to provide i18n.

[kschiffer]

Another question: During creation, should we provide a special webhook form that is limited to these template fields, or should we use something like the current normal form but with prefilled values?

[adriansmares]

The form should contain only the template fields. Users should provide only the values of the fields, and the underlying webhook is automatically instanced by the Console.

Regarding the field types: This is tricky. The idea is that the value should be replaceable directly into URLs and headers. Introducing types makes the whole system more complicated. Booleans (checkboxes) need to have both a true value and a false value, enums need to have a value associated with them, and writing the templates becomes slightly more complicated. Outside of the (very) rare cases where flags are needed, most of the fields are meant to be just strings.

[kschiffer]

The form should contain only the template fields. Users should provide only the values of the fields, and the underlying webhook is automatically instanced by the Console.

👌

Regarding the field types: […]

Still, we would pass on the problem to the console, where users will be able to fill boolean-type fields with invalid values. Would it not be possible to add a boolean flag only, which only uses true and false?

[kschiffer]

Which of those template props can I assume to be mandatory? Will I always have a logo, documentation and info URL, name and description?

And how should the field strings be validated? Should we be permissive there or have some sort of URI sanitizing? Should I assume all fields to be required?

Can the fields also be referenced in the header entries?

[adriansmares]

Which of those template props can I assume to be mandatory? Will I always have a logo, documentation and info URL, name and description?

All fields are mandatory, so yes, these should be there at all times.

And how should the field strings be validated? Should we be permissive there or have some sort of URI sanitizing? Should I assume all fields to be required?

I believe we should be permissive here. I'll probably create an issue to a regex validation field, and we can implement this in the future.

Can the fields also be referenced in the header entries?

Yes. No fancy RFC for it tho, just string replace {field_name} with value.

[kschiffer]

Regarding the base_url: https://thethingsnetwork.example.com/v3{?env,create,access_token} What other formats would be possible? Or can I just assume that all fields get added to the URL as query parameters?

[adriansmares]

The idea is to support the whole RFC 6570 URI template format. An external library would probably fit best here. I found this and this, but I leave the choice up to you, if you can find a library that's better/at least maintained (the ones mentioned above haven't seen updates in quite a long time).

[kschiffer]

Ah, no that's fine. Missed the bit about RFC 6570. One of those libs should be good.