Migration: replicating the request body of legacy webhook actions

[rowanseymour]

These currently send our own defined format of run steps, whereas the new call_webhook action has a body template.

[nicpottier]

Ya.. there's a bit of work required to allow us to fully represent our current format using a template, but also a good test that we are exposing everything necessary for other callers so worth it.

I think the biggest thing is figuring out how to represent say the path like we do in the old world and what that would look like. Maybe we need filter and map in expressions?

[rowanseymour]

What makes it extra challenging is that much of the data doesn't exist in goflow. The payload we currently send looks like:

dict(channel=channel.id if channel else -1,
     channel_uuid=channel.uuid if channel else None,
     relayer=channel.id if channel else -1,
     flow=flow.id,
     flow_uuid=flow.uuid,
     flow_name=flow.name,
     flow_base_language=flow.base_language,
     run=run.id,
     text=text,
     attachments=[a.url for a in attachments],
     step=six.text_type(node_uuid),
     phone=contact.get_urn_display(org=org, scheme=TEL_SCHEME, formatted=False),
     contact=contact.uuid,
     contact_name=contact.name,
     urn=six.text_type(contact_urn),
     values=json.dumps(values),
     steps=json.dumps(steps),
     time=json_time,
     header=header)

where each step looks like:

dict(type=step.step_type,
     node=step.step_uuid,
     arrived_on=datetime_to_str(step.arrived_on),
     left_on=datetime_to_str(step.left_on),
     text=step.get_text(),
     value=step.rule_value)

It's sent as urlencoded (?) form data tho steps and values are JSON encoded within that.

I'm leaning toward we introduce v2 webhooks now that have customizable payloads with only context items that we can migrate.

[rowanseymour]

Actually come to think of it, v2 webhooks in old world don't need to have customizable payloads, they just need to have payloads we know we can migrate in the new world, i.e. no database ids, no step types

[nicpottier]

What do you mean by it not existing in goflow? Don't all those fields exist (even if not in the context currently) in the new world?

[rowanseymour]

We don't send database id fields or have a concept of step_types. I'm all for adding cool mapping functions to excellent but I'm not completely convinced we can replicate this madness in an expression.

[nicpottier]

Ya, step type especially.. but how do we move everyone off then? Just settle on a new payload and send out a notification that this will be changed on x date?

[rowanseymour]

I think so tho we'd have to support both payloads for a while to give people a chance to migrate. Maybe have a v1/v2 dropdown on the action in the editor?

Arguably we should have done this a while ago since the current payload has accumulated quite a bit of cruft.

Question is do we add support for a completely customizable request body like in the new engine, or settle on a fixed schema that we can replicate as a customizable request body in the new engine. Old engine expressions context doesn't expose as much as new engine, and old engine functions definitely don't support rendering lists of steps and values etc.

It would be nice if webhook actions didn't have to send the complete run history every time - I can't imagine most people are using that.

[nicpottier]

Seems easiest for users would be to pick something similar to the current thing but what we could support in the new engine, something that we think for most users would require no action. And just make that the new "default" in the old world.

Ideally that would be the same default as the new world but I don't think it has to be since we can just populate it appropriately.

[rowanseymour]

See https://github.com/nyaruka/goflow/blob/master/flows/definition/legacy.go#L34. Still have some challenges if we want to match the old engine format exactly when migrating old flows.

[rowanseymour]

Here's what a v2 webhook payload from one of our unit tests in rapidpro now looks like for reference:

{
  "run": {
    "created_on": "2018-04-17T14:05:30.162624+00:00",
    "uuid": "89b97d17-5efa-47fa-aaa7-04689baa4e04"
  },
  "flow": {
    "uuid": "359eef6f-70c4-4a55-b4c6-afad87a12f38",
    "name": "Webhook Payload Test"
  },
  "results": {
    "age": {
      "category": "numeric",
      "node_uuid": "8787e026-32d8-41a2-8627-70250b0c7195",
      "name": "Age",
      "value": "39",
      "created_on": "2018-04-17T14:05:30.260824+00:00",
      "input": "39"
    },
    "disaster": {
      "category": "Tornado",
      "node_uuid": "2aac3729-3aba-4cf3-bc70-8462246518c2",
      "name": "Disaster",
      "value": "tornado",
      "created_on": "2018-04-17T14:05:30.348581+00:00",
      "input": "tornado"
    }
  },
  "contact": {
    "urn": "tel:+12065552020",
    "name": "Ben Haggerty",
    "uuid": "7bff201f-ba0d-4bfe-bac5-7b5c736d0624"
  },
  "input": {
    "text": "tornado",
    "urn": "tel:+12065552020",
    "attachments": []
  },
  "path": [
    {
      "uuid": "e10c356e-1525-415a-b63f-8684e7e36a43",
      "node_uuid": "673c85ec-3c46-4db9-b3ca-482c3d4f1272",
      "arrived_on": "2018-04-17T14:05:30.172138+00:00",
      "exit_uuid": "482e40ff-b3e4-437c-91c2-51bdf6e9279e"
    },
    {
      "uuid": "7711dc52-6843-40fa-9948-8a4beb5e5175",
      "node_uuid": "8787e026-32d8-41a2-8627-70250b0c7195",
      "arrived_on": "2018-04-17T14:05:30.213545+00:00",
      "exit_uuid": "256a62e2-9ccf-479a-a048-8e6daa409d54"
    },
    {
      "exit_uuid": "7a7ac2ce-9eb6-4e93-b168-8f933690e486",
      "node_uuid": "0a1e7e26-c628-4468-a827-bf4e5362b2e5",
      "arrived_on": "2018-04-17T14:05:30.272297+00:00",
      "uuid": "3f08a503-8f96-4196-9940-286294c7de8d"
    },
    {
      "exit_uuid": "139ecac9-3cb5-4f71-a726-989c9b2ba246",
      "node_uuid": "2aac3729-3aba-4cf3-bc70-8462246518c2",
      "arrived_on": "2018-04-17T14:05:30.299555+00:00",
      "uuid": "61e20180-bd8b-455d-9490-f682cf12723f"
    },
    {
      "node_uuid": "9e659cc0-d365-45b2-a646-ff45f66dfcd0",
      "arrived_on": "2018-04-17T14:05:30.352868+00:00",
      "uuid": "98f53c9c-0593-4088-8166-693b2e024062"
    }
  ],
  "channel": {
    "uuid": "a5313aff-7b14-4970-a791-664721b2c552",
    "name": "Test Channel"
  }
}

I think we're pretty close to being able to replicate that in goflow with the exception of path that is isn't exposed in goflow.

[rowanseymour]

Fixed in #263