Proposal: Add Studio to Serverless Toolkit

Note: This issue is a proposal and is subject to change and open for feedback.

At the end of April we launched the v2 REST API for Twilio Studio. There is a decent overlap of users between Functions/Assets and Studio and so adding support here could be beneficial for those users. This proposal highlights a potential way of bringing support for Studio flows to the Serverless Toolkit

User Stories

1. Debug Functions called by Studio

As a developer using Twilio Studio and Functions, I want to be able to debug the logic of the Twilio Functions that are being called by my Studio flow.

2. Keep assets and Studio flows in one place

As a developer using Twilio Studio, I want to be able to have better control over the assets that are being called by my Studio flow.

3. Version control Studio flows across multiple accounts

As a developer using Twilio Studio, I want to control and synchronize my Studio flows across multiple accounts and environments.

Known Limitations

We won't be able to run Studio flows locally. Doing so would be out of scope of this support because it would require us to implement an entire state machine that mimics Studio.
The best experience to edit a flow will remain to edit it in the Twilio Console through the graphical interface. But in theory people could edit the JSON by hand or build their own tools on top of the JSON.
The Serverless API and the Studio API follow some different concepts such as Environments & Builds (Serverless) vs Revisions & Drafts (Studio). We will use the Serverless conventions and apply them to the best of our ability to Studio not the other way around.

Implementation Proposal

Folder Structure

We'll continue with the goal of convention over configuration. Since a Studio flow is just a JSON file, they can easily be stored on filesystem.

  my-project
  ├── .env
  ├── .gitignore
  ├── .nvmrc
  ├── assets
+ ├── flows
+ │   └── my-flow.json
  ├── functions
  │   └── example.js
  ├── package-lock.json
  └── package.json

Studio does not support the concept of paths meaning we will not support any sub-directories in the flows directory for the time being. Any sub-directories might be used to add additional functionality in the future.

File Definition

A Flow JSON file could have two naming conventions. The first one would be containing the SID. This does not really work because those SIDs differ for every user and wouldn't allow User Story (3).

Instead we'll use the filename to inform the friendlyName of a Flow Resource.

Conventions

A flow deployed by this tool will have a friendly name computed the following way:

const flowName = humanizeString(flowFileName);
const friendlyName = `[${environment}] ${flowName} [Deployed by Twilio Serverless Toolkit]`

Examples deploying a project with one file called my-flow.json inside the flows directory:

twilio serverless:deploy --> [dev] My flow [Deployed by Twilio Serverless Toolkit]
twilio serverless:deploy --production --> My flow [Deployed by Twilio Serverless Toolkit]
twilio serverless:deploy --environment=stage --> [stage] My flow [Deployed by Twilio Serverless Toolkit]

Deploying Flows

Out of the box twilio serverless:deploy should "just work" if you create a new valid Flow JSON.

Under the hood the following should happen:

Deploy Functions & Assets first
Find all Studio flows in flows and load them into memory
Update the variables inside flow where necessary to point against new Functions & Assets
Verify flows to be valid against the API
Compute Friendly Name
Try to find existing Studio flow with said friendly name.
If existing flow can't be found, create a new flow
If existing flow was found, update existing flow
In both messages add a commit message to the flow update that contains Deployed via Twilio Serverless Toolkit. If the local project is a git repository include git hash. For example Deployed via Twilio Serverless Toolkit [a12g45]
Once all flows have been updated, set all of them as published (consider option to disable auto publishing)

The command should also respect the --no-functions and --no-assets flags.

Local Development

As mentioned before, we cannot locally run Studio flows. However, a user might still want to debug the Functions that are being called by their Studio flow. To enable this we should introduce a new flag to the start command.

Running twilio serverless:start --ngrok --publish-dev-flows will spin up an ngrok server and afterwards publish a version of all Studio flows that doesn't point at deployed Functions (like a normal deploy would) but instead points at the deployed ngrok endpoint. That way the user can point their Twilio app at the new Studio flow. Whenever a Function would be executed instead the local Serverless Toolkit development server will be hit and people will be able to debug their Functions locally.

Once the twilio serverless:start command is exited, the CLI will ask the user if the Studio flows should be deleted. Some users might want to keep them around as to not have to update their phone numbers again and again to point to a new Studio flow but others might want to keep their Studio flows list clean.

The flag --publish-dev-flows can only be used in combination with --ngrok.

The flows should follow a similar naming convention as outlined above. But the pattern should be:

const flowName = humanizeString(flowFileName);
const friendlyName = `[local dev] ${flowName} [Deployed by Twilio Serverless Toolkit]`

Since local dev wouldn't be a valid argument for --environment it will avoid naming conflicts.

Open Question: Should we prompt people if they want to deploy dev flows when we detect a flows directory?

Downloading an existing Flow

You can already download a flow using the Twilio CLI but depending on whether we need flows to adhere to some specific patterns, we might want to make it easier for users to download and configure existing flows.

Example: twilio serverless:download-flow <flow sid>

Fetches specified flow
Turns friendly name into kebab-case filename. It should ignore anything that the Serverless Toolkit might have added to a flow's friendlyName through a previous deployment. For example [dev] or [Deployed via Twilio Serverless Toolkit]
Checks flow for Functions widget and asks which ones should stay untouched vs which ones should refer to Functions inside the project
Stores new flow inside the flows directory

A user might also specify multiple flow SIDs in a comma separated list.

Function Templates

The next natural step would be to introduce support for flows into github.com/twilio-labs/function-templates and by extension into twilio serverless:init and twilio serverless:new. The behavior would be similar to what we do with assets today except that there is no namespacing concept.

What it shouldn't do

Aside of the command outlined above there shouldn't be any additional Studio specific commands. Any other Studio specific commands should be part of a Studio CLI plugin or people should use the API via the Twilio CLI directly. The primary goal here is to make it easier for Functions and Assets users to use Studio.

Timeframe

This is more involved work and will touch multiple packages. As a result we should do this work only after we moved to a monorepo. This should land in the next major version though.

This looks good to me. Some comments:

Flows would be save as a JSON file rather than a JS file (see under File Structure).

Will we ever have something that downloads functions and assets from a Serverless service? And should the download command be ready to work with either? Or is it better to keep download-flow and download-runtime-service separate?

For downloading can we allow the command to take more than one flow Sid?

Would the humanizeString extract the original friendly name from a flow that has a friendly name [dev] original name [Deployed by the Serverless Toolkit] in case someone pulls a flow down that had been previously deployed from the toolkit?

A Studio Flow may make HTTP requests from the HTTP Request widget. Conceivably, this could be to a function, do we try to match on the URLs for those widgets? If HTTP request widgets are remote URLs do we show a warning of some sort when in local dev?

Thanks for the questions @philnash! Here are the answers inline:

Flows would be save as a JSON file rather than a JS file (see under File Structure).

Good catch! Fixed!

Will we ever have something that downloads functions and assets from a Serverless service? And should the download command be ready to work with either? Or is it better to keep download-flow and download-runtime-service separate?

I think longterm we should have a download Functions & Assets functionality but that one will have quite the amount of complexity by itself since you'd have to specify more than just a Service SID. As a result I'd keep these two separate for the time being. And I wanted to be explicit with what it does right now as to not confuse existing users.

For downloading can we allow the command to take more than one flow Sid?

Yes absolutely. I'll add support for a comma separated list to the proposal.

Would the humanizeString extract the original friendly name from a flow that has a friendly name [dev] original name [Deployed by the Serverless Toolkit] in case someone pulls a flow down that had been previously deployed from the toolkit?

Yes that was the intention but I'll call it out intentionally in the proposal.

A Studio Flow may make HTTP requests from the HTTP Request widget. Conceivably, this could be to a function, do we try to match on the URLs for those widgets? If HTTP request widgets are remote URLs do we show a warning of some sort when in local dev?

During download we should identify any call to a Function that includes the HTTP Request widget and ask the user what to do with them. During local dev and deployment we should only touch those ones and ignore any other.

A couple of comments on my side:

Running twilio serverless:start --ngrok --publish-dev-flows will spin up an ngrok server and afterwards publish a version of all Studio flows that doesn't point at deployed Functions (like a normal deploy would) but instead points at the deployed ngrok endpoint.

Is the plan to allow the dev studio flows deployed with the serverless toolkit to be used for test with real incoming messages / calls? In my mind the development flow could be the following:

Creates a studio flow on the Twilio console, set the webhooks for a phone number and publish the flow
Fetch the studio flow and add it to the serverless toolkit
Update the studio flow locally and test it using the serverless toolkit

In this scenario, if the developer wants to test the new solution with a real trigger (e.g. incoming message) they have two options:

Provision a new number and set the webhook to the (temporary) studio flow created by the serverless toolkit (if I understood correctly this proposal)
Use the test users on a new draft of the original studio flow

I think the second option is much cleaner and easier for the developers, but it could be more difficult to implement within the serverless toolkit.

This does not really work because those SIDs differ for every user and wouldn't allow User Story (3).

Sorry, I'm not sure I understand the meaning of this statement. Do you mean that Studio flow have different SID depending on the users? Apologies for the silly question, but I think resorting to the friendlyName may not be the best solution for the studio flow file (for one, the friendlyName accepts a wide range of characters that are not always allowed for file names, e.g. \ and [])

Thanks for your feedback @vernig

Provision a new number and set the webhook to the (temporary) studio flow created by the serverless toolkit (if I understood correctly this proposal)

Use the test users on a new draft of the original studio flow

Yes what you described in the first scenario would be what would happen. The proposal intentionally ignores the fact that the concept of "drafts" exists. The reason why we do that is because we want to have the behavior as close to what Serverless Toolkit users are used to already. Especially if they would want to for example test the "request trigger" instead of an incoming message/call trigger.

Sorry, I'm not sure I understand the meaning of this statement. Do you mean that Studio flow have different SID depending on the users? Apologies for the silly question, but I think resorting to the friendlyName may not be the best solution for the studio flow file (for one, the friendlyName accepts a wide range of characters that are not always allowed for file names, e.g. '' and '[]')

Neither of the two options are optimal unfortunately. However, we need to give the filename a value that we can "find" on a Flow Resource so that we can detect if a specific flow has already been deployed or not. If we would give each filename a SID value you wouldn't be able to deploy the same project to any other Twilio Account which is not the intend of this project. Therefore we use the FriendlyName. The reason why this works is because we intend people to download flows using the proposed twilio serverless:download-flow command that would sanitize the existing FriendlyName to turn it into a valid filename (by running kebabCase on the name which will take care of these cases.) and the only way you would upload it is through the Serverless Toolkit.

We do similar behaviors for the Serverless Toolkit with Functions & Assets already. They receive FriendlyName values that are being computed based on the local filename that also informs the Path property in that case (which Studio doesn't have. Otherwise we would resort to that).

I hope that makes sense.

Hey Guys - Just checking in to see if there are any updates on this proposal and a timeline for when this will be implemented? The features described above would definitely be useful.

@tk63135 I'm afraid this is not currently on our priority list for work on the serverless toolkit, so currently I wouldn't hold your breath. We likely do need to take a look at the backlog, as well as other priorities, and determine if and when we can work on a big feature like this.

It is certainly useful to know that you are interested in this as a feature. If anyone else would like to see this, please do comment or :+1: so that we understand the demand. Thanks!

twilio-labs / serverless-toolkit