Deploy / Update Workflows via Git, API, CLI

[dylburger]

I want to write and test workflows on my local machine, storing the workflow's code in my existing version control system.

I also want to programmatically deploy workflows using a CLI, for example using a command like pipedream deploy. This way, I can deploy workflows locally or as part of my CI/CD pipeline.

Related: #1139

[dylburger]

We've launched the first version of the Pipedream CLI: https://docs.pipedream.com/cli/reference/

The CLI supports managing Event Sources for now (not yet workflows), so I updated the title of this ticket to reflect the need for managing workflows via CLI, and am keeping it open to track that request.

[morganism]

@dylburger Thank you! I am finding Pipedream really useful, Workflow management via CLI would be great. I'm pretty new to this service, but picking up the concept and how to get real utility out of it is not a chore. I'll DM you if I can.

[dylburger]

Adding a comment from #857 here. When we support an API / CLI for deploying workflows, we plan to also let you serialize workflows to plaintext (e.g. JavaScript code with some config locally), which means you'd be able to store them in Git. That feature would enable the request below.

"For testing my workflow with a real case, I need to deploy my code, so everytime I test something I create a new version of the workflow. My workflow ends up with lots of versions. Most of them have bugs or are here for testing/debugging Sometimes I want to go back to a specific version but I can't remember which one it was, it can be 10 versions back or more, it's hard to identify versions and difference between them

Describe the solution you'd like Something closer to real code versioning, so more like git commits:

being able to add a comment to a version. This would be a "quick win", as I could just tag a revision that I know is working before creating more revisions, So I can easily come back to it even if I created many revisions after. Nice to have: being able to compare 2 versions of a workflow"

[illusoryio]

This would be very useful.

[moltar]

Would love to get this supported, this way we can migrate our processes from unstable Zapier zaps and Integromat scenarios to proper code solution with some tests.

[sjn001tvh]

Geetings. I'm a newby to Pipedream, but after several weeks of working with workflows between Dropbox and ClickUp etc., I've been able to accomplish a lot processing information from phone to the ClickUp app. However, as I have developed these routines, there have been instances where I have deleted a step... I know, I know, even though it warns me about the delete, I just needed it to get out of the way so I could try another option, and of course when I did... I lost the routine. I only had to do that once, and fortunately it wasn't filled with a lot of functions, etc. I've looked and asked about a utility that will allow me to save a step, maybe as an archive, so it suggested I follow this issue.

At this point, my only option is to cut and paste the code between the async(events, steps, auths) {...} and place them in a .js files defined as functions. Does anyone have a better method? Then, I can test the code, and if it works, cut and paste the new version back into my .js functions file. Thanks.

[cakemountain]

👍 for this, right now it's the only thing preventing my team from using your service!

[dylburger]

@cakemountain I hear you and we are working on an early version of this now! We’ll be sharing details on that here. We’d love y’all and anyone else watching this issue to test the beta when it’s ready.

[cakemountain]

@dylburger love hearing it. Do you have an approximate release timeline for the beta?

[dylburger]

I don’t, but I’ll let you know as soon as we have clarity on that.

[cakemountain]

@dylburger okay gotcha. if you're taking applicants for early access or beta testing we're highly motivated. Reach out to me on LinkedIn if you want to connect.

[ak--47]

this is my local workflow for writing tests inside of VSCode and constantly redeploying changes to pipeDream:

one file, 4 shells:

essentially:

• pd dev (watch for changes)
• pd logs (so i don’t need the web browser)
• jest tests on my code
• a scratchpad for doing random stuff

community post: https://pipedream.com/community/t/developing-code-steps-in-a-workflow-locally/1489

[doronlh]

+1

[mmmoli]

Would LOVE ❤️ this feature.

We manage code in a monorepo and storing workflows would be 👌 👌 👌

[jondkinney]

This would be great! Looking forward to hearing some updates, hopefully.

[charlie-s]

@sjn001tvh and anyone else that needs it, you can hit https://api.pipedream.com/v1/users/me/workflows (use the Authorization: Bearer ... header from the settings page) to download a JSON array of your workflows, including custom code in any components.

[cfjedimaster]

Interesting - so this would be a code view of a complete workflow then? And in theory, we will next have a way to push these back? (Well one at a time. ;) And I assume we will have an endpoint to get one workflow eventually?

[charlie-s]

It's not listed as an endpoint in their REST API docs. I also tried getting a workflow by ID without success. You may find that some POST/PUT requests work, but I wouldn't count on it in terms of creating/updating nested objects and such.

I just don't like the idea of building a complicated workflow that my users will depend on without having version control / backups. So this is better than nothing.

[cfjedimaster]

Oh - duh - I misread you as someone from PD. :)

[andre-brunetta]

@sjn001tvh and anyone else that needs it, you can hit https://api.pipedream.com/v1/users/me/workflows (use the Authorization: Bearer ... header from the settings page) to download a JSON array of your workflows, including custom code in any components.

How would you paginate?

[mmmoli]

Any updates on when this feature is likely to land? 👀👀

[EricWu91]

Please make this happen! 🙌

[richayles]

+1

[dsolovay]

Glad to see this in progress! I'll share a use case that I didn't see discussed. I'm doing a demo of a new webhook feature in a different platform (Sitecore 10.3), and am using Pipedream to show how this feature can be used to build out an integration. I'm planning to share my code in "gist" form, but this will require a lot of copy/pasting by users. Having a standard export format (a la Postman collections) would make this much easier to share, and would also drive adoption.

[brandonminch]

+1 for this! We would love to be able to create workflow templates that can can be duplicated, configured, and deployed on behalf of various clients. For example, a client could visit our website and select a workflow template, input a few configuration options in a user-friendly UI, and deploy the workflow all via the pipedream API behind the scenes.

[renattomachado]

+1

[jibe-b]

+1 too :)

[jibe-b]

+1 too :)

[blangley88]

+1 @brandonminch nailed it here, this would be ideal for our business use-cases as well!

[kaneg]

+1

[ldbartlett]

+1 @brandonminch This would help streamline the solutions we develop to be deployed in a user-friendly manner!

[phefti]

+1 here as well, great future option that would add a lot of value.

[EricWu91]

👍 Would love to be able to edit my code at the comfort of my IDE's, instead of the online Monaco. I mean, it is the reason why we can deploy workflows today at all, but it's very hard to scale. I have "long" (+300 lines) workflows, and it becomes upsettingly slow.

[weiserman]

+1 for using in workflows. Would be great to leverage co-pilot for code completion and suggestions.

[The-TT-Hacker]

I think pipedream would see much more adoption if this workflows API existed. It would allow workflows to be managed using version control and IaC (e.g. a terraform module). They should really be deployed by CI/CD pipelines.

[EricWu91]

Long shooting here, but wouldn't it be possible to retrieve the workflow codes from git repos?

On Sun, Apr 23, 2023, 12:38 AM Tim Thacker @.***> wrote:

I think pipedream would see much more adoption if this workflows API existed. It would allow workflows to be managed using version control and IaC (e.g. a terraform module).

— Reply to this email directly, view it on GitHub https://github.com/PipedreamHQ/pipedream/issues/220#issuecomment-1518933634, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHJMBEB7YFFCSNLWDT6JB3XCSP35ANCNFSM4OC7QD2Q . You are receiving this because you commented.Message ID: @.***>

[mroy-seedbox]

This is 100% required in order to scale out Pipedream to hundreds of workflows, especially when many of them are very similar, or even identical (but running on different schedules or triggers, or with different parameters). Ex: run job A (via external API call), and wait for it to complete, then retrieve the results, and send them to Slack -- and do the same for job B, C, D, E, and so on...

Reusable actions/components are nice, but they only apply to one step. And then it is tedious to manually apply the new version in a multitude of workflows.

Manually managing hundreds of similar workflows is a no-go. It's kinda like copy pasting the same code all over the place: no one wants to maintain that mess (which version of the code/workflow is the most up-to-date, etc.). At least if we could look at all our workflows "at once" in an IDE, it would help to manage the repetitions, and make simultaneous updates/apply refactorings. I dream of editing Pipedream workflows directly in VS Code, and then using an awesome VS Code extension with all kind of cool features to interface directly with Pipedream from VS Code.

Also, with workflow management via API/Git/CLI, we could add a layer of abstraction on top of Pipedream, allowing us to define a generic/reusable/configurable workflow once, and then deploy it multiple times (with different parameters, schedules, or triggers).

Of course, reusable workflows or multi-step actions/components would also be very helpful, and could accomplish similar goals. Zapier has something similar, called "Sub-Zaps".

[peteratticusberg]

+1

[EricWu91]

Would love to hear updates from the team!

[n0vakovic]

+1

[danhstevens]

+1

[EricWu91]

First and foremost: Is this the feature we've all been waiting for?

If it is, it looks amazing.

But this is slightly more disheartening.

I understand your business model, and I know you guys also need to make a buck. But 99/month for a single dev, in slightly larger projects is prohibitive. That's five times as expensive as the previous pricing tag!

[mroy-seedbox]

Maybe they could make the feature usage-based for lower plans. Just charge an extra credit per invoke for workflows that are synced to GitHub.

[EricWu91]

For context, I'm a single developer working on a few personal projects. I've replaced some of my backend code with Pipedream workflows, because it is so much easier to have an app calling a fixed URL than having to worry about all that infrastructure.

That being said, my workflows are amounting to almost 800 lines each, and editing, testing, developing online becomes a big nuisance very fast. Monaco is fine for a few hundred lines tops, and shouldn't be used for slightly larger codebases!

So... how am I supposed to keep working on my PD workflows if the alternative to wasting hours working with a feature-less IDE is paying 100 dollars a month (1200 upfront)?

[Stvad]

Seems like it's indeed the feature we've been waiting for. I'm very surprised to see it gated behind paywall it's such a foundational thing 🤔

[EricWu91]

@dylburger can you please provide updates on this? Even if it means to answer dryly "Yes, we heard your requests, we developed them, and we'll paywall them. Live with it.". At least we will have an answer.

[dylburger]

First, I really appreciate all the feedback! We have a small team for the size of the business we're operating, so it can take us longer than we'd like to groom GitHub issues.

We recently reduced the base price of all of our plans for new customers (see our pricing page), and are considering further expansion of GitHub support to lower tiers (e.g. for personal vs. organizational use). Like some of y'all have mentioned, since we operate a business, we think carefully about the value we're providing at each tier, gating some features at different tiers accordingly. But we hear the feedback, and I'm sharing all of this with the broader team so they see it.

Since this issue also operated as a catch-all for Git serialization and a workflow API + CLI, I'm closing this master ticket and opening separate tickets for the remaining work:

• https://github.com/PipedreamHQ/pipedream/issues/8736
• https://github.com/PipedreamHQ/pipedream/issues/8735

[EricWu91]

@dylburger , Thanks for the answer. I noticed that you created two issues - one for CLI, one for API - but not for GIT. Should we assume that the latter won't be receiving further considerations given that Github Sync is already out?

Furthermore, can we (and if so, where should we) follow up on this?

are considering further expansion of GitHub support to lower tiers (e.g. for personal vs. organizational use)

[mroy-seedbox]

The CLI & API features will effectively allow users to roll up their own GitHub integration anyway (or use any other version control provider), by creating a very simple CI/CD pipeline/workflow.

Which kinda defeats the purpose of paywalling this feature. 🤷‍♂️

[dylburger]

@EricWu91 We aren't planning on adding a GitHub issue to track that. We generally use GitHub only for new features, bugs, and integration requests, but not requests to transition features to different plans. I did share all this feedback with the team, though.