[FEATURE] I’d like to self-host Pipedream

[dylburger]

I’d like to self-host the Pipedream service for security and compliance reasons.

Please let us know your specific desire for self-hosting in the comments below. We are considering the ability to self-host certain components of Pipedream (e.g. the ability to run event sources on your own infra), and the ability to run a Pipedream agent in your infra that allows Pipedream limited access to internal resources), so hearing your use case will help us prioritize the right features.

[dylburger]

See https://github.com/PipedreamHQ/pipedream/discussions/947

[nileshtrivedi]

+1 for the ability to self-host for multiple reasons:

• Various regulations for data being available in a specific availability zones
• Being able to process sensitive data without sharing it with third-parties (eg: Pipedream and their cloud platform)
• Process proprietary data that must not leak EVER outside our own AWS account

[nileshtrivedi]

@dylburger What is the current status of making Pipedream self-hostable? Is it possible for someone to submit a PR that fixes this?

(1) Are all necessary subsystems of Pipedream public and open-sourced on GitHub? (2) Will you accept a PR that fixes this issue? Do the dev need to sign a CLA? (3) Does it use any third-party services (eg: Firebase, Auth0 or AWS or GCP)? If yes, a list might be helpful. (4) Is the domain pipedream.com hard-coded in many places?

[dylburger]

Hi @nileshtrivedi , thanks for reaching out!

The pipedream.com hosted service is primarily closed-source at the moment. As we open up parts of our stack (e.g. we're planning to publish a runtime that would let you execute components locally / in your own infra), we'll update this issue.

[seventhsite]

It is strange for me that the project is not have an install possibility from the beginning. +1 to selfhosted

[DacoDev]

In "cross-shopping" this versus a service/application like n8n, I like that I can download and run my own n8n server in case they go out of business or get acquired. I'd be able to continue using it as usual or at least maintain functionality until a suitable alternative can be found, like what happened with Automate.io. In comparison, I lean toward preferring Pipedream's style either way, but I'm just adding another take on why it would be good to have it self-hostable.

[alexander-potemkin]

In "cross-shopping" this versus a service/application like n8n, I like that I can download and run my own n8n server in case they go out of business or get acquired. I'd be able to continue using it as usual or at least maintain functionality until a suitable alternative can be found, like what happened with Automate.io. In comparison, I lean toward preferring Pipedream's style either way, but I'm just adding another take on why it would be good to have it self-hostable.

Agree and choose n8n exactly for that purpose.

What happened with Automate.io? It seems to be working?

[DacoDev]

Alas, Notion acquired them and they're shutting down services in October: https://automate.io/blog/closing-down-automate-io/

You can't even sign up for an account anymore - it looks like it was a great service and definitely had a fair pricing scheme; kind of like Pipedream where it's pretty straightforward.

In general, I like Notion and plan to use Pipedream and/or n8n to integrate other services with it but It will be interesting to see if they introduce a workflow system to Notion.

[alexander-potemkin]

Thank you, it's a pity.

By the way, @DacoDev , let me invite you to think, what kind of node(s) could be added to n8n, to make it more like a Pipedream, if you don't mind?

Probably it wouldn't be that much complicated or would be something that could be started by modifying an existing function node, that I could pick up and implement one way or another?

[DacoDev]

This isn't exactly the right venue for that (a competitor's feature request page), but I'll email you at the address on your profile if you're open to it. I'm new to using both but came across this FR when trying to use both simultaneously and thought I could run both locally :)

[dylburger]

Self-hosting is still on our backlog! We're planning to develop a full, self-hosted version of the Pipedream platform, and also a way to run components locally (e.g. as a container in your own private network) that could trigger a Pipedream workflow hosted on the current platform.

Is the core concern the risk that Pipedream goes out of business? Or do you have other regulatory requirements that make it easier to self-host?

We'll add any relevant updates to this ticket, so keep an eye out here.

[DacoDev]

Is the core concern the risk that Pipedream goes out of business?

To answer this part, it's not my core concern; it's another reason with a recent example (Automate.io).

Or do you have other regulatory requirements that make it easier to self-host?

Self-hosting is still on our backlog! We're planning to develop a full, self-hosted version of the Pipedream platform, and also a way to run components locally (e.g. as a container in your own private network) that could trigger a Pipedream workflow hosted on the current platform.

This is a neat approach, like running a GitLab or GitHub pipeline with self-hosted runners but bi-directional traffic like an API gateway.

For me, the reason for using self-hosted is that it can live inside my network and access non-publically routed resources. Running a copy (or "runner") locally would mean being able to interact with my home devices, services, and machines.

For example, I have a WiFi-connected irrigation system; I can control it by making API calls directly to the device through the local network or a connection from the manufacturer's app on my phone. Could I set up Pipedream to talk to the manufacturer's internet-based system? Maybe.

Or I could run a service inside my network that can talk to my local device directly with a specific input trigger or schedule.

With other systems, I think it's pretty awesome that I can run the service from my laptop in a situation where I don't need my workflows to be on an always-available internet-based service. Say, a workflow where I sometimes need to process one image to add a watermark or scale it before I send it somewhere. Having that running locally would prevent the issues of:

1. internet connection in general, whether intermittent or unavailable at the time
2. internet that is fast enough that if transmitting a large image, video, or dataset, I am not constrained by poor speeds
3. service availability of the platform and other cloud services
4. privacy of that photo for whatever reason; if you don't intend to leave that photo on the internet, why send it there at all
5. service outages and having a way to fail over workflows assuming critical business cases

That is where I see the need for others to run the platform locally, not from a free-as-in-libre/beer approach, but not every company or person will allow an internet-based service to handle their data or access their systems.

I think this has just been a long way to say that having a self-hosted "serverless"-function-enabling-less-code platform is something that people are looking to use for one reason or another; otherwise, the people wouldn't have landed on this specific feature request!

Solving the business case may mean licensing certain sections or features under specific licenses that could prevent certain types of commercial use, like the Elastic v2 license, where a company can still run the product as part of their stack but not directly resell it as a hosted service.

APIs/connectors are likely where the bulk of your labor and data costs are, a freemium approach of "here is the base, there are add-ons available, but we do have to make money somehow."

If someone wants to code their own API/connector in the place of the paid one, they would be able to do it, open-source it, or use one that someone else has made with no warranty or guarantees, per the license! Or pay a fee as part of or on top of the service-level subscription if cloud-hosted, or enable it on the self-hosted version. With a business, most are willing to pay for services that are not their core competencies, make their lives easier, and have support and SLAs in place as risk control.

Suggesting this is obviously not in my own best interest wallet-wise, or from an OSS perspective, but if I ran a company with a product that exists to make a task easier in a way that COULD be done another harder way, that is a luxury and it would cost money. From the more benevolent open-source software view, having everything available as other services do, if someone is willing to put in the effort, resources, and develop the skills to host it themselves, then they should be able to make a copy of that software, learn and use it, and by the spirit of OSS, contribute back to it.

[dylburger]

Great feedback, thanks!

[alexander-potemkin]

This isn't exactly the right venue for that (a competitor's feature request page), but I'll email you at the address on your profile if you're open to it. I'm new to using both but came across this FR when trying to use both simultaneously and thought I could run both locally :)

it's turned out it could quite productive, thought :) I'm happy to continue a discussion though - here, via e-mail at profile or whenever else convenient.

[DacoDev]

Great feedback, thanks!

My reply got a little out-of-hand, but I think I covered most of the use-cases I could see that would use it self-hosted. I will also add though, I do use Pipedream in it's current web-based form either way because it's a great service, and I do have workflows that would not need to be run locally, connection of multiple SaaS services, and I love the flexibility of being able to use it like a step-machine!

[andreabisello]

i'm working as QA engineer on a 800+ employee italian company, and pipedreams is very useful for our automation. We cannot use for every kind of integration for security reasons handling confidential data. Until a self-hosted version of pipedream can be used, in the meanwhile i'm proposing this https://pipedream.com/docs/privacy-and-security/ to our security manager hoping it will accept the adoption of this product in our context.

[alexander-potemkin]

@andreabisello , could n8n with its functions be of any help to your case here? If not, do you see some showstoppers here?

I keep looking for a way to do easy self-hosted code and integration thing, would be happy to be of any help, if you like

[perru]

https://pipedream.com/blog/introducing-the-pipedream-source-available-license/

We remain committed to all components in our integration registry being source-available.

So it's a lie.

I can't use a project without a clear license and I really discouraged everybody to use a project without a clear license if you want to avoid any legal problem.

Actually, it smells like "open-source-washing". But it's not open-source. Please clarify the license.

[dylburger]

@perru Can you clarify the specific concern? What specifically about the license is not clear?

[perru]

@dylburger

On website :

Sources and actions are open source on GitHub. You can fork and modify them, contribute new ones, or publish private versions to your personal account or organization.

Not all sources are opensource since the code to self-host Pipedream isn't available. And :

The Pipedream Source Available License Version 1.0 isn't open-source compatible : Licensee is not granted the right to, and Licensee shall not, exercise the License for an Excluded Purpose. For purposes of this Agreement, “Excluded Purpose” means any commercial use of the software including, but not limited to, making available any software-as-a-service, platform-as-a-service, infrastructure-as-a-service or other online service that competes with the Software or any other Pipedream products or services.

It's the first rule of OSI definition :

• Free redistribution, the license does not restrict any party from selling or giving away the software

You can't restrict commercial distribution. Put some code on GitHub isn't enough to being open-source.

Pipedream can use a specific restrictive license, it's ok. But in that case, it cannot claim to be open-source.

[dylburger]

Thanks. Please note that before this change, Pipedream shipped components using the MIT license. We’re aware of the details you note, and modified “open source” references to read “source available” when we modified the license. e.g. see the second bullet point here: https://pipedream.com/docs/ . It looks like we just missed that reference, and we’ll update.

[acremonezi]

+1 Please! Great product! Great Idea!

[V-harshith]

+1 would love it to be opensourced

[backmind]

+1, absolutelly!

[ndtreviv]

I'm working on a project where I want users to be able to build and execute workflows with a limited set of workflow actions.

ie: I write workflow actions appropriate only for the project, and run and manage those workflows via a "Pipedream runtime service". This would involve having an API to execute workflows - send data in, receive a response. The response might include whether the workflow is complete, or waiting for action pre-execution conditions/schedules to be met, or errored somewhere along the way. It would include input data that has been mutated, appended to by the various workflow actions.

And obviously an API to create and update workflows as well.

Ideally it would also include a UI library that would enable users to build their workflow.

At the moment I've hand rolled a very simple workflow system that works fine, but now I need to think about user-configuration and I'd rather swap it all out for something way more awesome like Pipedream.

[mmcallister-selkirk]

+1 for self-hosting option, even if only available on a business tier. We would need this to access local SQL and Oracle databases since many of our legacy products are still hosted internally.

[dylburger]

@mmcallster are those DBs hosted in any sort of cloud VPC / private network? We do support peering / VPN connections from Pipedream VPCs. Feel free to reach out to dylan [at] pipedream [dot] com if you want to chat more about options.

[sebmellen]

We love Pipedream, but we want to handle sensitive data through it which (by law) cannot leave our systems. Self-hosting would be fantastic.

[anil-dewani]

+1, go for it

[SheryarKhaliq125]

+1, go for it