Custom Instance Domains

hardillb commented 2 years ago

Description

As a Team Owner, I want to be able to supply a custom domain to be used with my projects to differentiate them from other projects hosted on the system

Customer requests

### 1st Iteration Development Work
- [ ] https://github.com/FlowFuse/driver-k8s/pull/151
- [ ] https://github.com/FlowFuse/flowfuse/pull/3830
- [x] nr-launcher authentication plugin will need updating (and possible forge app) to handle correct redirect back to the editor/dashboard if request comes from custom hostname.
- [ ] https://github.com/FlowFuse/helm/pull/396
- [x] Testing on dedicated scaling cluster

### Production Updates
- [ ] https://github.com/FlowFuse/website/issues/2152
- [ ] https://github.com/FlowFuse/CloudProject/issues/418
- [ ] https://github.com/FlowFuse/CloudProject/issues/422

### Future work
- [ ] https://github.com/FlowFuse/terraform-aws-flowfuse/issues/8
- [ ] https://github.com/FlowFuse/flowfuse/issues/3982

sammachin commented 2 years ago

Just to confirm this should be a FQDN per project rather than a domain per team

cgsmith commented 2 years ago

Would this mean the FQDN would be domain.com for flowforge then each project would be a a subdomain or project folder?

project.domain.com would point to localhost:7880 or w/e?

hardillb commented 2 years ago

@cgsmith This is specifically talking about running with the docker or kubernetes container drivers not the localfs.

With k8s and docker drivers, the projects run in containers so do not have different port numbers on the host and are all mapped to unique hostnames on a single fixed domain at the moment. This epic is about allow teams to have their own root domain. e.g. team one can have <project name>.one.com and team two would have <project name>.two.com

You can not host projects (Node-RED instances) on different paths on a single domain (e.g. example.com/<project name>) due to the way browser local storage works as Node-RED use it as part of the editor authentication. This is a limitation of Node-RED, not the FlowForge platform.

protocolus commented 2 years ago

I have customers that would love to use the FF service but, depend on custom domains to serve their clients. Being able to tie a wildcard SSL to an account would be even better. That way new projects could all be under that domain automatically.

hardillb commented 2 years ago

First pass I see the following points that would need working on

Support in the forge application to bind a domain to a team
Modification to the docker/k8s container driver to allow the domain to be passed as part of project creation
Publishing of the Ingress controller hostname to allow domain owners to setup up CNAME entries to point to the ingress controller
Some way for the domain owner to provide/update a TLS certificate, most likely this should be a wildcard certificate

The first 3 items are relatively simple to implement, the last point will need very careful planning and may vary based on host environment. I can see a possible k8s approach, but docker may be different.

May be able to make use of letsencrypt to issue individual certs for each project.

sammachin commented 2 years ago

The use case of setting a domain for a team is separate to the original issue which is to set a custom FQDN for a project.

The Initial story is just per project not team domains and wildcards

hardillb commented 2 years ago

All the same problems still apply re TLS, the user is probably going to need to supply a certificate for the given domain. I suspect it will be easier to implement at a team level rather than per-project.

hardillb commented 1 year ago

@Pezmc stick a slot in the calendar and we can talk about how this will work. I think most of the work will actually be in the UI/UX apart from working out how to handle the TLS certs.

hardillb commented 1 year ago

Notes:

Scoped to a single project at a time
Only applies to Docker and K8s
Gated on a flag in team type? What is the business logic for this?
With out TLS it's simple
With TLS going to use LetsEncrypt (and work out what to do on AWS with ALB where that won't work)

Pez:

[ ] UI/UX for entering hostname with gates based on team type - https://github.com/flowforge/flowforge/issues/1344
[ ] UI/UX about deployment of change
[ ] Docker driver implementation (https://github.com/flowforge/flowforge-driver-docker/issues/48)

Ben:

[ ] k8s driver implementation (https://github.com/flowforge/flowforge-driver-k8s/issues/66)
[ ] Document user DNS tasks

hardillb commented 1 year ago

Note: Host names need to be valid and unique across all projects

hardillb commented 1 year ago

Blocking issue for deploying this to FlowForge Cloud https://github.com/flowforge/CloudProject/issues/110

Pezmc commented 1 year ago

This feature has been put on hold due to the above blocking issue. Progress, before it was put on hold, is as follows:

Server-side support for storing and validating hostnames: https://github.com/flowforge/flowforge/pull/1361
Work in progress client-side support for setting hostname: https://github.com/flowforge/flowforge/tree/wip-feat-hostname-field

When it is picked up again, the next steps were to:
- Finish the client side hostname setting and configure it to only show for EE users
- Add support for custom hostnames to the two drivers (https://github.com/flowforge/flowforge-driver-k8s/issues/66 and https://github.com/flowforge/flowforge-driver-docker/issues/48)
- Pass project hostnames from the server to the driver
- Document the DNS set-up process

knolleary commented 1 year ago

Putting this back on to the Product board for future prioritisation once https://github.com/flowforge/CloudProject/issues/110 has been planned.

ZJvandeWeg commented 11 months ago

@MarianRaphael interest from a prospect: https://app-eu1.hubspot.com/contacts/26586079/record/0-1/7563151

robmarcer commented 9 months ago

Interest from an existing customer - https://app-eu1.hubspot.com/contacts/26586079/record/0-2/8209301981

hardillb commented 9 months ago

The hardest part of this will be TLS certificates for HTTPS.

There is a PR to add cert-manager support to the k8s container driver which would allow the use of LetsEncrypt to issue certificates. We may be able to leverage this, but will need to work out how it will work/play with the current ingress set up on production.

hardillb commented 9 months ago

I wonder if we could add a second ingress entry point to handle custom domains, this might make things easier

ZJvandeWeg commented 7 months ago

@MarianRaphael @hardillb Is this currently still blocked? Or is this only blocked on priority?

hardillb commented 7 months ago

There are still technical problems to solve, mainly around issuing certificates for the hostnames using the custom domain. We still need to test running a second ingress instance

hardillb commented 6 months ago

Useful details

https://kubernetes.github.io/ingress-nginx/user-guide/k8s-122-migration/#how-can-i-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster

hardillb commented 5 months ago

I have now got a second ingress-nginx controller installed in a test cluster using

controller:
  ingressClass: "custom-nginx"
  ingressClassByName: true
  ingressClassResource:
    enabled: true
    name: "custom-nginx"
    controllerValue: "flowfuse.com/ingress-nginx-2"
  config:
    proxy-body-size: "0"
    use-gzip: true
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "ssl"
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "120"
    externalTrafficPolicy: Local
  allowSnippetAnnotations: true

Important note, the helm release name must not match the existing nginx-ingress release (otherwise the webhook and rbac entries clash)

joepavitt commented 5 months ago

Once you've caught back up with this @hardillb - can you update here with a status of the outstanding items before this can be considered finished please?

hardillb commented 5 months ago

Current state:

Testing of installing second nginx ingress controller on AWS EKS cluster (using the cluster built for the scaling testing)
K8s driver Draft PR raised https://github.com/FlowFuse/driver-k8s/pull/151, the basic work is done and tested on my local setup. Will need full testing with UI
UI and hostname storage work started in FlowFuse Project
nr-launcher authentication plugin will need updating (and possible forge app) to handle correct redirect back to the editor/dashboard if request comes from custom hostname.

joepavitt commented 5 months ago

UI and hostname storage work started in FlowFuse Project

Any draft PR or linked issues for this?

knolleary commented 4 months ago

I've updated the description with a task list based on the latest status update.

joepavitt commented 4 months ago

Status update from Ben:

UI should be functional today (PR: https://github.com/FlowFuse/flowfuse/pull/3830)
Manual testing for robustness
Test cases to write for automated tests
Documentation still required

Wednesday next week is when Ben hope it'll be rounded off