Feature Request: Define persistent docker network

[bjarnemagnussen]

Is your feature request related to a problem? Please describe. I am trying out LND's remote db backend support (using postgres). However, since I cannot just via the GUI add a postgres service (or any other external service that may be supported in the future) and supporting each such service feels a bit overkill, I was thinking about an easy way to handle this better.

Describe the solution you'd like It may therefore be nice to be able to define a persistent external docker network that must be used for a Polar environment, instead of Polar creating and deleting a specific default network with each start and stop. This would allow for additional services (e.g. postgres) to be spun up external/independent to Polar, while the nodes in the polar environment can make use of it.

Additional context Currently I have solved this by creating a standalone docker-compose file that spins up postgres and uses the docker network created by Polar automatically (e.g. 2_default for environment 2).

This requires that I always first start Polar, as otherwise the network will not be created.

When I want to stop the environment I have to first stop postgres, as Polar will try to delete the network it created when pressing the STOP button, which is impossible if any service is still running in this network.

---

Docker-compose file for my postgres service, which assumes polar network 2 exists:

version: '3.3'
services:
  postgres1:
    image: postgres:latest
    container_name: polar-n2-postgres1
    hostname: postgres1
    restart: always
    volumes:
      - ./networks/2/volumes/db/postgres1:/var/lib/postgresql/data
    expose:
      - '5432'
    ports:
      - '45432:5432'
    environment:
      POSTGRES_PASSWORD: lnd
      POSTGRES_USER: lnd

networks:
  default:
    external: true
    name: 2_default

[jamaljsr]

Thanks for the feedback and providing the details of your use case. What you're asking for makes total sense and I don't think it would be a huge amount of effort to implement.

My initial thought is that we add an additional field when creating a new Polar network. This field would allow you to specify an external Docker network to use. If a value is entered, then it will be paced into the docker-compose.yml file created by Polar, similar to what yo have above. Since the Docker network is external, Docker shouldn't attempt to delete it when stopping the Polar network.

Does this sound like a reasonable UX to you?

[bjarnemagnussen]

Thanks! That sounds fantastic and would be very helpful to me!

[amovfx]

I'm going to take a stab at this.

[amovfx]

@bjarnemagnussen @jamaljsr Here is an example docker-compose file of what I have polar producing. The external network name is formated polar-${Network.name} I think the formatting for the network name merits some discussion.

version: '3.3'
name: External
services:
  ....
networks:
  default:
    external: true
    name: polar-External

[amovfx]

A branch for testing this prototype is here.

@jamaljsr @bjarnemagnussen Hey guys so this a prototype for the workflow, a checkbox that creates an external network in the docker compose file, only available when the network is stopped. I was thinking maybe a modal for giving the network a name? And maybe an option for deleting the docker network. When we are happy with the workflow, Ill move on to tests and coverage.

Looking forward to feedback and suggestions.

[jamaljsr]

Thanks for the sneak peak @amovfx. I don't think this value needs to be changed multiple times. My suggestion would be to only allow setting this option when the network is initially created. In the New Network form, there could be an Advanced Options collapsed section which contains the checkbox to make the docker network external. What do you think?

Here is an example docker-compose file of what I have polar producing. The external network name is formated polar-${Network.name}

Using the network name in the YAML file is a bit risky. The user could put in characters which make the syntax invalid, such as : or \. It would be simpler to just use the network id for the docker network name instead, ex: polar-network-${network.id}.

[amovfx]

My suggestion would be to only allow setting this option when the network is initially created.

From my personal experience, I disagree. I've been in the situation in the past where I have built a network I liked with polar and then decided I needed an external network for further experiments attaching other services. I see and understand your logic though, if you think it's the best, Ill move that over to the network creation pane as per your suggestion.

Using the network name in the YAML file is a bit risky. The user could put in characters which make the syntax invalid, such as : or . It would be simpler to just use the network id for the docker network name instead, ex: polar-network-${network.id}.

I like it. I can see this being a bit tricky when working with a lot of networks though. Perhaps this is feature creep, but what do you think of an option in the network hamburger menu, the place where you export the network, an option to copy the external docker network name or display other information?

[jamaljsr]

I just re-read the initial request and my response. AFAICT the goal is to have Polar use a docker network that has already been pre-created by the user. So they would need to specify the name of the existing network somewhere in Polar. That's why I suggested placing this input on the New Network form. If this is the intent then Polar doesn't need to generate the network name, it's user-specified. This is why I didn't think there was a need to toggle it after the network was created.

My initial suggestion was solely focused on @bjarnemagnussen's use case. It sounds like you're solving for a different, but similar, scenario. It would be great if we can tackle both in one update. Maybe it would make sense to add the option to the hamburger dropdown menu labeled "External Docker Network" which would show a modal popup that allows the user to toggle the feature as well as specify the network name. The input could be pre-filled with polar-network-${network.id} but the user can change it if they choose to. What do you think?

[amovfx]

Maybe it would make sense to add the option to the hamburger dropdown menu labeled "External Docker Network" which would show a modal popup that allows the user to toggle the feature as well as specify the network name. The input could be pre-filled with polar-network-${network.id} but the user can change it if they choose to. What do you think?

I mis understood as well. Big fan. I'll do this.

[amovfx]

I've updated the workflow. Sorry for the delay. Diablo 4 has captured my attention as of late. Please let me know what you think of these changes.

I didn't add the placeholder of polar-network-${network.id} instead I let the user know that they can clear, attach or set the network. I'm happy to refill it if you think it's best though.

A blank entry clears the network back to default.

[jamaljsr]

@amovfx Thanks for the update. Can you open a PR so we can discuss the changes there?

[amovfx]

You bet. https://github.com/jamaljsr/polar/pull/734