Networking between components : ports

[MarieSaphira]

The ServiceRegistry throws a NullPointerException if there is no published port. The databases should only be reachable by the API/QL, not from outside the cluster/network.

In Docker Compose the services (i.e. containers) can reach each other by their names if they are in the same network, no ports are needed (e.g. curl <servicename> executed from inside a different container is working). In Kubernetes a service (different kind than above) takes care of the communication with a pod (or group of pods). It has to have a port in its definition, but can also be reached inside the cluster by just using the service's name. Example:	File	Extract
ML model	documentdb document-database {
DL model	database document-database : mongo {
docker-compose.yaml	services:    document-database:
kubernetes.yaml	kind: Service apiVersion: v1 metadata:     name: document-database

To conclude: we don't need port definitions in the database container specification.

Also interesting for QL @DavyLandman @pinoval @tvdstorm

[OrfenCLMS]

The API will longer produce an error when an external port is not provided.

In Docker Compose the services (i.e. containers) can reach each other by their names if they are in the same network, no ports are needed (e.g. curl executed from inside a different container is working).

The containers can network between them (machine to machine) using just the service names, however for service to service interaction within the containers the internal ports of those services are needed. Some drivers like the one mongo uses, can still connect without a port since it assumes the default one is in use. But, for example, i can't connect via the API to the typhonql-server REST service just by using http://typhonql-server/, i need to use http://typhonql-server:7000.

We could skip the ports entirely and make the assumption that depending on the type, the default port will always be used, but this will require me to provide default ports to the ql service, which i can do but i think limits the flexibility of users a bit.

[DavyLandman]

We could skip the ports entirely

@MarieSaphira correct me if I'm wrong, but there still exist the concepts of ports in k8s right? As I understand it, this issue is about exposing them to outside of the docker/k8s swarm.

Maybe there is some information missing from the metadata, we currently don't assume the local port (3306 for mariadb) but get it from the polystore API. I think @OrfenCLMS might be suggesting at this assumption? So that if we request connection information, we only get the host, and let the jdbc/mongodb library figure out the default port to connect to?

[MarieSaphira]

there still exist the concepts of ports in k8s right?

yes, and the service exposing pods needs to have one. But we could make it the target port (the one EXPOSEd in the dockerfile) as default and add it automatically to the DL model.

The API will longer produce an error when an external port is not provided.

That could be enough change to the API. Maybe DL could provide the default port but let the user choose to change it, e.g. when they provide a custom DB image and maybe expose a different port.

So the connection information includes host and port. Maybe DL will show a warning if a published port is added to a DB container.

[OrfenCLMS]

That sounds good to me @MarieSaphira. Exposing ports can be done on the docker compose itself, if anyone wishes to do so and is probably of no concern to the Polystore.

[DavyLandman]

Maybe we are saying the same thing, but shouldn't we just drop the whole ports in the docker-compose file? so that the databases aren't accessible from the outside? So the question then remains, how does QL (and the API for the status message) know the port to connect to, but assuming the default might be good enough here).

yes, and the service exposing pods needs to have one. But we could make it the target port (the one EXPOSEd in the dockerfile) as default and add it automatically to the DL model.

Wouldn't that mean you have to start parsing external docker files? Maybe we can avoid that by not adding any ports by default? Or keep letting users manually code it in the template?

[OrfenCLMS]

@DavyLandman i think we only need the private ports on the DL xmi. Public ports can be added by users if they wish to do so directly on the docker compose.

[DavyLandman]

ok, if that isn't too much work on the DL side? (since it supports all kinds of containers)

[MarieSaphira]

Wouldn't that mean you have to start parsing external docker files?

Since the full prototype will include Mongo and MariaDB/MySQL, setting default ports should not be a problem, they won't change their exposed ports. Also setting default ports for the polystore components should be fine, since we control the images. The user can overwrite the default ports by adding a "ports" field to the DL model.

[MarieSaphira]

Public ports can be added by users if they wish to do so directly on the docker compose.

Also in the DL file. The goal is not to edit the docker-compose.yaml

[MarieSaphira]

if that isn't too much work on the DL side?

It's ~6 default ports, so 6 lines in a .properties file, the parsing is already implemented.

[OrfenCLMS]

Public ports can be added by users if they wish to do so directly on the docker compose.

Also in the DL file. The goal is not to edit the docker-compose.yaml

Alright, so to summarize, the only change that will happen is that no public ports will be generated by default on the DL model? Is that correct?