Splitting services across containers

[kreczko]

Currently, all services are run in one big container: https://github.com/perfsonar/perfsonar-testpoint-docker/blob/master/supervisord.conf which is very different to Dockers microservices approach.

Do you have a list of services that only communicate via ports or the file system? I am very tempted to break things apart using docker-compose to make them easier to monitor.

Is there interest for this?

[daldoyle]

Besides being more towards the "spirit of Docker", does this approach have any big wins? One of the goals of the pS Docker effort was to make it really easy to use, and a single container was seen as a good way to make it accessible to people without a lot of Docker knowhow (also selfishly: easy to maintain / debug).

Dan Doyle GlobalNOC Software Developer 1-812-856-3892 daldoyle@globalnoc.iu.edu

On Nov 7, 2017, at 4:29 AM, Luke Kreczko notifications@github.com wrote:

Currently, all services are run in one big container: https://github.com/perfsonar/perfsonar-testpoint-docker/blob/master/supervisord.conf https://github.com/perfsonar/perfsonar-testpoint-docker/blob/master/supervisord.conf which is very different to Dockers microservices approach.

Do you have a list of services that only communicate via ports or the file system? I am very tempted to break things apart using docker-compose to make them easier to monitor.

Is there interest for this?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/perfsonar/perfsonar-testpoint-docker/issues/9, or mute the thread https://github.com/notifications/unsubscribe-auth/AG5fpFXa6TB-MesRMsiUqvy5xGqqnmbQks5s0CLfgaJpZM4QUfTB.

[kreczko]

Generally, I do agree with you. From that perspective, the only thing I would add is a docker-compose file (instead of the Makefile) to explicitly specify the ports and to set volumes (for the database, logs, etc).

However, from a certain point of view, there can be advantages. Firstly, as I understand from perfsonar has a hierarchical structure (please correct me if this view is wrong). However, while the image shows the perfsonar-testpoint as a superset of perfsonar-tools both PS docker descriptions are independent of each other: https://github.com/perfsonar/perfsonar-testpoint-docker https://github.com/perfsonar/perfsonar-tools-docker

Secondly, having some of the services (e.g. postgresql, http) separate allows for simpler backups, scaling (unlikely to be needed for perfsonar), and port mapping.

Such a split can be also beneficial for the actual service development, but that strongly depends on the developer base (a different mindset manifests when one realises that bits of the system could run on different machines).

Anyway, I thought I will at least ask. I will be putting together a docker-compose file anyway in order to handle the generated data & configs outside the container itself (thus keeping it stateless).

[emopinata]

Probably the most beneficial thing to split out and use docker-compose for would be the database, since you can easily spin up the postgres official image and reduce the size of your final image.

[mfeit-internet2]

There are a couple of situations in the installation process where the configuration of the database server is changed and it has to be bounced for those changes to take effect. That might pose problems if the perfSONAR container can't reach into the PostgreSQL container and do that.

[bgrounds]

Another benefit to using multiple containers: allow services to run with different network configurations.

Example use case:

I'm deploying perfSONAR to shared hosts.

While prototyping, I got my setup working on dedicated hosts using docker's --net=host option (which avoids any network virtualization). On the shared hosts I'm deploying to, port 443 is already claimed. This drove me to avoid --net=host, which allows me to remap port 443 inside the container to something that's available on the host machine. However, after making this change, owamp stopped working. After a some investigation, it seems that it stopped working because owamp doesn't support NAT (which is what we get with docker's bridge network) - https://github.com/perfsonar/owamp/issues/49

I did try to configure pscheduler to run on a port other than 443 inside the container ( https://github.com/perfsonar/pscheduler/wiki/Alternate-Server-Ports ) - but couldn't get that working.

My short-term solution was to trade my owamp tests for plain rtt tests, which is what I did for now.

If I could run owamp in its own container, using --net=host, and separately run pscheduler using a bridge network (to map 443 to something else on the host machine), I imagine I could resolve the port conflict issue without sacrificing the one-way tests.

[dimm0]

Horizontal scaling of central MA requires multiple instances pointing at the same DB.. Now it's not possible

[bgrounds]

Horizontal scaling of central MA requires multiple instances pointing at the same DB.. Now it's not possible

@dimm0

I'm not sure if your comment is a response to mine. What is MA, and what does horizontal scaling of MA have to do with running perfSONAR components in different docker containers?

Thanks!

[emopinata]

Horizontal scaling of central MA requires multiple instances pointing at the same DB.. Now it's not possible

@dimm0

I'm not sure if your comment is a response to mine. What is MA, and what does horizontal scaling of MA have to do with running perfSONAR components in different docker containers?

Thanks!

I believe he's talking about central measurement archives:

http://docs.perfsonar.net/multi_ma_install.html

[dimm0]

Yup, sorry my brevity :) To run several measurement archive pods in kubernetes cluster, the components (databases) should reside in different containers - then I can run several containers with django pointed at a single postgres and cassandra

[bgrounds]

Thanks for clarifying!

For my particular use-case, I'm deploying to vanilla docker hosts (not kubernetes), and I'm using the http archiver to send test results elsewhere.

I might be mistaken, but I don't think the caveat's you've pointed out apply in this case.

[kfox1111]

This would be relevant to running perfsonar in kubernetes. Which is of interest to some folks.

Has anyone started work on a Helm chart as well?

[kfox1111]

At least from the kubernetes perspective, heres some reasons having the option of having it broken up would be good:

• Log shipping/rolling/visibility. When process log through stdout, k8s log shippers can capture it and send it along at the cluster level. On a scaled out cluster, this makes logs much easier to deal with. Each container streams out its own logs. No fighting with log rollers either, or making sure you have the right log shipper per container. Load the log shipper on the cluster and your done.
• Separate db's mean you can deploy things like patroni for higher availability of the database in a containerized environment. This speeds up upgrades and minimizes downtimes.

I think a helm chart would be great because some Kubernetes clusters experience internal network issues. Being able to deploy perfsonar along with a tool node on each k8s node would allow much quicker detection of these sorts of issues. There is an untapped userbase here I think.