search

weso / rdfshape-api

API for validating and transforming RDF, ShEx, SHACL and more.
https://www.weso.es/rdfshape-api/
MIT License
36 stars 10 forks source link

Create a docker image with rdfshape-client, wikishape and rdfshape server all together #239

Closed labra closed 2 years ago

labra commented 3 years ago

Create a docker image that contains the following components:

In the case of rdfshape, some time ago we already created a docker version using sbt-native package

labra commented 3 years ago

We may consider include sheXer in the docker image.

ulitol97 commented 3 years ago

We could dockerize all applications individually and then use docker-compose to launch them altogether. What do you think?

labra commented 3 years ago

Yes, that's also what I was thinking.

ulitol97 commented 3 years ago

Both server and client are containerized now, you may see the Dockerfiles in the repositories and further explanations in their README files.

After some testing I consider the images to be good enough for now, so I uploaded both of them to Docker Hub with decent READMEs too✌🏼 (api, client).

Next steps are:

ulitol97 commented 3 years ago

Status update: I'm already deploying the full RDFShape stack locally with docker-compose without any problem. A third container using Nginx as a reverse proxy has proven to be the best way to handle connections (so that everything is accessible on port 80) and SSL certificates.

Some questions I'd like help with: Regarding the automatic recovery of failing containers, it is possible to use the healthcheck instruction in a dockerfile for Docker to mark a container as "unhealthy". However, it seems Docker itself isn't capable of restarting unhealthy containers automatically and I'm struggling to find the best solution for this problem in our use case. Some alternatives I've found:

  1. Having another container with an "always-restart" policy monitoring the rest of the stack (willfarrell/docker-autoheal looks like a popular solution).
  2. Having the healthcheck itself kill the container's main process, effectively killing it; as seen here.
  3. Using a container orchestration tool like Kubernetes or Docker's swarm mode. This looks like the least "hacky" way of doing things and may be easily deployable to Amazon's EKS in the future, but using these tools may be overkill for us.

Any ideas? @weso/rdfshape

thewillyhuman commented 3 years ago

Good! I think it's great that we already have it dockerized. I don't know if there is a CD pipeline that uploads the images to a repository right now. If not, it could be very interesting to explore this. Thus, every time something is modified in rdfshape, the corresponding image is automatically published.

Regarding the ephemeral containers (containers that can die and lift automatically), I think it's great! As I understand it, that is the philosophy of containers. Regarding how to do it, I have always used a container scheduler for it (swarm/k8). Your option 3. I don't know if the other two alternatives work. But, if you want to explore them, count on me!

ulitol97 commented 3 years ago

Yes, I'm planning on spending some time on creating automated actions that build and upload our docker images from our repos to Docker Hub.

Regarding the ephemeral containers, I'll give it a second thought in the meantime. For now I think I'll do some testing with K8s and keep you all informed ^^.

thewillyhuman commented 3 years ago

Not sure if GitHub provides an images registry. If it does, and if it fits our needs, we can centralize our artifacts and software development pipelines on GitHub. This is related to https://github.com/weso/shex-s/issues/185.

ulitol97 commented 3 years ago

Sounds like a good idea and alternative to Docker Hub (taking for granted that GitHub Container Registry suits us). I reckon that GCR integrates well with GitHub actions. Any opinions @labra?

ulitol97 commented 3 years ago

Update: