dependency volume changes when source code is changed

[ilgooz]

Problematic line. I'm working on WSS with mongodb, when I make a change on source code of WSS, mongodb's data volume also changes and I lose all the previous data.

/cc @antho1404

[antho1404]

I was thinking about this one too when I worked on the influxdb service and yes it can be annoying. In some cases it's actually a good thing to have this all automatic, but in some other cases it's really annoying so maybe it should be the responsibility of the user and have a parameter "bind" or something like that.

[antho1404]

Also the line you are pointing is really risky to remove because what happen if 2 services have the same name of dependency and same name of volume eg: db -> db that might be a big problem because data will be shared

[ilgooz]

I was thinking about this one too when I worked on the influxdb service and yes it can be annoying. In some cases it's actually a good thing to have this all automatic, but in some other cases it's really annoying so maybe it should be the responsibility of the user and have a parameter "bind" or something like that.

Updating a service's source code should never result with reseting the data volumes. This is not an expected behavior in real-world applications.

Also the line you are pointing is really risky to remove because what happen if 2 services have the same name of dependency and same name of volume eg: db -> db that might be a big problem because data will be shared

This is totally right. I have a new idea. To solve this issue we need to introduce the mesg-core service update --path SOURCE_PATH SERVICE_ID functionality. This way, devs can chose to create a new service from the updated version of source core, in this case data volumes will change like now or update an existing service and keep the data volumes as it is. We can create uuids on creation of services to make updating an existent service possible.

If you also agree with this let's create an another issue about this update feature and close this one by referencing to this new issue or forum topic.

[antho1404]

I think the update option is not really related and I think this issue will solve the problem https://github.com/mesg-foundation/core/issues/275 but until then I really think we could either force to bind the volume in the .mesg folder based on the name like we had before or even make an option for that. The final solution is really to work on this "alias/ID" for service that will always be static and we can use that as identifier for the volume and this will not change anymore.

[ilgooz]

moved to https://forum.mesg.com/t/fixed-service-id-name-service-update-create-feature

[ilgooz]

@antho1404 please confirm and close this issue in favor of forum proposal.

[antho1404]

This is still a bug that can be solved by the service ID feature but it should be kept open as reminder

[antho1404]

can't we for now revert to the initial system that was doing the binding all the time of the different volumes on the host disk based on the name of the service (with dash).

This will solve the problem and we can later on use the alias/id of the service.

What do you think ?

[ilgooz]

@antho1404 you mean using the service name in the yaml instead its hash for the volume name?

[antho1404]

yes

but even this might not be enough because with this we still rely on docker to create volumes but we cannot bind these volumes to the actual host so maybe using the name instead of the hash AND binding this to the host in the ~/.mesg/services/{serviceName}/{dependencyName}/{volumePath}

[ilgooz]

I agree, but I'd like to keep Docker volumes instead of binding volumes to host machine.

In future we can have new features like supporting volume plugins, providing data backups to users or easily moving core, its whole stack and data volumes to another machine. Having data in Docker volumes may give us more flexibility for these kind of use cases.

Is there a reason to the bind volumes to host machine?

[antho1404]

I totally agree that binding volumes is not an ideal solution and docker volumes are better but there is some cases where binding can be helpful. One example is to inject configurations in the service.

• We have for example @krhubert who is working on the integration with tendermint and needs to have the genesis file sent to the service (workaround is to have it created in the service with a special script)
• The ethereum service that require an endpoint and/or a network to connect to that could be sent with volumes
• Like you said doing some backup or using existing tools on their host machine

I agree that I would like to keep docker volumes here and not binding, maybe we can have an option here and it would be an undocumented feature for really advanced services. Services shouldn't be configured by the user anyway.

What do you think about that?

[ilgooz]

I totally see your point and sharing networks/files from host machine to services is an actual need. On the other hand we still should be using data volumes.

In this case we need to both support sharing networks/files and having data kept in docker volumes.

I suggest to get some insights from docker-compose too see how they handle this cases.

For data volumes they have this syntax: https://docs.docker.com/compose/compose-file/#volume-configuration-reference

version: "3"

services:
  db:
    image: db
    volumes:
      - data-volume:/var/lib/db
  backup:
    image: backup-service
    volumes:
      - data-volume:/var/lib/backup/data

volumes:
  data-volume:

This one demonstrates how network/files are shared: https://github.com/libresh/compose-nginx/blob/master/docker-compose.yml#L18-L20 Note that ./nginx is an actual folder in their repo.

nginx-gen:
  ...
  volumes_from: [nginx]
  volumes:
    - ./nginx/templates/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
    - /var/run/docker.sock:/tmp/docker.sock:ro

I'm not so sure about exposing an undocumented advanced feature if we can avoid its need. It should be our last choice.

[NicolasMahe]

The latest messages are off-topic and the conversation continued on the PR #571

[antho1404]

this one should be solved by the PR #583 volumes will be binded to the alias and not the hash anymore so whatever the version of the service, the alias stays the same and so the volume