Make docker-sync cross-platform

ignatiusreza commented 7 years ago

Goals

Summarizing the discussion in #250, there are 2 goals in general that we want to achieve in order to make docker-sync cross-platform:

make platform & strategy specific codes pluggable
add volume strategy which would use docker volume create natively

Linux Native Support

Since docker have nice linux support, docker-sync are basically not needed under linux.. but, in order to allow users to share the same docker-compose file between osx and linux users, we would need to be able to somehow parse the config files, and then boot up docker-compose with a modified docker-compose configuration.. for example, the following config:

# docker-compose.yml
version: "2"
services:
  app:
    volumes:
      - syncer:/var/www

volumes:
  syncer:
    external: true

# docker-sync.yml
version: '2'
syncs:
  syncer:
    src: './data'
    dest: '/var/www'

~~would be modified into~~

# docker-compose.yml
version: "2"
services:
  app:
    volumes:
      - ./data:/var/www

~~first before being used to boot up docker-compose~~

Cross Platform Pluggable Strategies

To make our code base cross platform, we would need to restructure code related to configuration and dependency (rspec, fswatch & unison) management into something like:

DockerSync::Config: responsible for parsing the configuration files and turn it into PORO

instance of this class will hold array of instances of either DockerSync::Config::Rsync, DockerSync::Config::Unison under #services.. each service provider would need to handle parsing raw configuration from yml into a format that is understandable by the actual sync provider (to be pass in as command line arguments)..
DockerSync::Dependency: responsible for checking availability and auto installing dependencies.. it will be read from DockerSync::Config and then run platform specific code to check and auto install as necessary.. the code can be structured like:
- DockerSync::Dependency:
- DockerSync::Dependency::OSX:
  - DockerSync::Dependency::OSX::Fswatch
  - DockerSync::Dependency::OSX::Rsync
  - DockerSync::Dependency::OSX::Unison
- DockerSync::Dependency::Volume
- etc
with the possibility for registering more dependencies by doing something like DockerSync::Dependency.register_dependency SomeClass, ->(config) { config.sync_strategy == 'awesome_strategy' }..

We would then need to update DockerSync::SyncManager and make it receive a config object, instead of config path, and also update DockerSync::SyncProcess to receive either one of DockerSync::Config::Rsync or DockerSync::Config::Unison.. which would reduce the responsibility of DockerSync::SyncManager and DockerSync::SyncProcess into just running and managing the actual processes..

Then, we modify the default strategy to be platform dependent, on supported platform (initially only osx), the default sync and watch strategy would be unison, while on other platforms the default would be volume.. This default setup would eventually be updated as unison support on other platform gets baked in..

ignatiusreza commented 7 years ago

@EugenMayer wrote some rough idea to make docker-sync cross-platform.. let me know what you think! :grin:

and in related to the code base, do you currently follow some sort of code style and/or naming guideline?

EugenMayer commented 7 years ago

I like both changes, to the SyncManager and to the Process, to basically just get the config parsed an prepared, while we put the config into a model.

I like the general idea. I think instead of doing under linux what you described here, you could just create a docker-compose.yml ( for production ) and a docker-compose-dev.yml for linux development, which includes the host mount nativily. Then you create a docker-compose-sync.yml and set docker-compose-sync.yml as "dev" compose file in the docker-sync.yml - here you put syncer as volume

Now, people under OSX do docker-sync start and people under linux do docker-compose up - to make docker-compose up work this way (without params) under linux, create a .env file and add

COMPOSE_FILE=docker-compose.yml:docker-compose-dev.yml

see https://docs.docker.com/compose/reference/envvars/

That said, you see, its just a conceptual thing - even with sharing compose files there is 0 needs to use docker-sync under linux

matleh commented 7 years ago

Not sure if you are aware of this - with this command, it is possible to create a Docker volume which just mounts any host location:

docker volume create --opt type=none --opt device=<host path> --opt o=bind <volume name>

For the Docker container, it is then transparent whether that volume is synced with docker-sync or host-mounted.

Note that there is a conceptual difference between volumes and mounts from a container point of view, AFAIK. With this, the container always sees a volume.

matleh commented 7 years ago

From my perspective, the mingling of docker-compose files done by docker-sync-stack is too much "magic". I just run docker-sync to sync directories to Docker volumes and then use these volumes in my docker-compose files or docker-run commands directly. This way I know what's going on and I can use the docker tools as usual - also helps in communication, because the workflow does not depend on docker-sync but just utilizes it. I or other developers can do the same thing without docker-sync. In my opinion, that helps the acceptance of a tool.

What do I want to say? I think, making docker-sync cross-platform (Windows would be a big plus since some of my co-workers are on Windows) is a great goal. When thinking about ways of how this can be achieved, I think it would be beneficial to see docker-sync as an optional tool in the workflow and not something that is forced on everybody on a team.

EugenMayer commented 7 years ago

@matleh docker-sync has been designed to be exactly that, in nearly all cases.

You can use docker-sync-daemon start and then e.g. use docker-compose -f docker-compose.yml -f docker-compose-sync.yml up or even using .env to use ENV to defin which docker-compose files are used when running docker-compose up

As far as i can see, docker-sync offers the easy way of docker stack start together with sync, for any starter. It offers the sync separately and lets you do your own docker-kunfugh afterwards, being as optional as it possible gets

And also offers you a custom way to actually "script" your own "start" by using docker-sync as a lib.

In most cases, when people ask for more freedom so they can use docker in a more general way, they rather yet do miss essential docker or docker-compose feature, not knowing, that it is already possible.

I do not really try to improve docker-sync beyond a level of "how it can be still maintained with a specific number of features" - so i have to stop implementing features and probably things like x-platform so the tool can still be maintained at all by me in my part time.

Offering x-platform for now seems to be too much of a deal, esp. in maintaining this, the extra 90% only windows issues we get in the issue queue and all the new glitches from cygwin to the ubuntu shell.

It would be great to have for sure, it would be also great to have docker-sync operate without any docker-compose file, but its just not worth the effort. Being in need of zero "mangling" and to have to pure docker experience, there is exactly one solution, and thats using linux :)

matleh commented 7 years ago

@EugenMayer I did not mean to criticize - I fully agree that docker-sync gives much freedom and flexibility (especially since version 0.2 uses volumes instead of data-containers). I really like that and just wanted to say that I would like it to stay this way and not have it force a special way of doing things upon its users - even if it is further extended to be cross platform. :)

ignatiusreza commented 7 years ago

@EugenMayer & @matleh Thanks for the feedback.. sorry, i'm still not yet that familiar with what is and is not possible.. but, so far the suggestions that you guys given doesn't sounds sustainable.. sure, I can keep an extra .env locally (.gitignore-ed, since we don't want mac users to see it) or run some extra docker volume command.. but, it means that these extra steps would need to be done everytime someone new join the team and/or some configuration got changed... we can no longer just checkout the code and run docker-compose up..

with that said though, I wonder if it's possible for us to do docker volume create --opt type=none --opt device=<host path> --opt o=bind <volume name> automatically on docker-sync start.. meaning that, we can do os check, then:

if linux, check if volume exists, if not run docker volume create and then do nothing and let docker do its job
else, use either unison or rsync as syncer as dictated by docker-sync.yml..

EugenMayer commented 7 years ago

You can implement that using https://github.com/EugenMayer/docker-sync/wiki/7.-Scripting-with-docker-sync, so just go forward. you will that have yourtool start which does exactly the above.

Adding this to docker-sync is not planned, if you like test drive it for a while - but that is exactly what scripting has been designed for

matleh commented 7 years ago

If I understand you right, you suggest that the docker volume create thing could be a 'fake' sync-strategy of docker-sync and then have some other meta- "best-for-platform" sync-strategy which does a platform check and uses this 'fake' sync strategy on Linux and 'unison' on Mac. This way, the development workflow and the configuration files could be the same for all team members (as long as they use either Linux or Mac). Sounds interesting. It still adds development burden of supporting multiple platforms (at least in some way) on docker-sync.

EugenMayer commented 7 years ago

there is already a "dummy" watcher strategy, you could also add a "direct volume creation" strategy, but no matter what you do, you need to switch the strategy depending on the OS, that will need scripting like outlined above

Adding such a strategy might be in scope of docker-sync, so feel free to contribute it - but the logic behind it (linux / OSX switch) is to far from being generic yet.

any way, thank you for the interesting discussion @ignatiusreza and @matleh

ignatiusreza commented 7 years ago

yup, that's correct.. updated the issue description to reflect this.. I'll see if I can squish some time to work on it in a feature branch..

EugenMayer commented 7 years ago

@ignatiusreza made a lot of effort for the configuration to be more flexible for this purpose

made some effort to split out the precondition section as a strategy patter and made linux stubs.

So we have quiet some progress here :)

EugenMayer commented 7 years ago

I added sync_host_ip auto-detection recently, wich can also help for x-platform, since some use a docker-machine, some do not - this way, you can use the same configuration

EugenMayer commented 7 years ago

Since i am running linux on my laptop in addition for a while, i am actually tackling similar topics and was in need of some features here already. I can now also see a lot of reasons why some would use docker-sync on linux - for team convinience. You have the same scaffolding, no matter which OS, stack starting and them up and go. Stuff can be solved also using different .env and different default docker-compose-dev-linux.yml selection, but startup / instance installation automation is then out of scope or needs to be handled extra.

Since we are using docker-sync as a library in a thor-based dev-stack control tool, having linux support build in is a lot more convenient. So heads up for this effort, i will join the team :)

EugenMayer commented 7 years ago

I think we can consider this implemented with 0.3.6 and 0.4. We have plugable preconditions and a native sync mode for linux

EugenMayer / docker-sync