search

cytopia / devilbox

A modern Docker LAMP stack and MEAN stack for local development
http://devilbox.org
MIT License
4.39k stars 653 forks source link

docker-sync support #105

Open bsormagec opened 7 years ago

bsormagec commented 7 years ago

I think docker-sync support will be good for mac because docker has poor performance on osx. Can you add this to devilbox ?

cytopia commented 7 years ago

@bsormagec thanks for pointing it out. Can you direct me to some more documentation about this.

bsormagec commented 7 years ago

Sure,

https://github.com/EugenMayer/docker-sync/wiki/2.-Configuration

vzlatev commented 6 years ago

Hello,

Any plans to include it in the upcoming releases - on OSX without docker-sync web apps are awfully slow.

Regards

coredevel commented 6 years ago

+1, without docker-sync there are a couple of options for os x when mounting shares using bind mounts:

(Source: https://docs.docker.com/engine/admin/volumes/bind-mounts/#configure-mount-consistency-for-macos)

Unfortunately though none of them appear to resolve the bi-directional sync without huge performance overheads (in the case of the default consistent setting) or persistence issues (e.g. if uploading images to a cms and building up a site's content locally with the docker container instance). Using vagrant in the past for large Magento builds I tried both unison and gatling-rsync, the latter being more performant and the final solution chosen which worked wonders. It ended up being a v similar setup to the docker-sync native_osx implementation, more reading material here:

As an aside a short term workaround to resolve performance issues at the moment is to use memcache/redis caching which works really well, managed to get a basic WordPress instance sub 80ms fully rendered in the browser!

mad99 commented 6 years ago

Any news about a well usable and reasonably fast system under osx?

Have tried to extend the Makefile on my own (process to include docker-sync-stack). But it does not work well... Maybe someone else is also to try to do this way. Would great to share or hear from news in this directions!

cytopia commented 6 years ago

This is still on the agenda, however the roadmap is currently really filled and me alone working on that takes some time. Probably somebody wants to jump in on this issue and submit a PR.

johnwbaxter commented 6 years ago

I tried setting up docker-sync but ended up somehow making docker use 400% cpu...will try again though as it's almost unusable on OSX as it stands.

johnwbaxter commented 6 years ago

@cytopia What would it cost to sponsor you to move this to the top of the list?

cytopia commented 6 years ago

@johnwbaxter I'd rather have someone helping me on this issue, as I dont' have a Mac at the moment. So I cannot get sponsored on this one and make an unreliable promises/guess. But thank you for bringing this up.

cytopia commented 6 years ago

There seems to be a native approach by Docker for volume mounts with the following options:

How would one of those strategy perform against the current mount options:

https://docs.docker.com/docker-for-mac/osxfs-caching/#tuning-with-consistent-cached-and-delegated-configurations

Might give it a try. If this does not work out, I will have alook into docker-sync

cytopia commented 6 years ago

@bsormagec @vzlatev @richdel @mad99 @johnwbaxter

The above suggestion should now be easily achievable via global mount options: https://devilbox.readthedocs.io/en/latest/configuration-files/env-file.html#mount-options

ericgerhardy commented 6 years ago

Has anyone solved this problem by using docker-sync or the global mount options? If yes, could you please tell us your configuration? Thanks in advance!

cytopia commented 6 years ago

@ericfynnmaier have you already tried one of these:

MOUNT_OPTIONS=,consistent
MOUNT_OPTIONS=,cached
MOUNT_OPTIONS=,delegated

Probably ,cached will give you some performance boost. I have no OSX anymore to try. Can you give it a shot and report how well it performs in terms of the empty default

ericgerhardy commented 6 years ago

@cytopia thank you! I added ,cached to the .env and e.g. Wordpress loads way faster!

vielhuber commented 6 years ago

,cached only seems to improve performance on mac, but not windows. how do windows people work with devilbox in a performant manner?

cytopia commented 6 years ago

@vielhuber I haven't looked into Windows yet for those kinds of performance improvements. Any idea what alternatives are offered for Windows

vielhuber commented 6 years ago

The only tool everybody is mentionning is "docker-sync" (http://docker-sync.io/). This seems to solve the problems everywhere. But it seems quite complex to install on Windows.

cytopia commented 6 years ago

OK, didn't know that this also works on Windows.

vielhuber commented 6 years ago

On which OS do you work? Do you also suffer from performance problems like everybody else when spinning up a normal Laravel project?

cytopia commented 6 years ago

I started this project on OSX and am now on Debian Linux. So far no performance issues (on Linux).

Update: I even tried Magento and Shopware, both very heavy projects and they have a decent speed on my machine.

vielhuber commented 6 years ago

Nice; Linux seems to have zero problems.

One temporary solution: Can we disable all filemounts in devilbox and to the sync on our own (e.g. with GoodSync via SSH)?

jeffwhitfield commented 6 years ago

I would love to know how to get docker-sync working with Devilbox. Running Magento on my Mac is absolutely atrocious! Tried everything but docker-sync which, based on what I'm reading, will result in a serious performance bump.

anselmdk commented 5 years ago

Just adding myself here. I spent the day yesterday getting up to speed with Devilbox (I absolutely LOVE it -great work!), however I'm also experiencing performance issues on the Mac...

krebbi commented 5 years ago

What about using SSHFS to access the dockers document root from the host?

https://github.com/libfuse/sshfs

chiribuc commented 5 years ago

Hi guys,

I think I successfully installed docker sync on my Mac.
This are the steps to make it work on your Mac OS machine:

Step 1.

Install docker-sync:

gem install docker-sync

Install unison:

brew install unison
brew install eugenmayer/dockersync/unox

Step 2.

Go to your devilbox directory, where you have the docker-compose.yml file and create two new files, docker-sync.yml and docker-compose-dev.yml.

In the docker-sync.yml file paste in the following but make sure to change the volume names and paths to reflect you local structure:

version: '2.1'

options:
    # the path to devilbox docker-compose.yml file
    compose-file-path: 'docker-compose.yml'

    # the path to your custom docker-compose-dev.yml file
    compose-dev-file-path: 'docker-compose-dev.yml'

syncs:
    # this name has to be unique and is used in docker-compose-dev.yml as well
    wordpress-sync:
        # which folder to watch / sync from - you can use tilde (~), it will get expanded. Be aware that the trailing slash makes a difference
        # if you add them, only the inner parts of the folder gets synced, otherwise the parent folder will be synced as top-level folder
        src: '~/data/www/wordpress/'

        # this does not user groupmap but rather configures the server to map
        # optional: usually if you map users you want to set the user id of your application container here
        sync_userid: '1000'

        # optional, default can be either rsync or unison See Strategies in the wiki for explanation
        sync_strategy: 'unison'

        # optional, a list of excludes. These patterns will not be synced
        # see http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html#ignore for the possible syntax and see sync_excludes_type below
        sync_excludes: ['.gitignore', '.git/', '.idea/', '.vscode/']

    # other app here 
    other-app-sync:
        src: '~/data/www/other-app/'
        sync_userid: '1000'
        sync_strategy: 'unison'
        sync_excludes: ['.gitignore', '.git/', '.idea/']

    # and so on...

In the docker-compose-dev.yml paste in the following but make sure to change the volume names and paths to reflect you local structure:

version: '2.1'
services:

bind:
    volumes: 
        # set the apps that you want to sync
        - wordpress-sync:/shared/httpd/wordpress:nocopy
        - other-app-sync:/shared/httpd/other-app:nocopy

volumes:
    # set each app volume as external
    wordpress-sync:
        external: true
    other-app-sync:
        external: true

Step 3.

Run docker with docker sync using:

docker-sync-stack start

To stop the devilbox ctr+c like always and docker-sync stop. I have an alias for this in my zsh.

It's working for me so I hope will work for you.

chiribuc commented 5 years ago

P.S. - It seems that for me it works even better if I use native_osx as sync_strategt in the docker-sync.yml file.

cytopia commented 5 years ago

@chiribuc This is really great news. I've also linked it here in case someone stumbles across this issue via the forum: https://devilbox.discourse.group/t/fix-macos-performance-issues-via-docker-sync-mvp-available/72

Btw, if you do have bigger applications, can you tell something on the performance gain with docker-sync in place vs without.

chiribuc commented 5 years ago

@chiribuc This is really great news. I've also linked it here in case someone stumbles across this issue via the forum: https://devilbox.discourse.group/t/fix-macos-performance-issues-via-docker-sync-mvp-available/72

Btw, if you do have bigger applications, can you tell something on the performance gain with docker-sync in place vs without.

I will try these days but at a simple look, the speed increased with more than 1 second.

cytopia commented 5 years ago

In the docker-compose-dev.yml paste in the following but make sure to change the volume names and paths to reflect you local structure:

version: '2.1'
services:

bind:
    volumes: 
        # set the apps that you want to sync
        - wordpress-sync:/shared/httpd/wordpress:nocopy
        - other-app-sync:/shared/httpd/other-app:nocopy

volumes:
    # set each app volume as external
    wordpress-sync:
        external: true
    other-app-sync:
        external: true

@chiribuc why are you overwriting bind in docker-compose-dev.yml and not the httpd and php container which actually specifiy the mounts? bind does not have any volumes anyway.

So I was thinking of something like this instead:

version: '2.1'

services:
  php:
    volumes:
      - devilbox-intranet-sync:/var/www/default:nocopy
      - devilbox-data-www-sync:/shared/httpd:nocopy
  httpd:
    volumes:
      - devilbox-intranet-sync:/var/www/default:nocopy
      - devilbox-data-www-sync:/shared/httpd:nocopy

volumes:
  # set each app volume as external
  devilbox-intranet-sync:
    external: true
  devilbox-data-www-sync:
    external: true

With the following docker-sync.yml:

version: '2'

options:
  compose-file-path: 'docker-compose.yml'
  compose-dev-file-path: 'docker-compose-dev.yml'

syncs:
  devilbox-data-www-sync:
    src: 'data/www/'
    sync_userid: '1000'
    sync_strategy: 'unison'
    sync_excludes: ['.gitignore', '.git/', '.idea/', '.vscode/']

  devilbox-intranet-sync:
    src: '.devilbox/www:/var/www/default/'
    sync_userid: '1000'
    sync_strategy: 'unison'
    sync_excludes: ['.gitignore', '.git/', '.idea/', '.vscode/']

Also note that docker-sync.yml did not allow me to specify version: '2.1', it asks for version: '2'

cytopia commented 5 years ago

I am currently trying to make a quick PR for testing, so it can still go into v1.0.0 next week. A few issues I already stumbled across:

chiribuc commented 5 years ago

I am currently trying to make a quick PR for testing, so it can still go into v1.0.0 next week. A few issues I already stumbled across:

It is better to use sync_strategy for each volume instead of the global default_strategy.

cytopia commented 5 years ago

It is better to use sync_strategy for each volume instead of the global default_strategy.

Good point, but this does not change the default_strategy issue on a per volume base either.

chiribuc commented 5 years ago

@cytopia you asked me why are you overwriting bind in docker-compose-dev.yml and not the httpd and PHP container which actually specify the mounts? bind does not have any volumes anyway.

I know it sounds strange but I symlink some WordPress development plugins from some external directories, outside of devilbox or docker and they don't work if I set php or httpd containers to be overwritten.

chiribuc commented 5 years ago

The purpose, I think, it's not necessary to overwrite the containers but to tell docker that docker-sync will use those volumes to sync data.

cytopia commented 5 years ago

I know it sounds strange but I symlink some WordPress development plugins from some external directories, outside of devilbox or docker and they don't work if I set php or httpd containers to be overwritten.

Can you check if the following workaround would make sense:

  1. Set your project directory to somewhere outside the Devilbox (where also the other word-press plugins are)
  2. Disable the wordpress plugins from being shown in the intranet/ not have vhosts created for them:

This would then not require you to have something outside the Devilbox (which is related to the Devilbox) and still have a clean WebUI and only vhosts that are actually vhosts.

cytopia commented 5 years ago

I know it sounds strange but I symlink some WordPress development plugins from some external directories, outside of devilbox or docker and they don't work if I set php or httpd containers to be overwritten.

This really sounds strange :-)

I would assume that when not set on httpd and php that you will probably run a mix of docker-sync and Devilbox volumes.

chiribuc commented 5 years ago

I'm in the root of my user ~/ and my structure is like this:

- devilbox
- data
  - www
     - wordpress-site
         - wp-content
           - plugins

- repositories
  - wordpress
     - plugins
        - some-plugin
           -- ***plugin-name***
           -- src
           -- webpack.conf.js
           -- composer.json
           -- package.json
           -- etc.
        - some-other-plugin
          -- ***some-other-plugin-name***
          -- src
          -- etc.

So in this case, I will symlink the plugin-name which contains all the plugin built files, to the wordpress-site/wp-content/plugins directory. I tried different scenarios but only the one that I use seems to work without any problems.

cytopia commented 5 years ago

@chiribuc can you create a separate issue for that including how you've altered docker-compose.yml in order to achieve such a symlinking magic.

chiribuc commented 5 years ago

@chiribuc can you create a separate issue for that including how you've altered docker-compose.yml in order to achieve such symlinking magic.

I haven't altered it directly. On the docker-compose.overwrite.yml file, I added

version: '2.1'

services:
    php:
        volumes:
            - $HOME:$HOME

    httpd:
        volumes:
            - $HOME:$HOME
cytopia commented 5 years ago

I guess this is something that you probably also want to have in docker-sync or? Can imagine that performance on this will also degrade otherwise.

cytopia commented 5 years ago

Docker Sync Support (PR open)

@bsormagec @vzlatev @richdel @mad99 @johnwbaxter @ericfynnmaier @vielhuber @jeffwhitfield @anselmdk @krebbi @chiribuc @OksanaRomaniv @akuehn @lesteak @Yame- @ohrlab

I have created a first working draft including instructions how to get this working.

Thanks again to @chiribuc jumping in and providing a working example.

buderus199 commented 5 years ago

I have tested docker-sync on my updated windows 10 machine with latest stable docker for desktop. Mainly because of slow magento 2 project.

I'm using latest devilbox from master branch and I have added docker-sync.yml & docker-compose-dev.yml file (from docker-sync branch) to devilbox root folder. I did run docker-sync and get "success Starting Docker-Sync in the background", but can't see any improvements in loading times or deployment (php bin/magento setup:static-content:deploy) for magento 2 project. Probably I'm missing something?

cytopia commented 5 years ago

@buderus199 Did you adjust (uncomment) the sync_strategy according to what is required for Windows?

https://github.com/cytopia/devilbox/pull/535/files#diff-09bf18a28f8967be55bc552a00f967dfR32

buderus199 commented 5 years ago

@cytopia yes I have used sync_strategy: 'unison'

This is the output when running docker-sync start from Ubuntu WSL:

/var/lib/gems/2.5.0/gems/docker-sync-0.5.11/lib/docker-sync/update_check.rb:41: warning: Insecure world writable dir /mnt/c in PATH, mode 040777 ok Starting unison for sync devilbox-intranet-sync doing initial sync with unison Unison 2.51.2 (ocaml 4.06.1): Contacting server... Looking for changes Reconciling changes Propagating updates UNISON 2.51.2 (OCAML 4.06.1) started propagating changes at 14:30:48.56 on 29 Apr 2019 [BGN] Copying from /app_sync to /host_sync [END] Copying UNISON 2.51.2 (OCAML 4.06.1) finished propagating changes at 14:30:48.56 on 29 Apr 2019 Saving synchronizer state Synchronization complete at 14:30:48 (1 item transferred, 0 skipped, 0 failed) real 0m 0.04s user 0m 0.04s sys 0m 0.00s chown ing file to uid 1000

And it stays like this for hours. Not sure if it is normal to stay like this...

cytopia commented 5 years ago

@buderus199 If the docker-sync command stales, its probably something to ask these guys: https://github.com/EugenMayer/docker-sync/

ClearMarble commented 5 years ago

I'm using the PR for this and noticing a definite performance improvement on MacOS. CPU usage seems significantly lighter on page loads.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ClearMarble commented 4 years ago

Hey, do you imagine this would still work if I updated to the latest version? Been holding off updating due to this but don't know if anything relevant has really changed.

pirex360 commented 3 years ago

Still at this date, the last version of devilbox on mac , using laravel...very slow.

krebbi commented 3 years ago

Same here. It's incredible slow on my Macbook Pro. I have to restart the Docker Images every few hours, otherwise they are slowing down over time.