search

EugenMayer / docker-sync

Run your application at full speed while syncing your code for development, finally empowering you to utilize docker for development under OSX/Windows/*Linux
GNU General Public License v3.0
3.53k stars 289 forks source link

Discuss if we should switch back to unison as default for now #370

Closed EugenMayer closed 7 years ago

EugenMayer commented 7 years ago

Due to the increasing number of issues with native_osx like #366 things become complicated for users.

We probably should go back to unison as default until those things have stablized - i probably was to optimistic with native_osx - we need more time to get things stable.

Sure, native_osx stays available, but not as the default - what do you guys think?

michaelbaudino commented 7 years ago

I totally agree we should switch back to unison as default 👍

Also, what about making auto upgrade an "opt-in" option (users would have to use --auto-upgrade explicitely) ?

ignatiusreza commented 7 years ago

hmmm.. ubuntu user here.. so, I can't really have a say.. :grin: .. though, I'll ask my fellow worker if they encountered any issue..

in term of auto upgrade being opt-in.. I guess we should consider it together with #304 by having an entry in the global config for auto upgrade behavior.. something like it's being off, confirm, or on..

rogamoore commented 7 years ago

Good idea - let the user decide if he wants to try the new strategy.

Zakay commented 7 years ago

I also suggest not requiring user input just displaying info/warning messages is enough. This to avoid breaking automated builds. Thing is even after you migrated to new configuration it still asks the questions, giving the feeling that the configuration that was just made still not up to date.

rwilliams commented 7 years ago

My laptop battery will be sad but i'll be happy knowing that it works consistently if we switch back to unison.

EugenMayer commented 7 years ago

@rogamoore the user can decide about his strategy since the start of docker-sync and even/alos now, so what do you mean by that? We just decide about the default (or rather offer one), the one you get if you did not decide yourself. The choice never has been taken, nor a strategy has been over removed

@Zakay that wont happen. The way docker-sync is upgraded, the way information is pushed so that users actually read the upgrade notes, read the changes, is required so the issue queue is not spammed with issues about the same thing over and over again. The point behind that is, that people not caring the RTFM will block other issues from getting done by flooding the queue, which will stall the project in the end - and i would like to avoid that and facing the consequences of broken updates, of issues and firefighting.

@rwilliams well we keep you posted, native_osx is by no means dead. But out of any reasons i expected it to be a smooth ride - the newer unexpected issues proven to be not talking the same language, so thats why i have to give myself defeated most probably, rolling back the default until the native_osx strategy is bullet proof ( or close to such )

codycraven commented 7 years ago

@EugenMayer while the native_osx migration came unexpectedly for our team (due to gems aggressively updating on their own). Now that we have our configuration modified and kinks worked out we're seeing a 2:1 improvement with native_osx over Unison.

Unless we run into some show stopper with native_osx we're going to continue using it for our team of 9 devs.

EugenMayer commented 7 years ago

@codycraven great news. I am not going to disable or remove native_osx, rather trying to make it a bit more robust and reintroduce it later in 0.5.x

EugenMayer commented 7 years ago

I fixed the issue that docker compose starts prior the initial sync now, try

gem uninstall docker-sync
gem install docker-sync --version 0.4.4.beta1

that was the most critical issue. Also introduce host_disk_mount_mode https://github.com/EugenMayer/docker-sync/blob/master/example/docker-sync.yml#L34 @codycraven if someone is using docker-for-mac edge, you could try using that to speed up the initial sync - since a sync time is given in the cli, i would be interested in how things improve.

With that done, i have no longer any critical issues open for native_osx - still any concerns?

ghost commented 7 years ago

I truly appreciate the work this project has done. Without it, using docker in development would not be realistic for many projects. But the constant updates creates a loss of productivity nearly every day (feels like it at least). Please create a stable/edge channel. Literally just updated strategies hours ago and now there is talk about rolling back / using a beta gem to get the current release working properly. This is not a fun user experience. You don't owe me anything. But I would massively appreciate if you would consider my feedback. Thanks.

ignatiusreza commented 7 years ago

some of my coworker have issues:

EugenMayer commented 7 years ago

@ignatiusreza do not use this as an bundle-issue :) a) possible since 0.4.3 - sync sync_args are now also available for native_osx b) needs far more analysis - seems the container has crashed or whatsoever - please open an issue and try to see what happens ( docker logs .. /tmp/unison.log .. /tmp/unisonXXX.log supervisor logs )

EugenMayer commented 7 years ago

@cartrdgeproduct i consider the 0.4.x release too early and and issue i have created - i apologise. I rushed it and i now face the consequences.

Right now, the constant updates, i try to work on this issue i created and remove its burden. I understand that every morning can be painful right now - on the other side, i have to put in far more work as expected into docker-sync, also blocking some of my daily work things.

I will be more conservative next time and the "beta" releases are one way to do that, since you will not be noticed about those. Still, docker-sync has not the volume of d4m, its not a full-time product with enterprise drifts and a whole drift behind it - so its hard to cope with the same QA.

Its a hobby project of myself - that one must be considered.

ignatiusreza commented 7 years ago

@EugenMayer sorry, was thinking that feedback about issue could give some value into weighting the decision of whether we should switch back or not.. I'll re-raise both of them separately..

michaelbaudino commented 7 years ago

I totally agree and understand your position @EugenMayer and would kindly recommend this to avoid breaking things:

  1. configure automated (regression) tests (see my comment on #23)
  2. follow semantic versioning
  3. make auto-upgrade opt-in only
EugenMayer commented 7 years ago

@michaelbaudino i understand your feedback, but sometimes we have to adjust to the reality :)

  1. docker-sync is not a product, not commitment of a company. Its a hobby project by devs for devs. If people want more control, more improvment, want to know when things break - they have to get involved. If they do not, they will be suprised.

  2. I have got complains about people criticizing that we use a upgrade-quesiton "did you read this. The point is, people are not getting involved, they are not watching and expect their magic tool to just work, migrate, fix and automatically do "whatever needed" to please them. It would be great if we could do all that - but that wont happen in this reality. We build a dev tool for developers - and we are those too. We will concentrate on fixing performance, issues and improve overall, but not fix the automatic migration, helping you find out why you should not have skipped reading the upgrade nodes. In fact, i think we do a good job being transparent and tell the user, when he should wake up to read things changed. A far better job then most of the projects we regularly use. Usually you just run into things without even having a clue why / what and how to fix this. We keep the release rolling and are more light footed. this can create more issues, but also fixes them quick and i think over all the quality becomes better over time

  3. Why we have opt-out auto-upgrade is actually a trade. We take from the users and give to the developers of the tool. There is no real arguing that this approach has negative aspects, mostly on the side of the users. But it does protect the contributors of dealing with a extraordinary amount of issues of people "forgot to upgrade". You will debug, and help to only find out, that the user forgot to update and "it already has been fixed". This leads to frustration, leads to huge issue queues what leads to the "i could not less about how many issues there are" and that leads to "everybody care about their own issues only" and in the end, it harms the user a far dramatically way: real issues are no longer addressed. And then, as long as it works for contributor a-z, nothing will happen anymore. And then the user has ultimately lost

  4. Tests to improve Q/A - yes please! But do not forget, its a part time project and you all can please get involved helping doing that.

If this project is really as critical for your team as you reflect - then get involved

You cannot expect a few to not only do the work, but to the work in the timeframe you require. That will not work with oss

It might be that some comments above sound harsh - but do not take them as those. You all know the reality out that, the forks of the forks, to the forks, to the alternative to the fork. And really all of them lack building a framework / ecosystem and rules, where they can actually survive more then 4 months.

We have 98% abandoned projects, and 2% actually work. And there is a reason behind that.

So i just try to set rules to ensure, docker-sync is not just one of this stalled projects.

And this involved tradeoffs, not only for the contributors, but also for the users. Even though i try to make both parties as happy as possible, most projects forget to make the contributors happy - and that really matters.

So you find some rules/strategies in here, which do taker for users and do give to contributors - by purpose. Since if contributors are happy, they will give back to the users anyway :)

michaelbaudino commented 7 years ago

Great enlightenment @EugenMayer, thanks a lot and I agree with all points (and I'm trying to get involved, as you can see, so I'll try to keep in mind both users and contributors happyness 😃 ).

And let me remind you that your work is highly appreciated ❤️

EugenMayer commented 7 years ago

Thanks @michaelbaudino !

Back to the topic, for now i see one blocker https://github.com/EugenMayer/docker-sync/issues/376 which i probably can fix today evening - but other then that, working with native_osx does work out to be very very well here.

i released beta2

gem uninstall docker-sync
gem install docker-sync --version 0.4.4.beta2

give it a shot - before i release it and you might complain with the whole team :)

rwilliams commented 7 years ago

Beta2 is looking good so far. Beta1 was hanging or failing the initial precopy and never started docker compose.

Benchmarks

Late 2016 Macbook pro on battery docker-sync .4.4.beta2 - d4m edge Syncing one folder - 190mb 7.8k items

Native osx

Run 1

real    0m 24.71s
user    0m 0.29s
sys 0m 2.34s

Run 2

real    0m 23.14s
user    0m 0.30s
sys 0m 1.87s

Native OSX with host_disk_mount_mode: 'cached'

Run 1

real    0m 12.03s
user    0m 0.21s
sys 0m 1.39s

Run 2

real    0m 14.12s
user    0m 0.10s
sys 0m 1.82s
EugenMayer commented 7 years ago

published 0.4.4 since it does do it better in all terms - used it the whole day pretty productive and others stated its working fine.

do no longer use the betas, since unsion:hostsync is rolled back to support < 0.4.3 while 0.4.4 will use versioned docker-images, nowdays unison:hostsync_0.1 https://hub.docker.com/r/eugenmayer/unison/tags/

EugenMayer commented 7 years ago

wow @rwilliams thank you! impressive already with :cached - that was what i have expected.

I just published 0.4.5-beta1 

gem uninstall docker-sync
gem install docker-sync --version 0.4.5.beta`

it does use unison for the intial sync this will:

in my case, without ignores, 970MB app_sync, with ignores, 137.

During runtime, this makes a difference beside also having ignores not being in effect is inconvenient.

This can slow down the initial copy time, but will speed up the unison daemon startup, since files ar e indexed.

With this fix, if it works out for you, i actually gained some trust in native_osx again. The :cached benchmarks by @rwilliams are promising in addition, so edge adds even more performance to the game.

Today, developing on a 970MB with about 40k files was not even triggering my CPU, while with unison, one core is on full load, if not 2. It makes a huge difference.

rwilliams commented 7 years ago

0.4.5-beta1 still isn't respecting my sync_excludes

Also 'cached' is 2x faster

EugenMayer commented 7 years ago

@rwilliams could you open a new issue for the excludes and explain how you measured it? Thanks!

rwilliams commented 7 years ago

@EugenMayer It's a non issue. I used the wrong syntax in my excludes.

ghost commented 7 years ago

@EugenMayer Yeah, that's totally reasonable. Do what is sane for you, for sure. Thanks again for all of your work on docker-sync 😄

EugenMayer commented 7 years ago

Things gone a lot better in the meantime - we had a bit of a silent moment, things improved for everybody and people actually report very good result with native_osx.

We might have fought through it, so i stay with the default.

Still considered this as a lesson for myself - nothing close to a smooth ride - apologies