Transfer files more efficiently.

[danieljimenez]

When I use -l debug, it seems like test kitchen is transferring every file one at a time, very slowly, rather than taring them before transfer. I'm not sure if it uses something under the hood, but this step takes over 2 minutes on some of my cookbooks.

[jessecrossen]

I'm seeing this as well. It seems like an even better solution would be to use rsync or similar.

[danieljimenez]

Yeah, +1 for rsync.

[jessecrossen]

@danieljimenez Looking at the source code it seems like in principle it would be relatively easy to use something like rsync, however the current code seems to use https://github.com/net-ssh/net-scp, and I'm not aware of any similarly abstracted (i.e. pure Ruby) equivalent for rsync. If anyone is able to find such a module, I'd be willing to take a stab at forking and writing in this feature if I can find the time.

Of course, yet another alternative would be to use modification times to skip files that are unmodified since the last run.

[pelletiermaxime]

https://github.com/coderanger/kitchen-sync

[jessecrossen]

@pelletiermaxime Thanks!

[jessecrossen]

But it does beg the question: why is this functionality in a separate repo that monkey patches its way into test-kitchen? Would anyone like to see this implemented as a pull request to this repo?

[danieljimenez]

I couldn't get it to work, just... nothing happened.

[jessecrossen]

@danieljimenez I couldn't get it to work either. I think it might be because the monkey patch was made for an earlier version and no longer maps onto anything useful, but I don't know for sure. That's why I'm suggesting making that functionality a part of test kitchen itself. It seems strange to have a modular transport system with no built-in alternatives.

[pelletiermaxime]

Ah, you're right, kitchen-sync seems broken since test-kitchen 1.4. But in my tests, the new version is way faster, probably thanks to the added compression in the ssh transport.

[danieljimenez]

I've been using 1.4 for a bit now, and still see very slow transfers when using kitchen-ec2.

[iiro]

I see very slow transfers with 1.4.0 too with kitchen-ec2 - making Test-Kitchen almost useless.

Not sure where the true problem lies:

https://github.com/test-kitchen/kitchen-ec2/issues/93

https://github.com/coderanger/kitchen-sync/issues/8

[iiro]

Any updates here...?

[northvankiwiguy]

I have a fix that I'll be creating a pull request for in the next day or two:

https://github.com/northvankiwiguy/test-kitchen/tree/ssh_tgz_transport

This adds a new transport for tar-gzipped upload of files. Please give it a try and see if it solves your problems. You'll need to add the following to .kitchen.yml:

transport:
   name: ssh_tgz

[iiro]

I tried to use it:

gem list|grep test-kitchen

test-kitchen (1.4.3.dev.0)

but in converge I still get:

M# kitchen conv -----> Starting Kitchen (v1.4.3.dev.0)

------Exception------- Class: Kitchen::ClientError

Message: Could not load the 'ssh_tgz' transport from the load path. Please ensure that your transport is installed as a gem or included in your Gemfile if using Bundler.

Please see .kitchen/logs/kitchen.log for more details Also try running kitchen diagnose --all for configuration

Ideas?

[northvankiwiguy]

If you're using a locally checked-out copy of test-kitchen, you'll need to set your Gemfile to point to the correct path of your local test-kitchen directory. Also, you'll need to run it with bundle exec kitchen conv.

I had a similar problem, but those things solved it for me. Let me know if that helps...

[iiro]

Yep, I cloned it onto the disk. I have a ChefDK 0.8.1 installed also.

I have now this in my Gemfile:

gem "test-kitchen", :path => "/Users/iiro/tmp/test-kitchen"

and I'm still getting the error when running:

chef exec bundle exec kitchen conv

-----> Starting Kitchen (v1.4.3.dev.0)

------Exception------- Class: Kitchen::ClientError

Message: Could not load the 'ssh_tgz' transport from the load path. Please ensure that your transport is installed as a gem or included in your Gemfile if using Bundler.

Please see .kitchen/logs/kitchen.log for more details Also try running kitchen diagnose --all for configuration

The log looks like this:

I, [2015-10-01T22:23:00.026684 #27300] INFO -- Kitchen: -----> Starting Kitchen (v1.4.3.dev.0) E, [2015-10-01T22:23:00.067125 #27300] ERROR -- Kitchen: ------Exception------- E, [2015-10-01T22:23:00.067229 #27300] ERROR -- Kitchen: Class: Kitchen::ClientError E, [2015-10-01T22:23:00.067259 #27300] ERROR -- Kitchen: Message: Could not load the 'ssh_tgz' transport from the load path. Please ensure that your transport is installed as a gem or included in your Gemfile if using Bundler. E, [2015-10-01T22:23:00.067284 #27300] ERROR -- Kitchen: ---Nested Exception--- E, [2015-10-01T22:23:00.067305 #27300] ERROR -- Kitchen: Class: LoadError E, [2015-10-01T22:23:00.067324 #27300] ERROR -- Kitchen: Message: cannot load such file -- kitchen/transport/ssh_tgz E, [2015-10-01T22:23:00.067345 #27300] ERROR -- Kitchen: ------Backtrace------- E, [2015-10-01T22:23:00.067367 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/transport.rb:48:in rescue in for_plugin' E, [2015-10-01T22:23:00.067391 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/transport.rb:40:infor_plugin' E, [2015-10-01T22:23:00.067415 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/config.rb:303:in new_transport' E, [2015-10-01T22:23:00.067437 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/config.rb:244:innew_instance' E, [2015-10-01T22:23:00.067458 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/config.rb:135:in block in build_instances' E, [2015-10-01T22:23:00.067479 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/config.rb:134:inmap' E, [2015-10-01T22:23:00.067500 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/config.rb:134:in with_index' E, [2015-10-01T22:23:00.067522 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/config.rb:134:inbuild_instances' E, [2015-10-01T22:23:00.067544 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/config.rb:110:in instances' E, [2015-10-01T22:23:00.067565 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/command.rb:115:infiltered_instances' E, [2015-10-01T22:23:00.067586 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/command.rb:145:in parse_subcommand' E, [2015-10-01T22:23:00.067610 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/command/action.rb:38:inblock in call' E, [2015-10-01T22:23:00.067635 #27300] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.1.0/benchmark.rb:279:in measure' E, [2015-10-01T22:23:00.067681 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/command/action.rb:37:incall' E, [2015-10-01T22:23:00.067707 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/cli.rb:56:in perform' E, [2015-10-01T22:23:00.067733 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/cli.rb:171:inblock (2 levels) in class:CLI' E, [2015-10-01T22:23:00.067756 #27300] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:27:in run' E, [2015-10-01T22:23:00.067788 #27300] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:ininvoke_command' E, [2015-10-01T22:23:00.067812 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/cli.rb:308:in invoke_task' E, [2015-10-01T22:23:00.067847 #27300] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor.rb:359:indispatch' E, [2015-10-01T22:23:00.067883 #27300] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/base.rb:440:in start' E, [2015-10-01T22:23:00.067918 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/bin/kitchen:13:inblock in <top (required)>' E, [2015-10-01T22:23:00.067953 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/lib/kitchen/errors.rb:154:in with_friendly_errors' E, [2015-10-01T22:23:00.067990 #27300] ERROR -- Kitchen: /Users/iiro/tmp/test-kitchen/bin/kitchen:13:in<top (required)>' E, [2015-10-01T22:23:00.068027 #27300] ERROR -- Kitchen: /Users/iiro/.chefdk/gem/ruby/2.1.0/bin/kitchen:23:in load' E, [2015-10-01T22:23:00.068098 #27300] ERROR -- Kitchen: /Users/iiro/.chefdk/gem/ruby/2.1.0/bin/kitchen:23:in

' E, [2015-10-01T22:23:00.068138 #27300] ERROR -- Kitchen: ----------------------

[northvankiwiguy]

Interesting... sorry for asking obvious questions, but did you: 1) Ensure that you've checked out the ssh_tgz_transport branch? The master branch doesn't have the necessary changes. Sorry, I probably didn't make this step clear. 2) Do you see the file: lib/kitchen/transport/ssh_tgz.rb in your source tree? 3) Did you do bundle update?

[iiro]

Maybe I should get some sleep... :)

It was the branch thing ;) Should had understood it from your message :)

The change looks :+1: - works really well.

[northvankiwiguy]

Thanks, I'll be adding some more test cases, and will submit a PR in a day or two. I'm glad it worked for you :-)

[iiro]

Excellent :)

[cheeseplus]

Closing this issue as we're not going to be adding this to core TK at this time and encourage the plugin route a la kitchen-sync https://github.com/coderanger/kitchen-sync

[Lifang13]

we can synced_folder to sync files, but test kitchen still transfer files even the files already synced. How to stop transferring files ?

[RomanKisilenko]

I understand the logic by test kitchen team, but I think a feature to transfer files fast is critical, otherwise it defeats the whole idea of TDD when you have to wait for 30 minutes for a test instance in a cloud to converge.

I think approach with sync for repeat transfers or packaging files into targz on first transfer whould be most efficient.

Do you know any plugins which are compatible with the latest test kitchen version which does like this? We considering contributing to development of such a plugin after waiting for a few days for file transfer on our test instances.

[iiro]

@RomanKisilenko I've been using this one: https://github.com/unibet/kitchen-transport-rsync

With this, the file sync takes about 10 seconds - which is ok. Works with the latest ChefDK as well. I've been upgrading it to the latest and no problems.