search

btm / minitest-handler-cookbook

Apache License 2.0
59 stars 36 forks source link

Minitest performance - Gem files #53

Closed Rucknar closed 10 years ago

Rucknar commented 11 years ago

Minitest gem files are downloaded and deployed upon each chef-client run. This adds a lot of time to our runs. Would it be possible to make it cache the gems?

dpetzel commented 11 years ago

Hey @superted666, what version are you running? The gems files are deployed using the chef_gem, so I wouldn't expect them to be downloaded and deployed each run (unless they were removed). Could you provide any output or log data illustrating the issue?

dpetzel commented 11 years ago

@superted666 What version of Chef were you running when you saw this? I'm am seeing painful slowness under 11.8.0

Rucknar commented 11 years ago

@dpetzel Sorry for the slow response. You are of course right, it's not the gem files that are being slow it seems. It seems to me that it's the actual copying over of the tests themselves that are taking the time. The recipe deletes everything and replaces it every time and this can take a while on our setup. We currently have around 140 tests / 300 assertions and copying them to the server in full each time seems unreasonably slow. Perhaps the code only update tests that have been updated?

Rucknar commented 11 years ago

Were using it against the latest Chef Private 11 build. Will check when i'm next in the office.

dpetzel commented 11 years ago

So the challenge I see with that approach is that you have to download them to know if they changed.

I believe it deletes them each run so that if you delete a test from your cookbook, it doesn't remain on the file system, resulting in a test running that you no longer wanted (I think). I think the only real compromise might be to introduce an attribute that (which defaults to true) on if they should be deleted or not?

Rucknar commented 11 years ago

True, is there any way you could compare size of the file or perhaps it's MD5 to avoid the download? If not then i agree your solution is probably the best bet. Chef as a whole doesn't deal with cleaning up behind changes so the fact that old scripts could remain isn't a new challenge.

How would updates be handled if it didn't perform the delete?

dpetzel commented 11 years ago

@superted666, sorry to leave you hanging so long.. I'm honestly not sure what the best approach is, but I do like the idea, I'm just not sure how/when something like this could be done.

Rucknar commented 11 years ago

Done a bit more research and it might be a mute point. I think i'm likely hitting against the slow performance in Chef detailed here : https://tickets.opscode.com/browse/CHEF-4423

Once this is fixed in 11.10 client then hopefully this will be less of an issue and perhaps we can leave it as is?

dpetzel commented 10 years ago

Hey @superted666 Is there anything actionable on this ticket or shall we close it out?

dpetzel commented 10 years ago

Closing this out with the assumption there is nothing actionable here. Let me know if you feel otherwise.