Intermittent Long Builds

[codeviking]

Our build times vary considerably. Sometimes 5 minutes, other-times 2 hours...

Upon examination it appears it relates to the installation of node v0.12.2 -- but sometimes the installation isn't necessary as it's installation appears to have been cached.

Why is this occasionally cached?

[codeviking]

[codeviking]

The project in question: 540e6be63479c5ea8f9e40b3

[manishas]

@codeviking there was a known issue yesterday with some builds timing out and we deployed a fix for that. We also changed out build system architecture from monolothic to microservices very recently and that migration wiped out the cache for customers.

Our caching is per host so it should build up again over the next few builds.

[manishas]

one quick thing - our node specific images have node 0.12 preinstalled, so maybe you can try using those? minv2 is a gigantic image and it might be better to start using a lighter weight image. The language specific images do not come installed with services and tools though, so you'll need to change your yml to install those.

https://github.com/shippableImages/ubuntu1404_nodejs

[codeviking]

@manishas But we use both scala and nodejs -- is there an image which would satisfy those requirements?

The builds are also slow today, and intermittently caching the nodejs installation.

What would help is clarification of why the image sometimes has to install nodejs and othertimes it doesn't...

[manishas]

@codeviking it sometimes installs and sometimes doesn't is because it depends on which build host picks up the build. Over a few builds, all hosts build the cache and after that every build is predictably cached. More here - http://blog.shippable.com/container-caching

[codeviking]

Ok, so is there a build image we could use that has both scala and node installed?

[manishas]

not today... minv2 is the only one with multiple languages. I forgot you needed both for the same repo.

We're working on making this easier.

[codeviking]

@manishas Any ETA?

[nathanmarks]

@manishas +1 on an ETA, I also use multiple languages and would like better a configured image so I don't have to worry about installing everything in before_script

[codeviking]

@manishas Also, our builds are now failing due to timeouts regularly. This is definitely having a high impact on my satisfaction with the service. Any immediate work arounds?

[codeviking]

@manishas Our builds for this project now seem to be frozen at the "waiting" stage. I see a lot of similar issues in the past where this was resolved by your team manually resetting the minion. Can you do that for us?

[manishas]

We're looking into this now.

[paulcullin]

+1 on very slow build times but I'm using minv2. Since I'm only using node, I'll try switching to a node-specific minion image and see if that improves performance.

[manishas]

@pcullin-ratio it's always better to use a language specific container.

That said, today's issues are related to docker issues on our build hosts and we're working to resolve them.

[rphutchinson]

@manishas is there a general issue with caching? We have seen our builds (sbt using minv2) go from < 5 minutes typically to consistently 20+ minutes over the last couple of days. It appears that the big difference from when the builds were quicker to now is that it's downloading our project dependencies each time where before those were loaded from a local cache. There haven't been any configuration changes on our end.

[codeviking]

@manishas Any ETA on a fix?

[manishas]

@codeviking can you point me to specific builds that are hanging/timing out or frozen? We made some fixes over the last 2 days and if things are still unstable, we'll look into the specific projects.

[codeviking]

@manishas Over the last 2 days? We were encountering issues just yesterday / last night.

That said our build does seem to be moving, just not as fast as it normally would. I believe that relates to the necessity of reinstalling node with every build.

Any ETA on having a build image with multiple toolkits pre-installed?

[manishas]

@rphutchinson we'll look into your caching issues today. I'll update you when I know more.

[manishas]

@codeviking yes, over the last 2 days. We moved from a monolithic build architecture to microservices, so a pretty major change. This caused some unforeseen issues but we've worked through most of them. More on that in a blog post today.

It will take us a few days to create an image with multiple toolkits. I will need to talk to engineering about a specific ETA. If you know exactly what you need and can provide a Dockerfile, we can get it done pretty quickly. Let me know.

[chriskinsman]

Our build times have gone from minutes on average to 10+ on average now...

[manishas]

@chriskinsman is this for a specific project or across the board? I looked at enabled projects and saw one with a 15 min build time, but the rest seem to be building in 6 mins or lower.... at least the latest builds were.

[manishas]

@rphutchinson caching should work for you now. It might take a few builds to catch up as explained here but let me know if it doesn't work after a few builds.

[codeviking]

@manishas Our builds also increased significantly in time -- they all seem now be at least 30 minutes. They were 2-5 minutes before.

[manishas]

@codeviking can you please point me to specific projects that are slower and we will take a look? I don't see all of your projects take 30 mins, so i want to make sure we look at the right ones.

[codeviking]

The project is that associated with the id I already gave you in this thread.

If you look at the history the build yesterday was (sometimes) taking 2-3 minutes, now it's taking 30 minutes everytime.

[manishas]

@codeviking the latest build took 7 mins. Also, it was taking much longer yesterday, at least according to the dashboard.

If this is the correct project (540e6be63479c5ea8f9e40b3), then the shorter build times were more like 2-3 weeks ago. The last build was 7 mins since it pulled from the cache. Let's see how long the next few builds take.

[chriskinsman]

If you look at ps-jobscripts I have seen some 15-20 minute build times as measured by start and finish time stamps.

I cancelled one today at the 16 minute mark and restarted it.

On Fri, May 1, 2015 at 9:15 AM, Manisha notifications@github.com wrote:

@chriskinsman https://github.com/chriskinsman is this for a specific project or across the board? I looked at enabled projects and saw one with a 15 min build time, but the rest seem to be building in 6 mins or lower.... at least the latest builds were.

— Reply to this email directly or view it on GitHub https://github.com/Shippable/support/issues/1389#issuecomment-98168562.

[manishas]

@chriskinsman thanks for the info. Last build took 6 mins, so I'll track times for the next few builds.

We're watching the system very closely today to make sure all our fixes are working + fixing a few smaller bugs which should make everything robust again.

[codeviking]

@manishas How can we resolve the build issue where nvm use 0.12.1 causes our build to be 2-3x as long.

Is there a version of node your team would suggest we use that is either pre-installed or is quicker to install?

[nathanmarks]

@codeviking @manishas I'm installing node 0.12 itself really quickly on a PHP image.

[codeviking]

Huh, I wonder why it's slower on our box. @manishas Any insight?

[nathanmarks]

@codeviking Note that I'm not using NVM.

curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
sudo apt-get install -y nodejs

[codeviking]

@nathanmarks Maybe I'll try switching to that! Thanks!

[codeviking]

@manishas Any update?

[manishas]

Closing this issue. Please feel free to comment or re-open if you are running into this problem.