Slug compilation very slow

[wesleyfsmith]

I feel like the slug compilation is very slow with this buildpack. It often will take more than 8 minutes to build the app. When I've built node apps in the past they have compiled much more quickly. Does anyone have any thoughts on this?

[dutchakdev]

+

[vintesh]

It is slow but yes it is good compared to other available options, we switched to "horse".

[joegoldbeck]

it's actually faster than some others I've used.

not as fast as pure node though for sure

[yourcelf]

Just a quick note that I've just merged changes in #90 that were necessary to support mobile platforms correctly for Meteor >= 1.3.1, but which have the unfortunate side effect of making slug compilation much slower for Meteor < 1.3. So if you're on 1.2 and finding it's horrendous, it'll be a little less so if you upgrade meteor.

[acasas]

I am experiencing something very similar, after upgrading from 1.2.1 to 1.3.2.4, the slug compilation is timing out on me.

From heroku logs:

2016-05-02T19:06:15.510305+00:00 heroku[slug-compiler]: Slug compilation started
2016-05-02T19:06:15.510310+00:00 heroku[slug-compiler]: Slug compilation timed out: 2922.207902553 seconds.

and this is the output of the deploy to Heroku:

remote: Compressing source files... done.
remote: Building source:
remote: 
remote: -----> Fetching set buildpack https://github.com/AdmitHub/meteor-buildpack-horse.git... done
remote: -----> Node.js app detected
remote: -----> Installing node
remote: -----> Installing meteor
remote: Downloading Meteor distribution
remote: ######################################################################## 100.0%
remote: 
remote: Meteor 1.3.2.4 has been installed in your home directory (~/.meteor).
remote: 
remote: Now you need to do one of the following:
remote: 
remote:   (1) Add "$HOME/.meteor" to your path, or
remote:   (2) Run this command as root:
remote:         cp "/tmp/buildpack20160502-165-16pa4oh/meteor-3hQT/.meteor/packages/meteor-tool/1.3.2_4/mt-os.linux.x86_64/scripts/admin/launch-meteor" /usr/bin/meteor
remote: 
remote: Then to get started, take a look at 'meteor --help' or see the docs at
remote: docs.meteor.com.
remote: -----> Bundling bundle
remote: -----> Checking if this meteor version supports --server-only
remote: -----> Building Meteor with ROOT_URL: https://xxxxx.herokuapp.com
remote: npm-container: updating npm dependencies -- open, softap-setup, spark,
remote: node-mandrill...
remote: Verifying deploy.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
remote: 
remote: !   Push rejected to xxxxx.
remote: 
To https://git.heroku.com/xxxxx.git
 ! [remote rejected] 84b7bc870101c29c855cc8e326c24fffd5008ce7 -> master (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/xxxxx.git'

Any advice?

[yourcelf]

@acasas I think your issue is a separate one; "very slow" in this issue has never meant so slow that the build times out. Please open a new issue, and share the results of building with the config var BUILDPACK_VERBOSE=1 (from the command line: heroku config:set BUILDPACK_VERBOSE=1). That way we can see more detail about where this might be succeeding or failing.

[n0isy]

@acasas Looks like not enough memory. 512 MB? Add swap like here: https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04

[n0isy]

@yourcelf git push dokku - build 10 minutes. Why it downloading meteor everytime for minor changes?

[yourcelf]

@n0isy: this is the heroku way. Each time, it builds as though on a brand new host that has nothing. While it is possible to do some caching between builds, meteor doesn't make this easy because of how it installs itself. If you prefer incremental changes rather than full rebuilds, you should consider a different hosting model than heroku/dokku.

[dpeet]

@yourcelf do you see any improvements on caching in your pipeline? I searched around for similar things and found several references to this https://blog.alexmaccaw.com/faster-deploys. I'm not particularly familiar with the internals of how it work, but could we cache npm/meteor and then only redownload if a version is different?

Thanks for your great work with this buildpack, and your attentiveness to updates!

[yourcelf]

@dpeet I've just added caching of the meteor installation to the beta branch (https://github.com/AdmitHub/meteor-buildpack-horse.git#beta), and will merge it in soon if no one reports issues with it.

[dpeet]

Just gave it a go (Meteor 1.3.5.1). Size went from 250mb to 25 AND it was faster! Awesome work @yourcelf!

[yourcelf]

130 is now merged with significant improvements to build time. We now download only the target release (previously, we were downloading the latest release and then upgrading/downgrading to the app's target release), this halves the time for acquiring meteor if you're not running anything other than the latest stable release for the app. Additionally, we're now caching the installation of meteor between builds, so unless your app's target release changes, we don't have to wait to re-install meteor (which was the most time consuming step).

The 3 time-consuming-est parts that remain are: installing npm dependencies in the source environment, running meteor build, and installing npm dependencies on the built target. We're unlikely to be able to get significant speedups to those without upstream improvements from meteor and npm. So I think for now, I'm going to close this out and call it "as fast as we can be" right now.

I'd welcome any suggestions or pointers to way to improve build speed in the future -- please open specific issues for the thing that could be sped up.

[hems]

Has anybody tried this patch which uses yarn instead of npm ? Shouldn't that me merged on master as well?