Investigate reducing the dependencies to install vips

[ide]

One of the hardest parts about using Sharp is setting it up. Being able to just say "npm install sharp" would make it easier for a lot of people to try it out.

Currently libvips isn't very easy to set up on an Amazon Linux box, and on Mac OS X it requires brew (there are some instances in which one may wish not to install brew). So if there were an easier way to build vips and it were integrated with Sharp's installation that could help a lot.

[lovell]

Instructions in the README for yum-based installation (to include AWS's default AMIs) would definitely help, thanks for the suggestion.

How about a sample Dockerfile? That would help with an Elastic Beanstalk deployment.

Not sure I'd want the maintenance overhead of including and compiling all dependencies from source!

[ide]

Instructions for yum (EC2 in particular) and a dockerfile would certainly be useful for production setups.

I was chatting with a friend earlier today who didn't have brew installed and he found Sharp difficult to set up on Mavericks. I don't think it's reasonable for the Sharp project to precompile vips for major platforms but it would be nice for compilation to be easier.

For example node-fibers is pretty easy to install on all major platforms and it ships with the source of its dependencies. So something similar where Sharp ships with the source of vips could be helpful, even if it's a longer term goal that might require help from the vips team.

On Tue, May 27, 2014 at 11:31 PM, Lovell Fuller notifications@github.com wrote:

Instructions in the README for yum-based installation (to include AWS's default AMIs) would definitely help, thanks for the suggestion. How about a sample Dockerfile? That would help with an Elastic Beanstalk deployment.

Not sure I'd want the maintenance overhead of including and compiling all dependencies from source!

Reply to this email directly or view it on GitHub: https://github.com/lovell/sharp/issues/42#issuecomment-44370251

[lovell]

The libcoro dependency of node-fibers appears a lot simpler than libvips (which then has its own dependencies to worry about too).

What I really like about the node-fibers repo is the provision of build.js, an idea sharp could borrow as a "bootstrap" to verify that dependencies are present, providing more useful feedback/help in the event things are not quite ready. Thanks for the inspiration!

I'm also happy to provide installation instructions for other Mac OS package managers, e.g. Fink, if there's enough call for it.

[jonathanong]

i would love if vips was included so i can deploy it to places like heroku without crazy images and stuff. however, it makes total sense not to as it's super complicated. it could also take forever to build

[lovell]

@jonathanong There's a Heroku Buildpack at https://github.com/alex88/heroku-buildpack-vips which I've not tried but seems only slightly out of date with respect to required version numbers. Perhaps send a PR to upgrade to libvips 7.38.5 and liborc 0.4.19.

[lovell]

I've added a preinstall.sh shell script on the preinstall branch that can be used to install libvips and its dependencies on a wide variety of OS and package manager combos (but not Windows).

If everyone/anyone following this issue is able to test the instructions at https://github.com/lovell/sharp/tree/preinstall#prerequisites - especially on Mac OS as I've not been able to verify this - I would very much appreciate it.

[lovell]

When everyone is happy that preinstall.sh is solid enough, I'd like to either add it as the preinstall script in package.json (completely ignoring the "INSTALL SCRIPTS ARE AN ANTIPATTERN" warning) or somehow get gyp to call it.

[ide]

Here's my output on Yosemite. Seems to be working.

$ ./preinstall.sh 
Found libvips 7.40.11
$ brew remove vips
Uninstalling /usr/local/Cellar/vips/7.40.11...
$ ./preinstall.sh 
Could not find libvips using a PKG_CONFIG_PATH of '/usr/lib/pkgconfig:/usr/local/Library/ENV/pkgconfig/10.10::/usr/local/lib/pkgconfig:/usr/lib/pkgconfig'
Detected Mac OS
Installing libvips via homebrew
==> Downloading http://www.vips.ecs.soton.ac.uk/supported/7.40/vips-7.40.11.tar.g
Already downloaded: /Library/Caches/Homebrew/vips-7.40.11.tar.gz
==> ./configure --prefix=/usr/local/Cellar/vips/7.40.11 --with-magick --with-magi
==> make check
==> make install
==> Caveats
If you need Python to find the installed site-packages:
  mkdir -p ~/Library/Python/2.7/lib/python/site-packages
  echo '/usr/local/lib/python2.7/site-packages' > ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth
==> Summary
🍺  /usr/local/Cellar/vips/7.40.11: 196 files, 13M, built in 65 seconds

[lovell]

Great news, thanks @ide.

"I was chatting with a friend earlier today who didn't have brew installed and he found Sharp difficult to set up on Mavericks."

Is your friend available to help? What, if any, package/dependency management tool(s) do they use?

[toulouse]

That's me.

I use vanilla ./configure with $PREFIX set to a subdirectory of my home directory, and make install things. I'm not a UNIX neckbeard, but it's a very reasonable expectation to expect to be able to make local, non-global installations of compiled software.

Only being able to have a single active version of a dependency on an entire system is literally a singleton pattern and it makes a number of compromises to the system's integrity I'm not really willing to make, not the least of which is making a subdirectory of /usr/local user-owned-and-writable.

Not to mention, new OS upgrades often break - sometimes subtly, sometimes blatantly - changes made by the package managers, and I prefer not to be dependent on upstream fixes to get a system up and running to where I'd like it to be.

[papandreou]

The preinstall script works fine on Ubuntu 14.04 here. This patch makes it work with Debian Wheezy as well: #111

[lovell]

@toulouse Thanks for your feedback - you're definitely not a "neckbeard"! What you say is pretty much why the separation of Linux Containers via tools such as Docker is attracting so much attention at the moment.

Do you rely on pkg-config for dependency management? If so, are you manually modifying/setting the PKG_CONFIG_PATH environment variable? If not, how to you achieve this?

My current thinking is that the preinstall shell script will install libvips and dependencies globally only if run with root/sudo access, otherwise it will attempt to compile at least libvips from source into a relevant directory under node_modules and add the path to PKG_CONFIG_PATH.

@papandreou Thanks Andreas, I've merged your patch.

[papandreou]

My current thinking is that the preinstall shell script will install libvips and dependencies globally only if run with root/sudo access, otherwise it will attempt to compile at least libvips from source into a relevant directory under node_modules and add the path to PKG_CONFIG_PATH.

Very much in favor of that approach. It needs to work when not sudoing npm install, and I also prefer a local install.

Would it be possible to bundle the libvips source code à la https://github.com/bnoordhuis/node-iconv?

[lovell]

The preinstall.sh script, which currently requires root/sudo access, is now in master. The Travis and Snap CI jobs both use this script for their test environments. I'll only make this part of npm install when it can be run as a non-root user.

@papandreou The upstream libvips library changes often so I'd rather have an install script fetch a specific version (one line change) than include it in the repo (1000+ file change).

There's also the small matter of libvips' dependencies (libjpeg, libpng, libwebp etc.) to consider. How much can we assume is already present on a system?

[toulouse]

IMO, pkg-config is definitely an acceptable mechanism for locating libraries - it's well-known and mature tool that has worked well in my experience.

As for including libvips in the repo, I think it would make a lot of sense to include it as a git subtree (for making tracking and merging changes pretty straightforward), and track the libvips branch versions.

One major priority for running code in production is to have well-known combinations between libraries, so if there's a bug that only manifests in version X of libvips when combined with version Y of sharp, it's a lot easier to test for / verify / prevent.

On the other hand, your point about libjpeg, libpng, libwebp, etc. is quite valid.

[papandreou]

@lovell Sure, it's a balance, and I understand why you would like to keep the size of the package itself down. However, on top of @toulouse's points about predictiblity it's also a big plus not to depend on downloading stuff from the Internet at install time.

The source code of the mentioned libraries adds up to about 16MB, which is about the same size as libiconv, and node.js itself bundles about 80MB of dependencies. It's a lot, but it could be done.

[marcbachmann]

FYI: I just created a Dockerfile which installs libvips from source. It's the same installation procedure like in the install script. I created the dockerfile because I failed installing libvips using the provided script (some pkg-config problem).

Here is the file https://registry.hub.docker.com/u/marcbachmann/dockerfile-libvips/dockerfile. It's based on ubuntu & the image has about 726MB, which is really huge. I have no problem with that size but I think it can get smaller.

I'm sorry for ignoring the comments in here. It's late :sleepy:

[lovell]

Thanks for creating a Dockerfile @marcbachmann, I'll add a link to the prerequisite docs.

"I failed installing libvips using the provided script (some pkg-config problem)."

Feel free to open a new issue if you'd like more help with this specific problem.

[binarykitchen]

Many packages place OS dependant binaries in the local .bin folder inside node_modules. Would that be an option?

[lovell]

186 provides the a rather compelling solution to this.

[lovell]

You'll be pleased to hear that the next priority for sharp is improving the situation with dependencies.

My current thinking is to try to build and publish a static library comprising libvips and its dependencies required by sharp. This should target at least 64-bit Linux and Mac OS. Windows can probably use the pre-built static libraries without too much effort.

These can then be retrieved via HTTPS, decompressed and placed locally at npm install time via the use of a custom JS or Python (as node-gyp requires Python) script.

We should avoid static-linking licensing problems as libvips is LGPL2.1+, which includes LGPL3, which is compatible with Apache 2.0.

[papandreou]

Very pleased to hear that! Currently I'm very challenged because our production platform doesn't have a sufficiently recent libvips, so this could be a life saver.

Please keep us posted about how it's going :)

[lovell]

I'm currently building a Dockerfile to help isolate the compilation of libvips and its dependencies, the output of which should hopefully be a compressed bundle of static libraries.

Initially this will target 64-bit Linux and I'm aiming for compatibility with at least Centos 6 and Ubuntu 12.04, which probably covers >80% of the Linux user base.

The complete (and I mean complete, it even includes zlib) set of static library dependencies will involve a ~15MB download during npm install.

[lovell]

Update: the static library approach was far too complex. I've now got a pre-compiled libvips and all of its dependencies as shared libraries installed locally (without root) on Ubuntu 12.04, which sharp is able to compile and link against. The compressed tarball is down to ~12MB.

Next step: test across a number of Linux flavours. It's probably going to be kernel v3.2+ only, which means skipping Centos 6. This also means we can drop the somewhat esoteric Snap CI and instead move all of the Linux-based CI into scripted Docker containers.

[lovell]

Sudo-less libvips installed within node_modules is now available for recent (libc v2.15+) 64-bit Linux and Node.js v0.12+ on the "look" work-in-progress branch for the next release.

npm install lovell/sharp#look

https://travis-ci.org/lovell/sharp/jobs/85895589

Next step: local install for win32.

[acuervof]

hello @lovell

great work with sharp!

My idea is to use your module on AWS lambda.

I use "look" branch and works on node v0.12.x locally. The problem I have is to run my function on AWS lambda with node v0.10.36 it has installed.

Any idea to got it?

[lovell]

@acuervof The new binding-time script makes extensive use of execSync introduced in v0.11.x.

I wasn't aware of Lambda's current v0.10 limitation and it looks like there's ongoing discussion about Node versions. AWS's @listonb has been very helpful in the past and might be able to provide more details about v0.12+ support.

[lovell]

I've managed to shift most uses of execSync back into shell commands within the binding.gyp configuration instead so this should (eventually) work with Node v0.10. Nothing ready to test here yet, sorry.

Next step: change the pre-compiled binaries to use .tar.gz as that seems to be more generic/portable than providing platform-specific .tar.xz and .zip files.

[acuervof]

Great solution while waiting for node version upgrade on Lambda machines

I think sharp could be extensively used on AWS Lambda as alternative of slower imagemagick provided by default.

[lovell]

The look branch is now using tar.gz files and appears to be working with Node v0.10 - see https://travis-ci.org/lovell/sharp/builds/87784074

@acuervof if you're able to test this with Lambda that would be most appreciated.

[acuervof]

@lovell

Testing in local I'm getting in require('sharp') line:

Error: libvips.so.42: cannot open shared object file: No such file or directory

I guess is not necessary install libvips with this approach, isn't it?

[lovell]

@acuervof I've updated the (getting crazier) runtime link options - please can you try again.

If this doesn't work, please can you provide the results of running ldd node_modules/sharp/build/Release/sharp.node (perhaps as a gist).

[lovell]

The look branch also now downloads and uses a local node_modules stored copy of libvips for Windows, which means no more manual downloading, unpacking and setting of environment variables for Windows users.

The Windows CI environment now uses this - see https://ci.appveyor.com/project/lovell/sharp/build/220/job/h2b76b26dbyuba7x

Next step: investigate how we could use clang on Travis CI's Mac OS environment to build shared .dylib libaries for libvips and its dependencies.

[acuervof]

Congratulations @lovell, you got it!

sharp working on AWS Lambda getting better times about imagemagick

Thanks!

[lovell]

@acuervof :tada:

[vlapo]

great job :)

[mahnunchik]

I set dependency to lovell/sharp#look in development branch of gulp-responsive.

[emmanuelmillionaer]

@acuervof How were you able to run it on aws lambda?

After I installed sharp with npm install lovell/sharp#look

my package is simply to big to run on aws lambda, the limit is at 50MB http://docs.aws.amazon.com/de_de/lambda/latest/dg/limits.html

Regarding this topic, I also wrote an question: https://github.com/lovell/sharp/issues/292

Thanks for your help!

[vlapo]

@lovell Why my self share compiled libvips+sharp ZIP on AWS has 6MB and look branche is so big ?

[lovell]

@vlapo See https://github.com/lovell/sharp/issues/292#issuecomment-152721575

[vlapo]

I know about comment. I check it again.

$ du -h --max-depth=1 | sort -hr
22M     ./node_modules
18M     ./vips

This is my libvips compiled in vips and node_modules with sharp ~40MB

And here is look branche without tests

57M     ./node_modules

hmm maybe, I have lite compilation or skip some libs ~17MB. :D Okey, 17MB are a little bit real.

[lovell]

@vlapo I'd expect shared libs to be a bit larger than static libs. Did you include libMagickCore?

[mahnunchik]

What about binaries for Mac OS X?

npm install lovell/sharp#look fails even if vips is installed by brew.

> sharp@0.12.0 install /Users/User/node_modules/sharp
> node-gyp rebuild

ERROR: Please install libvips on Mac OS by running the following command:
 brew install homebrew/science/vips --with-webp --with-graphicsmagick
gyp: Call to 'LDD_VERSION="" node -e "require('./binding').download_vips()"' returned exit status 1. while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/Users/User/.nvm/versions/node/v4.2.2/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:355:16)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Darwin 15.0.0
gyp ERR! command "/Users/User/.nvm/versions/node/v4.2.2/bin/node" "/Users/User/.nvm/versions/node/v4.2.2/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/User/node_modules/sharp
gyp ERR! node -v v4.2.2
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok 
npm ERR! Darwin 15.0.0
npm ERR! argv "/Users/User/.nvm/versions/node/v4.2.2/bin/node" "/Users/User/.nvm/versions/node/v4.2.2/bin/npm" "install" "lovell/sharp#look" "--save"
npm ERR! node v4.2.2
npm ERR! npm  v2.14.7
npm ERR! code ELIFECYCLE

[lovell]

@mahnunchik The look branch requires libvips v8.1.1+ as this is the minimum (recent) version that will work correctly with Windows. Currently only v8.1.0 is available via brew so if you're able to submit a PR to the homebrew-science repo to bump the version to v8.1.1 that would be great.

I am investigating the provision of pre-compiled .dylib files - see https://github.com/lovell/package-libvips-darwin-x64/blob/master/.travis.yml - but this may have to wait until a future release.

Mac users are generally well-served by homebrew. The vast majority of installation-related questions raised here are Windows and Linux related.

[mahnunchik]

I'm going to find out how to update vips in the homebrew-science repo https://github.com/Homebrew/homebrew-science/issues/2977

Thank you for Linux version. I have tested it locally and inside the docker. It works like a charm!

[lovell]

@mahnunchik :+1:

[mahnunchik]

@lovell vips has been updated in the brew repo to 8.1.1 but sharp#look could not be installed. The error is the same as above.

[lovell]

@mahnunchik Thanks for the brew update - I've been able to reproduce this now too.

I know what the problem is so will fix and improve the OS X error messaging (e.g. "you need to install ... " vs "you need to upgrade ... ") over the weekend.

[lovell]

The look branch has been updated and is ready for further testing.

The Windows and Linux CI environments are both healthy.

Windows support will be 64-bit only from v0.12.0.

@vlapo The shared libs for Linux are now a mere 6MB when compressed, about 19MB when uncompressed. I'd forgotten to strip them of debug symbols.

@mahnunchik This should now work with brew-installed vips again on OS X.

[lovell]

Any more comments, suggestions or test results before the look branch merges into master?