Issues with Meteor 1.3

[ndarilek]

Perhaps this is already present and I don't see it. There are lots of changes coming down the pipe in Meteor 1.3, and (I don't think) it is yet clear how to meteor build. However, one thing is certain, NPM integration works out of the box.

To that end, it'd be nice if meteor-spk shipped a recent version of NPM. We still have to hack our build scripts to insert npm installs and copy package.json around, but having NPM installed automatically would make figuring out the correct incantations easier. Having a release that shipped NPM would be very helpful.

Thanks!

[kentonv]

Hi Nolan,

Sorry, I'm a bit confused. Meteor's packaging system has had built-in support for declaring dependencies on NPM packages for a long time. If you do that, meteor-spk should take care of making sure those packages make their way into the SPK. Specifically, meteor-spk pack performs a meteor build and then an npm install as specified in the Meteor docs. It uses Meteor's bundled npm to do this.

I haven't kept up with the changes in Meteor 1.3, though. Is it breaking this somehow?

[ndarilek]

You can now put a package.json in your Meteor 1.3 apps and are expected to do an npm install as a step before meteor build. This is separate from, and will likely replace in many cases, putting NPM package dependencies in Meteor packages simply to use NPM packages in Meteor.

So now where we have:

cd /opt/app
meteor build --directory /home/vagrant/

in build.sh, you'd now do:

cd /opt/app
if [ -e package.json ]; then
   npm install
fi
meteor build --directory /home/vagrant/

Anyhow, I've dug a bit deeper, and it looks like the vagrant-spk build scripts access npm from within the .meteor directory. So it's included, just getting at it isn't obvious, and I need to in order to invoke npm at the right points. Maybe this issue should be closed, or repurposed as a general "what's needed to get Meteor 1.3 working in Sandstorm" discussion. I'll be hacking on that, but finding npm was a bit much for my coffee-deprived brain this morning when I filed this. :) I was looking at the dependency-packing logic in meteor-spk and didn't see anywhere referencing the npm binary as was true with node and others.

[kentonv]

Well, npm is a build-time dependency, not a runtime dependency, so it shouldn't be included in the spk like the node binary is.

FWIW this line does npm install: https://github.com/sandstorm-io/meteor-spk/blob/master/meteor-spk#L72

From what you are saying it sounds like meteor-spk may need some tweaking for Meteor 1.3. Although, I'm a little confused by the suggestion that npm install would execute before meteor build. Normally (when not targeting Sandstorm) meteor build is expected to run on the developer's machine, but npm install really needs to run on the target machine in order to install the correct architecture for native-code components. If npm install now happens first, it seems like you'd get the wrong arch, or the build process now needs to run entirely on the target. I'd like to see what the docs have to say about this.

@zarvox I forget, does vagrant-spk invoke meteor-spk for the build step, or will it need to be updated separately?

[ndarilek]

So I added code similar to the above to my build.sh. I'm able to build the app, but am told that Meteor requires node 0.10.41 or greater to run.

The version of node shipped with Meteor 1.1.3 is 0.10.36. I'm wondering if the launcher script is using that, despite my Meteor app being set to 1.3 beta? I definitely have the correct Meteor build tools in place.

Where is METEOR_WAREHOUSE_DIR set initially? I see it modified in build.sh. It isn't immediately obvious to me where the scripts are pulling the version of Meteor they run binaries against.

Thanks.

[ndarilek]

See this issue for more about the npm install && meteor build workflow under 1.3.

[zarvox]

@kentonv It will need to be updated separately, but it should be reasonably straightforward to do so: https://github.com/sandstorm-io/vagrant-spk/blob/master/stacks/meteor/build.sh

@ndarilek usually METEOR_WAREHOUSE_DIR is unset; the script as written lets a value from the environment override the default value of $HOME/.meteor.

However, we are using the node from the meteor-spk bundle, which copies in a node (and a mongo) at meteor-spk's build time. What we should probably do instead is do something like what Sandstorm itself does with find-meteor-dev-bundle.sh and pull in the node from the Meteor dev_bundle that matches the current project's .meteor/release, and use that copy of node. Of course, since the default sandstorm-pkgdef.capnp hides /home, we'll probably need to copy it somewhere else, or adjust the pkgdef...

This does seem like a problem for meteor-spk in general though - the version of node will be whatever meteor-spk ships, not whatever your project specifies.

[kentonv]

I see, it looks like npm install is needed to install build tools invoked by meteor build, not runtime packages. Fair enough.

As for the node version error, yeah, we generally need to rev meteor-spk every time there is a new Meteor release. Happy to do that once 1.3 ships. In the meantime you may be able to get away with swapping out the binary manually.

[ndarilek]

No, NPM does ship runtime dependencies alongside MDG's own package system. Meteor is ditching its own React package for the official version from NPM, for instance. Also, lots of newer Meteor packages are shipping as NPM packages. See https://github.com/mantrajs/mantra-core for instance. It's meant only for use with Meteor and is shipped via NPM.

I'm a bit confused about the confusion here. :) My app's NPM-based runtime dependencies aren't any different from its Meteor-based ones, right? They're being installed in a Vagrant VM just like any other app dependency, so are shipping with binaries compiled therein.

Might be worth retitling this issue to one about Meteor 1.3, which I'll do tomorrow if no one beats me to it. Initially I went looking for npm in /opt/meteor-spk and didn't find it, didn't think to poke around in the .meteor directory as I'm used to using toolchains I install myself. In the interim I'll go ahead and replace the older Meteor tools with the newer and see what that gets me. Thanks for the tip.

[ndarilek]

I've done the following in my build.sh:

cd /opt/app
cp 
/home/vagrant/.meteor/packages/meteor-tool/1.1.13-beta.12/mt-os.linux.x86_64/dev_bundle/bin/* 
"$METEOR_DEV_BUNDLE"/bin
"$METEOR_DEV_BUNDLE/bin/npm" install

I've confirmed that node -v in $METEOR_DEV_BUNDLE/bin is 0.10.41, but I still get the error.

Where is meteor-spk getting its node binary? I don't see a path or any other environment variable set in launcher.sh, just exec node. If it's getting it from $METEOR_DEV_BUNDLE/bin then I don't know what I'm missing. Looks like Meteor may be hiding a few additional node binaries around but I haven't investigated each find . -name node result.

Thanks.

[kentonv]

meteor-spk gets its node binary from the download bundle linked in the readme. It will need to be rebuilt against 1.3.

[ndarilek]

So admittedly this won't work until 1.3 released Monday, but the new meteor node command can be used to run whatever version is intended for the app's Meteor release--0.10.43 for now, but 4.X/5.X very soon. However, unfortunately meteor node first does some nonsense or other attempting to open its package database, which I assume it does RW so it can upgrade itself if necessary, which of course fails on a read-only filesystem. I'm not entirely sure of that analysis but it seems likely, and Meteor annoys me enough sometimes that I don't care to understand its insanity further.

However you can use meteor node to introspect itself and symlink the expected node/npm binaries from the development bundle into /usr/local/bin. This in build.sh:

export BASEPATH=$(meteor node -e 'c=process.execPath.split("/"); console.log(c.slice(0, c.length-1).join("/"))')
sudo ln -sf $BASEPATH/node /usr/local/bin
sudo ln -sf $BASEPATH/npm /usr/local/bin

I have an app using the diy stack, community Mongo packages and a vanilla Meteor install, no more meteor-spk. If there's interest, I can work it into a meteor-new stack. You'll still need meteor-spk for older releases (unless you'd rather switch to official node packages.) Or maybe I can make the shell script not bomb out if meteor node doesn't work, meaning /usr/local/bin/node and /usr/local/bin/npm won't get created. If they are created they'd then override the 0.10.X version with whatever Meteor expects, since official versions would be in /usr/bin. It does take a bit longer to provision, but it's a lot less magical and (I think) easier for anyone looking to do anything vaguely more complicated than a vanilla Meteor app. Also, since meteor-spk's environment is very stripped-down, poking around in grains via nsenter is painful. I had to do echo $* instead of ls for instance, and my diy solution doesn't have that issue.

[kentonv]

Note that the meteor command itself is not included in the spk -- it's only available at build time.

But yeah, this could be a more-robust way to find the path of the node binary.

[ndarilek]

In my case it is included in the SPK because I've done away with meteor-spk and use the diy stack. It doesn't run at all however, failing silently. I've attempted to run it via the nsenter method but no luck.

In any case I'm having pretty good results with this setup except, perhaps, for larger SPKs. My ebook reader is at 62M but it does have to pull in lots of Calibre and various Python standard libraries.

[kentonv]

Ah, yeah, the meteor tool is not going to like running inside a sandbox. It wants to check for updates, send tracking stats back to meteor, etc.

[kentonv]

I just pushed meteor-spk 0.1.9 to support Meteor 1.3.1.

I concluded that it is not meteor-spk's job to run meteor npm install in your source directory, since the user can do this themselves and indeed they probably will have done so. However, under vagrant-spk this may be more complicated, as it may be necessary to run meteor npm install inside the VM in order to install tools which will be needed in order to run meteor build -- and if there is any C code involved, the modules installed outside the VM won't work inside of it. I guess it's up to @zarvox and @paulproteus to figure out how vagrant-spk should deal with this...

[paulproteus]

I guess my take on this is that vagrant-spk dev ought to do the meteor npm install if needed, depending on how slow it is. I'll comment further on https://github.com/sandstorm-io/vagrant-spk/issues/152 . Thanks @kentonv !

[ndarilek]

Out of curiosity, why is Meteor a special snowflake? Why does it need its own specific SDK, prepackaged Node binaries, exclusion of the /usr/bin/meteor binary, etc. when just about every other SDK/stack (or every other, AFAIK) bootstraps itself out of the distribution package repositories?

I know it came as a surprise to me when I wanted to build a Meteor app that interfaces with non-Meteor binaries/libraries that there was this whole special sandbox for Meteor, and I couldn't just count on being able to install what I want and have spk pick that up.

Sorry if this sounds snappish, it isn't intended to be. :) It just seems like you're making a whole lot more work for yourselves, particularly when Meteor starts bumping its Node version soon and you're not able to keep up, or when MDG starts promoting Apollo and suddenly I can't use my Rethink/Postgres/SQLite Meteor app because the SDK doesn't support me. I'd kind of hoped we could iron out any issues with the setup here and make that or something like it the new Meteor stack. Or even if we don't use that work, just use something that doesn't lock me in to the old and quickly-going-away pre-1.3 way of doing things. MDG is already working on an SQLite-based Apollo app, and given all that work is being packaged as NPM packages, I wouldn't be surprised if maybe much of it is usable in 1.3.1.

Thanks.

[paulproteus]

Wow! I didn't realize sqlite in Meteor is coming soon! That's shocking and thrilling.

Nolan, to be a little pedantic, but hopefully in a useful way: Sandstorm (the web app) itself doesn't special-case Meteor. But vagrant-spk sure does special-case Meteor, and the way it does has absolutely stymied people from developing complex apps that use (e.g.) Meteor and a MySQL database, which there is absolutely an app that I'm thinking of (from personal email exchanges with others). So I share your concern that the meteor-spk approach for vagrant-spk is difficult to handle, and inconsistent with our documentation on vagrant-spk generally.

Given that, I think the best place for this particular conversation is an issue on the vagrant-spk git repo, perhaps entitled "retire meteor-spk within vagrant-spk" so we can discuss further there. Would you be willing to file it? If not, I can give it a shot; let me know.

[ndarilek]

I can get to it, but probably not for a day or two as I'm super busy. If you do it by then feel free to mention me, otherwise I'll definitely file it as I don't want to lose flexibility with Meteor. :)