DNS not working under sandstorm - getaddrinfo returns EBUSY

[gischer]

I have a meteor app, which among other things, downloads a file from a server on the internet, and does stuff with it.

I have managed to launch a powerbox, get the claim token and exchange it for the access token as described in the docs. However, when I go to fetch the file, I see

Error: getaddrinfo EBUSY <hostname>

Outside of Sandstorm, I do not get this message, the file downloads correctly. I've spent the morning reading the Sandstorm source, and this does not appear to me to be the result of an incorrect Authorization header, but rather some basic failure of the DNS lookup.

The most likely failure seems to be running out of file descriptors. I looked at all the procs that I thought were likely (that is, the ones that started at the time I launched the meteor-spk dev, and they all seemed to have plenty.

[zenhack]

Does this happen every time?

Maybe try adding ulimit -n 8192 or such early to launcher.sh; that should solve the issue if it is indeed an fd exhaustion issue.

[gischer]

It fails every time this way.

I could easily be missing something, but it seems to me that being a Meteor app it doesn't use launcher.sh, it instead calls node on 'start.js'

But this means there is no shell available. For instance I tried to use the node package "child_process" to run "prlimit --nofiles=8196", and under Sandstorm I get:

Error: spawn /bin/sh ENOENT

Which I take to mean that under Sandstorm, we don't have a /bin/sh, is that right? What's a good way to address this?

[ocdtrekkie]

It's definitely plausible to me that an app built with meteor-spk might not have a shell. vagrant-spk apps tend to include it because they generally use .sh scripts to launch the app. As you can see in https://github.com/zenhack/ttrss-sandstorm/blob/master/.sandstorm/sandstorm-files.list bin/sh has been included in the TTRSS package.

I'm not sure the best way to force it's inclusion for an app made with meteor-spk though.

[ocdtrekkie]

You'd almost certainly end up with a shell if you used vagrant-spk's meteor stack (which, in turn, uses meteor-spk), though it's bitrotted and explicitly calls an old version of meteor-spk and I'm not sure the last time someone tested it.

[gischer]

I might need to include extra binaries and a shell in a Meteor app for a different reason, so the meteor vagrant stack could be worth bringing up to snuff. I would think there's some way to get the meteor build to do it, but I'm not seeing how.

It seems as though Sandstorm does a chroot which makes the normal /bin/sh on my system invisible. Where does it chroot to?

However, I really doubt that the app is running out of file descriptors, given that I examined every plausible running process, and none of them had more than a dozen open FDs. So this seems to me like it's maybe not the right direction, except why else would getaddrinfo fail? I'm at a loss beyond finding the source for getaddrinfo and reading it.

[gischer]

Well, I just found the most fascinating comment in the POSIX source of getaddrinfo:

/* Could not locate any of the lookup functions.
--
The NSS lookup code does not consistently set
errno, so we need to supply our own error
code here.  The root cause could either be a
resource allocation failure, or a missing
service function in the DSO (so it should not
be listed in /etc/nsswitch.conf).  Assume the
former, and return EBUSY.  */

This is the one and only place EBUSY is returned from getaddrinfo. I find the possibility of a missing or mismatched DSO quite plausible, since earlier I tried using the "capnp" node module, but it failed under sandstorm because it couldn't find the shared library it wanted.

[gischer]

Ok, here's something else I found. Reading through the vagrant-spk build.sh for meteor, I see it is careful to use the version of node that is packaged with meteor to install the dependencies of the project. However, meteor-spk merely does

(cd .meteor-spk/bundle/programs/server && meteor npm install)

I find it curious that the vagrant-spk script does a lot of work to avoid using meteor npm install since that very command is used earlier in the script for a different purpose. Instead, considerable scripting effort went into finding the exact version of meteor-tool and adding it to $PATH.

Now, granted that in my own illustrious programming career, I've ended up doing things like that by accident, it still looks pretty intentional that meteor npm install was avoided here. Does anybody know more?

[ocdtrekkie]

I don't think we have any particular insight available as to why beyond anything you might find in commit history or issues/PRs.

[gischer]

Fair enough. Just hoping.