sandstorm vs. sandcats...

[noci2012]

I have a sandstorm instance running, but do have certificate issues, the setup nececitates a wildcard certificate.. (nginx webserver on web facing system acting as a reverse proxy, sandstorm running on a backed server). Now i'd like a sandcats setup, but the certificates ends up on the backend. And stripping the certificate on the nginx doesn;t go well as well. The 80 & 443 port are already taken by other webservices (all handled through nginx)...

I did try to setup a haproxy service on port 7443, connecting to the backend sandcat server configured also on port 7443, but this fails. (req_sni is used to branch off). One error is: mongo is also moved to 7081 due to 6081 already being used by the sandstorm local service. Using: export MONGO_URL=mongodb://127.0.0.1:7081/meteor export MONGO_OPLOG_URL=mongodb://127.0.0.1:7081/local

did start it, but shows errors again. and then i get a certificate directory with: var/sandcats/https/client-does-not-support-sni.sandstorm-requires-sni.invalid/ and cannot connect...

Obviously trying this didn't work out, so what went wrong... or is there a basic assumption that sandcat servers should be internet facing themselves, on port 443/80... which will not work here. (Only one IP address is available).

Q what settings need to be done to get a sandcat based server behind a proxy / alternate port, or reverse proxy, or working haproxy setup.

[paulproteus]

Hi @noci2012 ,

Thanks for filing this. We haven't documented anywhere the steps for sandcats+https behind a proxy, and I'd love to see that fixed.

I think req_sni is the right approach. Can you say more clearly what you're trying there (with haproxy config & /opt/sandstorm/sandstorm.conf) and we'll see if we can make that work?

[paulproteus]

Also note I might be a bit high latency over the next few days, but I hope to respond as quickly as I can.

[noci2012]

/opt/sandcats/sandstorm.conf ---8<--- SERVER_USER=sandstorm PORT=7080 MONGO_PORT=7081 BIND_IP=192.168.6.5 BASE_URL=https://xxxxxxxxx.sandcats.io:7443 WILDCARD_HOST=*.xxxxxxxxx.sandcats.io UPDATE_CHANNEL=dev ALLOW_DEV_ACCOUNTS=false SANDCATS_BASE_DOMAIN=sandcats.io HTTPS_PORT=7443 ---8<---

haproxy on 192.168.6.1 ---8<---

global
    log  /dev/log  daemon
    maxconn  4096
    uid  0
    gid  0
    daemon

defaults
    log   global
    mode  http
    option  httplog
    option  dontlognull
    retries  3
    #option http-use-proxy-header
    option  redispatch
    #option  http-server-close
    maxconn  2000
    timeout  connect 61s
    timeout  client 50s
    timeout  server 50s

frontend fe_public_ssl
    bind *:7443
    log global
    mode tcp
    option tcplog
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    acl is_sandcats req_ssl_sni -m end -i 'xxxxxxxxx.sandcats.io'
    use_backend be_sandcats_ssl if is_sandcats
    default_backend be_sandcats_ssl

backend be_sandcats_ssl
    mode tcp
    server sandcats_ssl 192.168.6.5:7443

---8<---

The redirect from port 80 --> 7443 is done in the nginx webserver.

[paulproteus]

First thing I notice - you seem to have two 192.168.6.x IP addresses on the same host. I have nearly never gotten this to work correctly.

Are those different interfaces?

Anyway, leaving that aside, you could run:

curl --resolve xxxx.sandcats.io:443:192.168.6.1 https://xxxx.sandcats.io/

and see what that prints, in case it's useful. That will tell curl to use your haproxy IP address and let you test.

[paulproteus]

I hit "send" a smidge too soon, but anyway, you can set curl's IP address it resolves things to that way. I'm curious what output you get.

[noci2012]

The Sandcats instance for sandstorm.io runs on 192.168.6.5... (on a system named core) The haproxy / nginx system has a public facing address and 192.168.6.1 (on a system named firewall) so those are two different systems. The core system runs a sandstorm instance behind nginx proxy that strips a wildcard certificate, it just isn't a trusted certificate. (so 6080, 6081 for mongo are both used). The sandcats.io instance started on port 7443, & 7081 for mongo.

Curl delivers: (51) SSL: no alternative certificate subject name matches target host name 'xxxx.sandcats.io' The certificate (curl -v shows one that isn't even on the 192.168.6.5 system).... so... The curl check needs to be made against 7443...

curl --resolve xxxx.sandcats.io:7443:192.168.6.1 https://xxxx.sandcats.io:7443/ This connections fails as the sandcats instance mentions this in the logfile:

Error: $MONGO_OPLOGURL must be set to the 'local' database of a Mongo replica set at Error () at [object Object]..extend._startTailing (packages/mongo/oplog_tailing.js:221:1) at new OplogHandle (packages/mongo/oplog_tailing.js:81:1) at new MongoConnection (packages/mongo/mongo_driver.js:216:1) at new MongoInternals.RemoteCollectionDriver (packages/mongo/remote_collection_driver.js:4:1) at Object. (packages/mongo/remote_collection_driver.js:38:1) at Object.defaultRemoteCollectionDriver (packages/underscore/underscore.js:750:1) at new Mongo.Collection (packages/mongo/collection.js:98:1) at packages/accounts-base/accounts_common.js:141:1 at /programs/server/packages/accounts-base.js:243:4, * Front-end died immediately after starting. * Sleeping for a bit before trying again...

/programs/server/node_modules/fibers/future.js:245 throw(ex);

AFAICT the MONGO(_OPLOG|)_URL in the sandcats is equivalent to the one sandstorm one.

Another option (although on the bottom of the list) would be to get a mechanism to put the certificates to the haproxy system (automated due to the 7 day validity) and let the proxy strip them... Although portforward might work, i rather have a proxy in between.

[noci2012]

The sandstorm instance runs from /opt/sandstorm, the sandcats instance from /opt/sandcats i hope there are no assumption in the code about /opt/sandstorm being the top level.

[noci2012]

The working (private sandstorm instance)....

14546 ? Ss 0:00 service sandstorm start 14547 ? S 0:00 _ service sandstorm start 14548 ? S 0:00 _ service sandstorm start 14549 ? S 0:05 _ service sandstorm start 14827 ? S 0:00 | _ supervisor -ePATH=/usr/local/bin:/usr/bin:/bin -eSANDSTORM=1 810ee8c6683b1b0705ed06f1191bed84 2kNBapYWeXxboFApkQwHB6 -- /sandstorm-http-br 14828 ? S 0:00 | | _ supervisor -ePATH=/usr/local/bin:/usr/bin:/bin -eSANDSTORM=1 810ee8c6683b1b0705ed06f1191bed84 2kNBapYWeXxboFApkQwHB6 -- /sandstorm-htt 15986 ? S 0:04 | _ supervisor -ePATH=/usr/local/bin:/usr/bin:/bin 6108b868beb961450e460f9b693251da uzxEFHydzzRSfRBDQMfzNh -- /sandstorm-http-bridge 33411 -- 15987 ? S 0:02 | _ /sandstorm-http-bridge 33411 -- ./rungrain.sh 15988 ? S 0:00 | \ /bin/bash ./rungrain.sh 15989 ? S 0:00 | \ make ETHERCALCARGS=--vm 15997 ? Sl 0:05 | _ node app.js --vm --cors 14552 ? Sl 3:29 \ /bin/mongod --fork --bindip 127.0.0.1 --port 6081 --dbpath /var/mongo --logpath /var/log/mongo.log --pidfilepath /var/pid/mongo.pid --auth -- 14606 ? S 0:00 _ service sandstorm start 14607 ? Sl 3:44 \ /bin/node sandstorm-main.js

The non working sandcats version

17769 ? Ss 0:00 service sandcats start 17770 ? S 0:00 _ service sandcats start 17771 ? S 0:00 _ service sandcats start 17772 ? S 0:00 _ service sandcats start 17775 ? Sl 1:33 _ /bin/mongod --fork --bindip 127.0.0.1 --port 7081 --dbpath /var/mongo --logpath /var/log/mongo.log --pidfilepath /var/pid/mongo.pid --auth -- 17826 ? S 0:00 \ service sandcats start

(I did edit the /etc/init.d/ script to modify the name to show the different sets). Here environments of the node.js processes

Environment of the sandstorm instance:

cat /proc/$(ps axlf | grep 14548 | awk '/sandstorm-main/ { print $3 }')/environ

LANG=C.UTF-8 PATH=/usr/bin:/bin LD_LIBRARY_PATH=/usr/local/lib:/usr/lib:/lib MONGO_OPLOG_URL=mongodb://sandstorm:hOoHaN00omjW5B9vbd5l8mRsZ92@127.0.0.1:6081/local?authSource=admin PORT=6080 SANDSTORM_SMTP_PORT=30025 MONGO_URL=mongodb://sandstorm:hOoHaN00omjW5B9vbd5l8mRsZ92@127.0.0.1:6081/meteor?authSource=admin BIND_IP=0.0.0.0 MAIL_URL=smtp://xxxx:xxxx@firewall.local:25 ROOT_URL=https://sandstorm.mydomain.net/ WILDCARDHOST=sandstorm-.mydomain.net METEORSETTINGS={"public":{"build":132, "kernelTooOld":false, "allowDemoAccounts":false, "allowDevAccounts":false, "isTesting":false, "hideTroubleshooting":false, "wildcardHost":"sandstorm-.mydomain.net", "sandcatsHostname":"sandcats.io"}}

Sandcats attempt:

cat /proc/$(ps axlf | grep 17771 | awk '/sandstorm-main/ { print $3 }')/environ

LANG=C.UTF-8 PATH=/usr/bin:/binLD_LIBRARY_PATH=/usr/local/lib:/usr/lib:/lib MONGO_OPLOG_URL=mongodb://sandstorm:qddhUROWOGtD8A-x5n9BbLipWe5@127.0.0.1:7081/local?authSource=admin PORT=7080 SANDSTORM_SMTP_PORT=30025 MONGO_URL=mongodb://sandstorm:qddhUROWOGtD8A-x5n9BbLipWe5@127.0.0.1:7081/meteor?authSource=admin BIND_IP=0.0.0.0 MAIL_URL=smtp://xxxx:xxxx@firewall.local:25 ROOT_URL=https://xxxx.sandcats.io:7443 WILDCARD_HOST=.xxxx.sandcats.io METEOR_SETTINGS={"public":{"build":132, "kernelTooOld":false, "allowDemoAccounts":false, "allowDevAccounts":false, "isTesting":false, "hideTroubleshooting":false, "wildcardHost":".xxxx.sandcats.io", "sandcatsHostname":"sandcats.io"}}

(The latter one is popping up and getting dying).

[noci2012]

ok issues solved in #1253 the instance now runs using port forwarding. On with haproxy.....

[paulproteus]

Great!

BTW rather than haproxy, you might try sniproxy - https://github.com/dlundquist/sniproxy - which is designed for HTTPS proxying in the form that Sandstorm+Sandcats needs, for what it's worth!

[noci2012]

All trouble was appearantly with the sandcats instance on the server. After recreating it while there was nothing else active on the server (http/https ports, as well as the sandstorm instance) and then renaming & moving all stuff to the sandcats intended ports & directories as well as updating the config files to match that; the haproxy config just worked. (HA proxy is one of a few that also plays nice with the microsoft brew of Exchange and friends over RPC over https....).

The Sandstorm installer isn't nice when there are already existing servers on ports 80/443. (it will create an invalid sandcats.io certificate with all trouble from that) And it appearantly creates some invalid mongoDB replicationset when there is another active instance. With an invalid database as a result. (master:no Slave: no) and if rs.slaveOk() is issued there is no valid admin user. (Hence the crashing node.js meteor process)

I will try SNI-proxy anyway just to see what it does.

Oh btw, making a backup download and uploading the image in the other instance works like a charm. (after the application of choice is installed.)

[noci2012]

Tips: Create a better way to specify ports/directories for installing more than one instance on a system. Be a better neighbour to existing services like webservers etc.

[paulproteus]

Thank you for these suggestions, and sorry you had trouble. I can work on addressing those problems in the install script over time, and can write tests so that we never regress on that behavior.

[noci2012]

Should i insert new separated issues for the tips or leave this open.

[paulproteus]

Hi @noci2012,

​New issues would be great. I'm traveling on vacation at the moment, or else I would create them myself. If you don't have time to, I can create them in a few days (Monday or Tuesday). Thanks!

[noci2012]

close with inherritence of individual problems with their own issue.