search

sandstorm-io / sandstorm

Sandstorm is a self-hostable web productivity suite. It's implemented as a security-hardened web app package manager.
https://sandstorm.io
Other
6.71k stars 704 forks source link

HTTP Proxy for outbound requests #693

Open JonTheNiceGuy opened 9 years ago

JonTheNiceGuy commented 9 years ago

Further to the email discussion with @kentonv:

Currently there are several places in Sandstorm's code where we use Node's "http" and "https" modules to make outgoing HTTP requests.

It looks like the following files use these modules:

shell/server/sandcats.js: To talk to the Sandcats server for dynamic DNS updates, if you use sandcats. shell/server/installer.js: To download packages. shell/server/hack-session.js: To implement the httpGet() API method, which e.g. TinyTinyRSS uses to fetch RSS feeds. shell/server/drivers/external-ui-view.js: To implement a more general mechanism for apps to make outgoing HTTP requests (not yet used by any apps).

And this file uses Meteor's HTTP API:

shell/packages/sandstorm-db/user.js: To fetch profile pictures from Google and Github.

Meteor's HTTP is a wrapper around the request npm package, which in turn wrap's node's HTTP client. It looks like request explicitly supports proxies (and Meteor's HTTP supports passing arbitrary options to request):

https://www.npmjs.com/package/request#proxies

It's also possible to get Node's raw HTTP client to use proxies per this StackOverflow question:

http://stackoverflow.com/questions/3862813/how-can-i-use-an-http-proxy-with-node-js-http-client

So we need to make sure that all the places where we make HTTP requests use the proxy.

I think the right way to do this would be to introduce a new Sandstorm-internal HTTP API that either wrap's Meteor's HTTP or the npm request package (probably the latter) and allows us to force all requests globally through a porxy, configurable through the admin interface. Then we should update all the above code to use it.

Additionally, run-bundle.c++ shells out to curl to fetch Sandstorm updates. I believe you can make curl use a proxy by setting an environment variable, so if you set that variable in the initscript that calls sandstorm start it should apply. (Maybe it makes sense for the Sandstorm shell to pick up the http_proxy environment variable automatically as well, rather than require setting it in the admin settings?)

A-Picogna commented 8 years ago

Hi,

I also have a problem with proxy, there is a way to add it to sandstorm for the moment ? to force the plate-forme going through it ?

paulproteus commented 8 years ago

Hi @A-Picogna ,

It's your lucky week! I'm testing this right now and one of my goals for the week is to make sure that this works.

I'm going to leave a few comments here about how I test this, so that I can test it again more easily in the future.

paulproteus commented 8 years ago

Goal: Run a Sandstorm process on my laptop with no access to the actual Internet, except via a HTTP proxy

Strategy:

grep SERVER_USER= /opt/sandstorm/sandstorm.conf
iptables -A OUTPUT -o wlp6s0 -m owner --uid-owner sandstorm -j DROP
sudo su - sandstorm -c 'cd /tmp ; wget http://www.google.com/'  # make sure this fails
sudo apt install squid
sudo su - sandstorm -c 'cd /tmp ; https_proxy=http://localhost:3128/ http_proxy=http://localhost:3128/ wget https://www.google.com/'  # this should succeed now

Great. Next up, adding it to Sandstorm.

(Note: wget requires lowercase http_proxy and doesn't use http_proxy for https:// URLs so requires both environment variables set.)

paulproteus commented 8 years ago 
[Unit]
Description=Sandstorm server
After=local-fs.target remote-fs.target network.target
Requires=local-fs.target remote-fs.target network.target

[Service]
Type=forking
ExecStart=/opt/sandstorm/sandstorm start
ExecStop=/opt/sandstorm/sandstorm stop
Environment=http_proxy=http://127.0.0.1:3128/
Environment=https_proxy=http://127.0.0.1:3128/

[Install]
WantedBy=multi-user.target
paulproteus commented 8 years ago 
sudo systemctl daemon-reload  # to get the new sandstorm.service into systemd's mind
sudo service sandstorm restart
paulproteus commented 8 years ago
paulproteus commented 8 years ago

I'm working on this right now, so self-assigning.

A-Picogna commented 8 years ago

Hi @paulproteus

I'll be very interested by your results ^^. I have a distant machine on an internal network which need to get through a proxy to acces internet.

the command : systemctl daemon-reload dosen't exist on ubuntu 14.04 apparently :)

paulproteus commented 8 years ago

Per #1438 I have app install working with a proxy now. Not everything is ready though, and this isn't in a released Sandstorm build yet.

kentonv commented 8 years ago

Github auto-closed this because it saw the sentence "does not fully close #693" and only paid attention to the "close #693" part. But this is not actually done yet, so re-opening.

paulproteus commented 8 years ago

Thanks @kentonv for fixing the metadata fail!

paulproteus commented 8 years ago

Next up - test/fix the following:

mitar commented 8 years ago

Could we also allow HTTP POST requests? This would allow apps to do bot POST requests in the name of the user. For me Sandstorm is really about making personal apps and such apps often also want to automatize some tasks. Sending an outbound HTTP POST is pretty common.

kentonv commented 8 years ago

@mitar This issue is not about grains doing HTTP requests, it's about making the Sandstorm shell's HTTP requests go through a proxy (as is required on e.g. many corp networks).

mitar commented 8 years ago

Oh, sorry. I misread. I read "To implement the httpGet() API method, which e.g. TinyTinyRSS uses to fetch RSS feeds." Is there a more appropriate ticket? Or should I open a new one?

kentonv commented 8 years ago

@mitar Dunno if there is a ticket. Secretly you can do posts today using HackSessionContext::getUiViewForEndpoint() and passing a regular old URL (rather than a webkey) and then requesting a WebSession. However, all these APIs will be replaced soon with Powerbox APIs (which will definitely support POST). Feel free to open a ticket or don't, either way it'll get fixed just as fast.

mitar commented 8 years ago

Opened #1924.

paulproteus commented 8 years ago

I haven't done any work on this lately, so I'm de-assigning from myself for now so that no one thinks I am.