[http_proxy_over_p2p]

[cboddy]

This implements an http-proxy over p2p-streams, for context see https://github.com/ipfs/go-ipfs/issues/5341.

This script is a useful test of the functionality. In case it causes portability issues I've not included it as a sharness test (since it uses python to serve HTTP content, although happy to add it since python be ~ as available as bash).

(inline since GH doesn't support *.sh as an attachment)

#!/bin/bash                                                                                                                                                   

#                                                                                                                                                             
# clean up all the things started in this script                                                                                                              
#                                                                                                                                                             
function teardown() {                                                                                                                                         
    jobs -p | xargs kill -9 ;                                                                                                                                 
}                                                                                                                                                             
trap teardown INT EXIT                                                                                                                                        

#                                                                                                                                                             
# serve the thing over HTTP                                                                                                                                   
#                                                                                                                                                             
SERVE_PATH=$(mktemp -d)                                                                                                                                       
echo "YOU ARE THE CHAMPION MY FRIEND" > $SERVE_PATH/index.txt                                                                                                 
cd $SERVE_PATH                                                                                                                                                
# serve this on port 8000
python -m SimpleHTTPServer 8000 &

cd -

IPFS=cmd/ipfs/ipfs

PATH1=$(mktemp -d)
PATH2=$(mktemp -d)

RECEIVER_LOG=$PATH1/log.log
SENDER_LOG=$PATH2/log.log

export IPFS_PATH=$PATH1

#
# start RECEIVER IPFS daemon
#
$IPFS init >> $RECEIVER_LOG 2>&1
$IPFS config --json Experimental.Libp2pStreamMounting true >> $RECEIVER_LOG 2>&1
$IPFS config --json Addresses.API "\"/ip4/127.0.0.1/tcp/6001\"" >> $RECEIVER_LOG 2>&1
$IPFS config --json Addresses.Gateway "\"/ip4/127.0.0.1/tcp/8081\"" >> $RECEIVER_LOG 2>&1
$IPFS config --json Addresses.Swarm "[\"/ip4/0.0.0.0/tcp/7001\", \"/ip6/::/tcp/7001\"]" >> $RECEIVER_LOG 2>&1
$IPFS daemon >> $RECEIVER_LOG 2>&1 &
# wait for daemon to start.. maybe?
# ipfs id returns empty string if we don't wait here..
sleep 5

#
# start a p2p listener on RECIVER to the HTTP server with our content
#
$IPFS p2p listen /x/test /ip4/127.0.0.1/tcp/8000 >> $RECEIVER_LOG 2>&1
FIRST_ID=$($IPFS id -f "<id>")

export IPFS_PATH=$PATH2
$IPFS init >> $SENDER_LOG 2>&1
$IPFS config --json Experimental.Libp2pStreamMounting true >> $SENDER_LOG 2>&1
$IPFS daemon >> $SENDER_LOG 2>&1 &
# wait for daemon to start.. maybe?
sleep 5

# send a http request to SENDER via proxy to RECIEVER that will proxy to web-server

echo "******************"
echo proxy response
echo "******************"
curl http://localhost:5001/proxy/http/$FIRST_ID/test/index.txt

echo "******************"
echo link http://localhost:5001/proxy/http/$FIRST_ID/test/index.txt
echo "******************"
echo "RECEIVER IPFS LOG " $RECEIVER_LOG
echo "******************"
cat $RECEIVER_LOG

echo "******************"
echo "SENDER IPFS LOG " $SENDER_LOG
echo "******************"
cat $SENDER_LOG

[ianopolous]

(I tested both formats of the endpoint with the full Peergos web ui)

[ianopolous]

@Stebalien how can we get the gateway address out of iptb? Any chance you could update the sharness tests to the new spec?

[Stebalien]

(also, please upgrade to go 1.11; go made a go fmt breaking change)

[Stebalien]

LGTM now (although I'd like to improve the documentation, either now or later). However, we should get at least one more signoff (@magik6k or @hsanjuan?).

[hsanjuan]

Overall looks OK, but I'll look into this with a bit more time in a couple of days, unless it's super urgent, in which case just ping me.

[hannahhoward]

@hsanjuan -- I wonder if this is worth moving forward cause it's blocking https://github.com/ipfs/go-ipfs/pull/5762 -- which is needed to fix everyone's problems with gx-go linking subpackages with go-ipfs/master right now. Or, if you feel it's worth moving #5672 ahead of this for that reason (understanding it will be a challenging rebase)

[Stebalien]

Is this blocked on me in any way?

[ianopolous]

@stebalien I don't think there's anything else left to do here?

[ianopolous]

@magik6k I believe @Stebalien said that those random multipart failures are independent of this PR and you guys have been hitting them elsewhere too?

[Stebalien]

I think I've fixed the netcat issue. We'll see if we can get the tests to pass.

[Stebalien]

OK, tests pass and @hsanjuan has given a :heavy_check_mark: :steam_locomotive: (this train may be a bit slow...).

[ianopolous]

Hooray! The last critical feature we need for Peergos!

@stebalien is the plan to enable this on ipfs.io?

[cboddy]

@Stebalien @magik6k @hsanjuan thanks for your help with this one!

[Stebalien]

@Stebalien is the plan to enable this on ipfs.io?

Probably not for a while, if ever. We'll have to seriously consider the security and performance implications and, really, the public gateway's more of a crutch (we'd prefer if users would just load js-libp2p from their DAPP).

[ianopolous]

Probably not for a while, if ever. We'll have to seriously consider the security and performance implications and, really, the public gateway's more of a crutch (we'd prefer if users would just load js-libp2p from their DAPP).

:-( That's a shame, because can do some really cool demos if that is enabled. Loading js-libp2p will never be an option for us for security reasons.

[Stebalien]

Are you forbidding javascript entirely? Can you not spin up a js-ipfs node in a sandboxed worker?

I do agree this would allow for some pretty cool demos but I'd like to sit on it at least until it's stabilized.

[ianopolous]

@Stebalien In general, we veto anything that uses npm which is anathema to basic security practices. We're also trying to keep the JS down to a small easily auditable amount (as much as possible) for the same reasons. As well as being much harder to audit for attacks, having a complex P2P daemon running in the same process as that which has the secret keys is also asking for trouble. Browsers explicitly don't defend against spectre attacks within the same page to my knowledge (they'd have to run every worker in an independent OS process, not just different OS threads). We also have reproducible builds for the web-ui already.

For now we can just preface all ipfs-only demos with, install ipfs and enable these two flags, then browse to this local url.

[Stebalien]

In general, we veto anything that uses npm which is anathema to basic security practices.

Well, I can't really blame you there...

Browsers explicitly don't defend against spectre attacks within the same page to my knowledge

Sandboxing has more to do with origins/contexts than pages. I'm not familiar with browser spectre mitigations but, a page in a sandboxed iframe/worker should have equivalent security to a page running in a separate tab.

[ianopolous]

Sandboxing has more to do with origins/contexts than pages. I'm not familiar with browser spectre mitigations but, a page in a sandboxed iframe/worker should have equivalent security to a page running in a separate tab.

If the p2p daemon was running from a separate domain in an iframe, I believe this might be safe in some browsers (depending on their site isolation properties), but it's much harder to prove it safe, and it requires you to have two separate origins. We don't have any 3rd party hosted content in our web-ui - it's all self hosted (partially to enable easy offline and localhost access).

How about we continue this on irc to stop spamming all the other contributors to this pr? :-)