Tor onion integration - Githubissues

david415 commented 9 years ago

chatting with Juan Bennet at c-base, Berlin about Tor onion service integration

here's what we've identified as necessary for proper Tor integration:

adding /onion to go-multiaddr - /onion/<onion-key>/ipfs/<ipfs-key>
adding /onion dialing to go-multiaddr-net
make a build of go-ipfs that:
- only uses /onion
- only bootstrap to onion nodes
- disable mdns service
- disable NAT service

I know plenty of people who'd be willing to run some Tor onion bootstrap nodes.

(edited by @jbenet for links)

david415 commented 8 years ago

the API returns the key unless you ask it not to... and in this case why would you not want to? of course we want to reuse onions. we are creating a new IPFS network with many onions.

cpacia commented 8 years ago

Yeah. I just looked at my code, I was having Tor put the hostname and private_key files in the .openbazaar directory and just reading in the hostname afterwards.

david415 commented 8 years ago

ah ok so you aren't using the new ADD_ONION control port command... it's really the recommended way to create onion services these days.

seanlynch commented 8 years ago

Added a couple of comments to the code, but I don't know the code at all so they're just stylistic/documentation-related. And just because I have a bounty doesn't mean you have to pay attention to my comments ;-)

What more is necessary for support? If this is it and someone can verify the work, should the bounty just go to david415? This is the first bounty I've seen come even close to getting claimed so I have no idea how the claim process works. I am pretty sure the person who wants to claim it should indicate on bountysource they intend to work on it, though.

cpacia commented 8 years ago

I was using this code to create a hidden service. It uses the tor control port. https://github.com/postfix/goControlTor

Edit: but I like the bulb you used much better.

cpacia commented 8 years ago

@david415 I added a comment as well.

david415 commented 8 years ago

haha yeah i forked yawning's bulb a long time ago when i was playing around... but instead everyone should use yawning's bulb. he's maintained and improved it.

david415 commented 8 years ago

@seanlynch the onion transport may be nearly done but i suspect there's still a lot more work to make ipfs work with tor... hopefully @whyrusleeping and @jbenet will have some time to chat and consider these things soon.

jbenet commented 8 years ago

yeah, have time soon. im in shanghai this comingweek, but if we can find a time that works, im game. maybe around the ipfs calls on monday. this or next.

cpacia commented 8 years ago

@david415 there's a number of compilation errors I'm working on fixing.

@jbenet I have some issues trying to integrate it into ipfs. Specifically the go-libp2p-transport library calls some methods that don't exist in it's dependencies (at least the current versions) and uses some non-exported structs. The main offenders are are reuseport and the utp library.

And it also is using the multiaddr from jbenet.

cpacia commented 8 years ago

Took a lot of hacking around but...

jbenet commented 8 years ago

@cpacia

wow that's excellent. 👏 :tada: :heart:
want to make a list of all the problems you ran into and what can make it easier/better for you? you have beggining of list up there, but maybe go beyond?
cc @whyrusleeping take a look! o/ -- also see https://github.com/ipfs/notes/issues/37#issuecomment-247684039
do you think https://github.com/ipfs/xtp would make life easier in the short term, or no? (we could dump some quick effort into this)

@david415

in light of this working, let's accelerate the "get ipfs ready for Tor" part
generate a list of all the things secrecy sensitive IPFS nodes MUST NOT do (i.e. the "privacy stripped out libp2p") and create milestones for that. (There may be an issue we started already-- i cant search quickly enough to find it on this plane...)
plan out how we will deploy this-- config to start? different distro to avoid fails? (go-ipfs-tor and go-ipfs-i2p or go-ipfs-anon)

cpacia commented 8 years ago

Yes I will make a list of things I ran into.

I also submitted a PR to @david415 for that transport it probably needs some clean up as it's kind of more of a proof of concept at this point.

I would also be interested in the list of things a node must not do because I only have a partial knowledge of what the codebase is doing.

david415 commented 8 years ago

hi @cpacia, thanks for the PR... actually we need to NOT decode the key like you are doing in your patch because the tor control port API ADD_ONION must receive this key in encoding form... it is a line protocol after all. anyway i will including your code changes and also make some additional corrections as well.

david415 commented 8 years ago

ok i made some more corrections here: https://github.com/david415/ipfs-onion-transport

how does this gx vendoring thing work? we need to give it a version of go-multiaddr with the recent tor onion changes. i'm not sure how to properly build this with all the correct versions of each dependency... but if it was to be built correctly then maybe the unit test would pass. anyway you can see yourself what output go test brings.

cryptix commented 8 years ago

Nice work @david415!

~~I know this sounds cheeky after you went through all this but how would you feel about decoupling this from ipfs so that it can be used by non-multiaddr tooling?~~ sorry.

david415 commented 8 years ago

@cryptix what work? can you articulate exactly which code you want decoupled from ipfs? actually I'm going to go ahead and say that makes no sense and it's off topic. please look at https://github.com/yawning/bulb i think that is what you want. it's "already decoupled" so please enjoy using bulb onions in your non-ipfs golang projects.

david415 commented 8 years ago

i'm quite frustrated that i was unable to build this software correctly. if the wrong imports are used and a gx hash for go-multiaddr is used that does not contain the onion to string decoder then the unit test fails horribly.

jbenet commented 8 years ago

@david415 sorry you had a hard time with gx-- the UX on it is definitely early and it needs much better docs/errors/help. (i've often been frustrated by it too-- but much less lately, it's been improving at a good pace, we're getting there! :] ).

@whyrusleeping can you help @david415 with gx Qs?

cpacia commented 8 years ago

@david415 re the key. I'm not sure I follow what you're saying. The bulb library takes in an RSA key.

I save the private key on disk in pem format just as Tor does when you create a hidden service using the config file. The following function turns that encoded key into a RSA key which is used by the bulb library.

block, _ := pem.Decode(key)
privKey, _, err := pkcs1.DecodePrivateKeyDER(block.Bytes)

I ran this code and it works. Doing it the other way didn't.

david415 commented 8 years ago

@cpacia yeah you wrote this correctly because the bulb API takes the key in the struct form via the crypto library... i was thinking in terms of the tor controller spec. it's a line protocol which takes the key in encoded form... not raw bytes... however this doesn't matter since bulb is the api we are using.

i'm glad you got it working for you! i ran my unit test and it did not work because the multiaddr it was built with didn't have the multiaddr to onion string conversion code... whereas it was just committed to the master branch a few days ago.

xloem commented 7 years ago

hi, just some feedback, as an extensive user of tor I really only want to create services with the config file. opening the control port up to network-connected apps does not make me feel safe. note that I may be running tor on an actual different machine than the service. that said, it is incredible that ipfs is so very close to supporting an onion transport ! and having that exist is certainly way more important than any specifics of how the service is created.

david415 commented 7 years ago

Application developers making tor friendly apps use the control port because it exposes the new ephemeral onion api (ADD_ONION and DEL_ONION control port commands). This is the recommended way to create onion services.

If you are commenting that the control port provides excess authority to the tor controller application, then you are correct. However this is a separable concern; There are Tor control port filter programs which white-list control port commands that are allowed to be used by a tor controller program. For example Subgraph OS has excellent Tor integration and we wrote Roflcoptor to deal with this problem of filter control port access: https://github.com/subgraph/roflcoptor

leycec commented 7 years ago

As expected from an open issue spanning several years, the thread of discussion has become fairly impenetrable to the lay outsider – namely, me.

Would one or more IPFS developers be willing to supply a high-level status update on both this and I2P integration? My gut intuition is that development on both has stalled out, for arcane reasons I'll probably never understand. (I'd love to be demonstrated wrong on that.)

I note that david415's ipfs-onion-transport purports to implement Tor integration support as an external gx package. Naturally, development on that appears to have stalled out as well. It also fails to provide even the thinnest veneer of usage or installation documentation. Naturally.

I'll be blunt. IPFS without anonymity is functionally useless for me.

I doubt I'm the only one. Thanks to the overtly DMCA-friendly jurisdiction in which I live, merely running a deanonymized IPFS node would unacceptably increase my risk profile. Default denylists would do little to nothing (...probably nothing) to ameliorate this risk and obviate the entire motivation for using IPFS in the first place. After all, pseudo-decentralized, censorship-resistant, fully anonymized file sharing protocols already exist (e.g., I2P-enabled BitTorrent).

IPFS was supposed to be something more – not less.

cpacia commented 7 years ago

I've had it working for a while now https://github.com/OpenBazaar/go-onion-transport

leycec commented 7 years ago

But... there's no documentation. On the other hand, go-onion-transport is at least well-maintained. On the gripping hand, lgierth of ipfs/infrastructure fame admits that:

Not gonna run an /onion bootstrapper until go-onion-transport has been audited.

So, we're back to square one.

My running assumption is that, like ZeroNet's official Tor integration, go-onion-transport requires (A) a system-wide installation of Tor (as opposed to the Tor instance bundled with Tor Browser), (B) the Tor control port to be enabled, and (C) the cookie authentication file for access to that port to be enabled. I suppose I should probably forward this query to go-onion-transport's issue tracker. </sigh>

Regardless, the larger issue remains: anonymity is a first-class issue. The difference between a cryptographically secure anonymity stack and an unofficial, undocumented, ad-hoc third-party plugin is citizens locked in cages at gunpoint. This is not hyperbole. Anonymity is not a trivial consumer preference to be idly left to unofficial, unverified, unaudited vendors with possibly dubious motives.

ZeroNet provides out-of-the-box support for Tor-based encapsulation with additional support for I2P on the way. So should IPFS.

In the long twilight of the 21st century, anonymity is no longer an option. It's a necessity.

flyingzumwalt commented 7 years ago

Thank you to @david415 @cpacia and all of the other contributors who have been directing their time, creativity and determination towards this feature. Making IPFS and the whole decentralized web support Reader and Writer privacy certainly is one of the many first-class issues we have to tackle. There will be terrible consequences for some users if we don't achieve Reader and Writer privacy soon.

Building IPFS is a huge open source community effort involving hundreds of contributors who are building technology that will impact millions of people. The core IPFS maintainers are constantly forced to choose between many equally relevant features. It often falls to individual contributors (or their employers) to prioritize the features that they need most urgently.

I'm confident that these features will get the attention they need in the coming months because IPFS continues to gain traction worldwide and we keep making it easier for people anywhere to run IPFS nodes. This forces the issue of Reader & Writer privacy.

cpacia commented 7 years ago

My running assumption is that, like ZeroNet's official Tor integration, go-onion-transport requires (A) a system-wide installation of Tor (as opposed to the Tor instance bundled with Tor Browser),

Either or. Browser bundle works fine.

(B) the Tor control port to be enabled, and

Yes

(C) the cookie authentication file for access to that port to be enabled.

I believe at this time only pw based authentication is supported.

ig0rsky commented 7 years ago

Any updates on this?

flyingzumwalt commented 7 years ago

The onion transport for IPFS still has not been audited yet. Until it has been audited you should consider it experimental and should not assume that it guarantees anonymity. When we audit it, we will announce the results. In the meantime, people are welcome to conduct their own audits, or arrange for third parties to conduct audits. That would not be wasted effort; It's good to have multiple entities audit systems like this.

david415 commented 7 years ago

ugh.

seanlynch commented 7 years ago

Anyone have suggestions for professional code auditors and/or know the general price range for such a thing?

ian-gtv commented 7 years ago

Personally I would use Cure53 and I would guess of the order of 10,000 pounds depending on the scope, could easily go up to 40,000.

seanlynch commented 7 years ago

Hmm, OK. I don't see personally contributing more than 1500 USD toward an audit, and even the bottom end of that range seems like it would be a challenge to raise given that the original bounty I posted hasn't even been doubled so far. Maybe if someone knows some researchers who might be interested in doing a paper on the topic?

On Mon, Aug 14, 2017 at 10:38 AM ian-gtv notifications@github.com wrote:

Personally I would use Cure53 and I would guess of the order of 10,000 pounds depending on the scope, could easily go up to 40,000.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ipfs/notes/issues/37#issuecomment-322256603, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKO-vFW_VPGFA_qtjnnOKjkAIqC_fv6ks5sYIX_gaJpZM4FzoVc .

Kelketek commented 7 years ago

@seanlynch It might be possible to use crowdfunding. I'd chip in a bit. Could notify a few mailing lists and subreddits or what have you.

seanlynch commented 7 years ago

I'll happily chip in if someone finds an auditor and sets up a gofundme or whatever that's refundable if the goal isn't met, assuming that it has support of the core devs.

On Mon, Aug 14, 2017 at 11:38 AM Kelketek notifications@github.com wrote:

@seanlynch https://github.com/seanlynch It might be possible to use crowdfunding. I'd chip in a bit. Could notify a few mailing lists and subreddits or what have you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ipfs/notes/issues/37#issuecomment-322272648, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKO-urgiSSRSUe5mnP0Ucm2z7P2bS-Aks5sYJQRgaJpZM4FzoVc .

ianopolous commented 7 years ago

The first thing to do would be to contact cure53 and ask for a quote: https://cure53.de/#services-analysis You can also look into funding sources like the Open Technology Fund.

jbenet commented 7 years ago

Protocol Labs is now capable of funding independent audits of the whole IPFS stack. We have started some work and RFPs on this and we can accelerate it to cover this part first. This shouldn't hold back any other groups wanting to audit. we can take care of at least one of the bills. I think our ETA was in the next few months (IPFS is big), but we can see what's needed.

cc @diasdavid.

ghost commented 6 years ago

@jbenet @flyingzumwalt Is this the code that you need audited?

https://github.com/OpenBazaar/go-onion-transport

notzoom commented 6 years ago

So what's up with this ? How do we run it over TOR ?

Stebalien commented 6 years ago

We want to get all of IPFS and libp2p audited before we start encouraging people to use it with TOR. Unfortunately, this will take a while.

llgoer commented 6 years ago

Has there been any recent progress?

Stebalien commented 6 years ago

@llgoer we've been pushing through a large refactor of how transports work that'll make this simpler. We've also made some progress internally towards auditing the IPFS stack. However, you shouldn't expect a blessed and fully functional/recommended tor transport for a while.

cretz commented 6 years ago

An issue on https://github.com/OpenBazaar/go-onion-transport just reminded me to update this issue...I wrote https://github.com/cretz/bine which may make it easier to use go-ipfs over Tor. Granted a simple SOCKS5 proxy is enough to reach the public web out of an exit node, but if you want to start ephemeral onion services or statically link Tor, give that project a look.

andyli commented 6 years ago

For the record, Tor 0.3.3.6 was released earlier this week:

There is now a documented stable API for programs that need to embed Tor. See tor_api.h for full documentation and known bugs.

lkjh654 commented 5 years ago

Has any progress been made on the issue recently?

david415 commented 5 years ago

From my perspective this Tor integration experience was suboptimal. At least Open Bazar is using the code we wrote.

lkjh654 commented 5 years ago

If this integration was suboptimal is there any other integration in progress that will be "optimal"?

Stebalien commented 5 years ago

We have no plans on integrating TOR into go-ipfs officially until there has been full security audit.

david415 commented 5 years ago

It's Tor not TOR. That policy decision is a bad decision in my humble opinion for many reasons that I shall not elaborate on here. I explicitly told Juan that I was concerned about the prospect of writing a bunch of code for free that will then not get used. Best wishes to you all in your present and future software projects. Cheers.

ipfs / notes

Tor onion integration #37