Closed dan-da closed 8 years ago
ManfredKarrer
commented
8 years ago Yes that is on the todo list as well. The higher prio for me is to get BitcoinJ routed over Tor as BitcoinJ leaks more important data (bloomfilter) that just price requests or block explorer api calls. Though it is definitely planned to implement that. Any volunteer to work on that? Should be not too hard....
dan-da
commented
8 years ago yeah, I'm looking into the bitcoinj thing, that's why I noticed this issue with blockchain providers. I saw your todo tor comment for HttpClient. Just added this issue for tracking purposes.
ManfredKarrer
commented
8 years ago Thanks. If you have time to check out how to route the http content over our tor socket would be great!
dan-da
commented
8 years ago update.
It is actually very simple to route the existing HttpClient over a socks proxy. You just pass a Proxy object to the url.openConnection() call, and voila! The big problem with this approach is that DNS resolution is done locally, so it leaks IP in that fashion. Maybe not a big concern for most, but I was not satisfied. I feel like it is a much simpler story if we can just tell people that all network requests are routed over tor when Tor proxy enabled.
I did a long search for solutions on the internet and it is pretty depressing actually. It boils down to, java framework does not directly support remote DNS nor any third party libraries I could find. Apache HttpClient does support socks, just not remote DNS, and I was able to find a stackexchange answer demonstrating a technique that I successfully integrated into BitSquare.
I tested it, and it works fine with bitcoinaverage. That's the good news.
The bad news is that poloniex uses cloudflare which puts up a captcha page whenever they detect traffic from a Tor exit node. I don't see any good way around that. And I suspect several of the other API providers may as well, though I haven't tested them explicitly.
Possible solutions:
Anyway, I will go ahead and finish up the basic integration and submit that as a pull request. The code is written so that the proxying functionality can be easily enabled/disabled at runtime.
ManfredKarrer
commented
8 years ago Wow great progress! Had you looked into jsocks (that lib/module inside Bitsquare)? @JesusMcCloud recommended that instead of Java's implementation.
I am still very busy with the trade statistic so I cannot have a closer look. But seems the F**king cloudflare issue with Tor is a real problem. I don't want to change the price provider as that would break backward compatibility. Thought the prive provider is the least privacy critical part. Blockchain explorers leak more info (which addresses a user is interested in). Getting the price feed should not even easily be possible to assign to Bitsquare (we use randomized interval). But to get that solved would be cool of course...
dan-da
commented
8 years ago Yeah, I reviewed the jsocks classes a few times. It only implements a socks proxy; there is nothing to help with the client-side tasks of sending DNS queries over the proxy or making http requests.
That's true that the blockchain providers are more important. I'm not sure what yet exactly they are used for in bitsquare, or how to test without actually making trades. Anyway, it should be simple enough for me to wire them up, and then you can review when you have more time.
oh, one more possibility would be to setup a bitsquare hosted proxy somewhere that acts as a middle man between tor exit nodes and cloudflare'd services. I know not ideal. edit: I don't like it. too centralized, too many trust issues.
ManfredKarrer
commented
8 years ago The blockchain providers are only used when the user clicks on a tx id or address. Bitsquare never connects to them by itself. The price feed providers are called automatically though. All http calls use Utilities.openWebPage().
I was thinking as well on that hosted proxy, but did not follow it up for the same reason :-).
dan-da
commented
8 years ago hmm. well Utilities.openWebPage() invokes external web browser, so as far as I can see, that's not our responsibility to route over Tor. Though quite possibly should warn user!!
It is the callers of HttpClient.requestWithGet that I believe this issue deals with, though I could quite possibly be missing something. Haven't done an exhaustive search yet. let's see...
core/src/main/java/io/bitsquare/btc/pricefeed/providers/BitcoinAveragePriceProvider.java: LinkedTreeMap<String, Object> treeMap = new Gson().fromJson(httpClient.requestWithGET("all"), LinkedTreeMap.class);
core/src/main/java/io/bitsquare/btc/pricefeed/providers/BitcoinAveragePriceProvider.java: .parse(httpClient.requestWithGET(currencyCode))
core/src/main/java/io/bitsquare/btc/pricefeed/providers/PoloniexPriceProvider.java: String response = httpClient.requestWithGET("?command=returnTicker");
core/src/main/java/io/bitsquare/btc/pricefeed/providers/PoloniexPriceProvider.java: .parse(httpClient.requestWithGET(currencyCode))
core/src/main/java/io/bitsquare/btc/blockchain/providers/BlockTrailProvider.java: .parse(httpClient.requestWithGET(transactionId))
core/src/main/java/io/bitsquare/btc/blockchain/providers/TradeBlockProvider.java: .parse(httpClient.requestWithGET(transactionId))
core/src/main/java/io/bitsquare/btc/blockchain/providers/BlockrIOProvider.java: .parse(httpClient.requestWithGET(transactionId))FeeProvider:
PriceProvider
Ok, so I tracked down the callers of FeeProvider and it seems it is only used by CreateOfferDataModel and TakeOfferDataModel, both of which have commented out the call to requestFee. So the fee providers are not being used at all now, correct?
In that case, this issue only involves BitcoinAverage and Poloniex.
correct statement?
ManfredKarrer
commented
8 years ago Ah yeah, my fault, sorry... Yes the blockchain explorers are only called via web browser. So as you said a warning should be enough. The fee request might be implemented later, so that would be a use case (I forgot to mention).
dan-da
commented
8 years ago I'm having some difficulty getting HttpClient to accept a Socks5Proxy in its constructor. This is because PriceFeed is used in many places and I can't seem to find exactly where instantiated. I guess this has to do with injection, which I'm not familiar with?
I suspect the right thing to do may be to create a Singleton for Socks5Proxy. or maybe something to do with injection? I'm not really a java guy, and not sure of best approach. So if you can point me the way, or just make a Socks5Proxy singleton available for HttpClient, that would be helpful, and then I can wrap this up.
ManfredKarrer
commented
8 years ago Yes with guice you jus pass the object in the constructor and get in instantiated by guice.
You can create a Singleton for it if that works for you. I can refactor it later using guice.
ManfredKarrer
commented
8 years ago Any new idea regardign Poloniex/Cloudflare issue? I think I will route it to non-proxy if the bitsquare proxy is used and display a popup with info. If the user has setup an external proxy i dont route non-proxy. Worst case the dont get price feed id he use external Tor proxy, but there is no really point to use an external tor proxy as it will behave the same like the internal. Though if the user uses other proxies (i2p,...) he might have luck to get over f**cking cloudflare.
ManfredKarrer
commented
8 years ago I pushed to Dev branch. If you like you can check it out and test if all works. I did not test much yet...
Browsing through the source code I noticed that calls to API providers are not routed over Tor. This is a privacy leak.
A quick list I found:
Fee Providers: blockr, blocktrail, Tradeblock core/src/main/java/io/bitsquare/btc/blockchain/providers/
Price Providers: bitcoinaverage, poloniex. core/src/main/java/io/bitsquare/btc/pricefeed/providers/