Migrate to Tor v3 onion service protocol

[freimair]

This project is about migrating from Tor Hidden Services version 2 to Tor onion services version 3 and the required steps to do so.

Rationale

Why do we want to update?

• we expect significant improvements during startup
• improves security/privacy
• might fix https://github.com/bisq-network/bisq/issues/3987
• might prevent clients/seednodes from occasionally being denied access to the Tor network
• we have to upgrade at some point, HSv2 might someday hit EOL

Why do we want to update now?

• a recent tor update now defaults to HSv3
• the tor binary is due to be updated anyways
• might just do the HSv3 switch now

Risks

• the HSv3 has proven itself for years now, so very little risk there
• if we do not move forward, we may risk getting left behind and then do the work in a hurry

Details

The Onion Router (TOR) offers a [version 3 of its hidden service technology](https://blog.torproject.org/tors-fall-harvest-next-generation-onion-services) (HSv3) for quite [some time](https://blog.torproject.org/we-want-you-test-next-gen-onion-services) now. Bisq, until now, held onto HSv2 because the Tor devs themselves did not consider HSv3 as ["BEST"](https://gitweb.torproject.org/torspec.git/commit/?id=b32a9bf315e270cbb35a65f2c0db1dc2f4861810), at least until Tor [v0.3.5.1](https://gitweb.torproject.org/torspec.git/commit/?id=b32a9bf315e270cbb35a65f2c0db1dc2f4861810). However, a [tiny line of code](https://trac.torproject.org/projects/tor/ticket/29669) prevented HSv3 to actually become the expected default back then, so we spent our time working on other Bisq issues. Now, motivated by an upcoming tor [0.4.2](https://trac.torproject.org/projects/tor/query?id=29669) and https://github.com/bisq-network/bisq/issues/2873 it is time to seriously think about HSv3 and how Bisq can make a transition from HSv2 to HSv3. ### Tor Hidden Service Versions - non-functional differences [source](https://2019.www.torproject.org/docs/tor-onion-service.html.en#four) - HSv3 has improves security/privacy - HSv3 features new introduction/rendezvous protocol (may boost performance - source needed!) - HSv3 features a cleaner codebase - technical differences [source](https://trac.torproject.org/projects/tor/wiki/doc/HiddenServiceNames) - 56 char address HSv3 vs. 16 char address in HSv2 - HSv3 is based on SHA3 and EC cryptography, HSv2 is based on SHA1 and RSA cryptography (in words, SHA1 is on the brink of being broken, and Elliptic Curve Cryptography (ECC) is more futureproof AND faster than the old and trusty RSA)

Tasks

Milestone: Get Bisq ready to talk to HSv3

• [x] done

  Milestone: Proof of concept

• [x] done

### Tasks - [x] upgrade v2 to v3 in a branch ([via](https://github.com/bisq-network/bisq/pull/4028)) - [x] deploy a test network and test all platforms, mix of v2 clients and v3 - [x] measure performance improvements, if any - [x] decide whether to move forward ### Criteria for delivery - [x] successful TestPlan test run - all major platforms - mix of v2 and v3 clients - mix of v2 and v3 seed nodes - [x] publish performance eval here and [there](https://github.com/bisq-network/proposals/issues/101) - client startup time - message RTT using Ping/Pong - get baseline from https://monitor.bisq.network ### Estimates - USD 400,00 to update tor - USD 1200,00 for testing - USD 400,00 to create performance reports

Milestone: Ship it

• [x] done

### Tasks - [x] make Bisq report its HS version to the pricenodes, similar to the Bisq version [via](https://github.com/bisq-network/bisq/pull/4027) - [x] amend version spread metric infrastructure to also collect hidden service version spread [via](https://github.com/bisq-network/bisq/pull/4027) - [x] deploy/upgrade the metric infrastructure - [x] configure https://monitor.bisq.network to display HS version spread similarly to Bisq version spread - [x] create one new seednode to use v3 hostname ([here it is](https://monitor.bisq.network/d/qclhStdWz/server-metrics?orgId=1&var-Server=wizseedscybbttk4bmb2lzvbuk2jtect37lcpva4l3twktmkzemwbead)) - [x] add it to the main seed nodes for the upcoming release ([via](https://github.com/bisq-network/bisq/pull/4113)) - [x] look to support team to report potential issues - brief them - alert them to specific error cases - [x] release feature - [x] wait and see how it goes over time ### Criteria for delivery - [x] all new clients use v3 hostnames (release the feature) - [x] create one new seednode use v3 hostname - [x] deliver a report to foster a decision on whether we proceed to milestone 3 - report is due on 2020-06-30 ### Estimates - USD 950,00 for additional metrics - USD 250,00 for v3 seednode (200) + make it a default seednode in Bisq-update (50) - USD 500,00 for report

Milestone: Allow old clients to upgrade

Since Tor has officially deprecated Tor onion services v2 source we should prepare Bisq as well. Final deadline (v2 is no longer available) is 2021-07-15, although, we can delay adapting to new tor binaries and thus gain a few days, on the other hand, Tor will show warnings in upcoming releases (as soon as 2020-09-15, i.e. next week) and we do not know how good their v2 "DNS" servers will hold up. Finally, on 2021-10-15 there will be no v2 "DNS" servers anymore and thus, v2 onion addresses will not be accessible anymore.

Tasks

• [ ] reenable HS version reporting on the monitor to see our progress in migrating to v3
• [ ] create the necessary tools to migrate properly (by reenabling https://github.com/bisq-network/bisq/pull/3044 as it is 90% there)
• [ ] create test suit, test code and testpad
• [ ] find a solution to optionally transfer reputation from one onion address to another?
• [ ] create a timeline with measures to motivate users to switch to new onion addresses
• [ ] execute the timeline

  Criteria for delivery

• [ ] enable users to renew onion addresses, warning them that reputation is not transfered
• [ ] transfer reputation when migrating from Tor Hidden Services version 2 to Tor onion services version 3 (?)
• [ ] all clients have moved to v3 onion addresses

  Estimates

• USD 4350,00 for work done on https://github.com/bisq-network/bisq/pull/3044 in the past
• USD 1550,00 for bringing the PR up to speed and over the finish line
• USD 2250,00 for testing
• USD 1000,00 for creating the tools necessary to execute the timeline
• USD 1500,00 for creating and executing the timeline

Please note that compensation for the first three points should be available as soon as these tools make it into master. There is no point in delaying the compensation for a year. The tools necessary for executing the timeline may only be created according to the timeline.

Milestone: Cleanup

Tasks

• [ ] revert https://github.com/bisq-network/bisq/pull/4027

[cbeams]

Reflecting our call last week on this project, I've just applied the has:approval and has:budget labels with regard to the first (proof of concept) milestone. @freimair, please transition this issue to the In progress column if indeed work is underway (which I believe it is).

[freimair]

Report Review Meeting 2020-03-19

• all on track
• testing setup is in progress
  • talked to @devinbileck about testing strategies and he will create a testplan instance once required
  • binaries for all platforms and testing will be prepared
  • then, testing can commence
• I am confident that we can complete the proof-of-concept milestone by end of march.
• first tasks of the ship-it milestone are in progress already

[cbeams]

Roger that, @freimair. Thanks for the update even in the absence of yesterday's planned (but cancelled) review meeting.

[freimair]

Performance eval

TL;DR hidden service publishing time has dropped from around 33 seconds to <5 seconds, no significant changes otherwise

• 2 evaluations
  • use the monitor for "isolated" Tor measurements
  • Integration tests: start Bisq app to confirm findings

Recent changes to Bisq had an impact on the startup times already, so I decided to get the baseline manually instead of relying on the monitor.

Isolated Tor measurements

• to get an idea of what we can expect

  Setup

• single PC, Arch linux, Kernel 5.5.13, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, Samsung_SSD_970_EVO via M.2 interface
• devices under test (DUTs)
  • bisq-monitor with HSv2: compiled from v1.2.9 tag
  • bisq-monitor with HSv3: compiled from v1.2.9+https://github.com/bisq-network/bisq/pull/4028
• DUTs have been copied to their own workspaces for isolation purposes
• metrics
  • tor startup time: time until Tor.getDefault() returns, includes installing tor to disk and starting it up
  • hidden service startup time: time between telling tor to publish a hidden service until the service has been confirmed by the tor network (in detail, time until the first tor relay announces that it added the hidden service to its "DNS" entries)
  • RTT
  • use a started tor to query an hidden service; ie. v3 onion address of whonix.org
  • 5 measurements each, calculate average, min, max, p25, p50, p75
• test runs alternate between v2 and v3

  Results

• hidden service publishing times dropped from around 33 seconds to < 5 seconds
• no significant difference in Tor startup times
• no significant difference in RTT

raw data

Integration Tests

• to see if the results of the previous evaluation holds for the real thing.
• please note, measuring the startup-time of the Bisq app until ready is tricky
  • lots of external factors (seed node availability, age of local database, request size...)
  • no definite point of "bisq has started"
• hence, only limit measurements to the stuff we changed

  Setup

• single PC, Arch linux, Kernel 5.5.13, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, Samsung_SSD_970_EVO via M.2 interface
• devices under test (DUTs)
  • bisq-desktop with HSv2: compiled from v1.2.9 tag
  • bisq-desktop with HSv3: compiled from v1.2.9+https://github.com/bisq-network/bisq/pull/4028
• DUTs have been copied to their own workspaces for isolation purposes
• test run script pseudocode
  • clean tor hidden service directory
  • fire up bisq-desktop
  • terminate it after 60 seconds
  • wait for ~10minutes
  • repeat
• metrics
  • tor startup time: extract from bisq.log.
  • hidden service startup time: extract from bisq.log.

    ################################################################
    Tor started after 7182 ms. Start publishing hidden service.
    ################################################################

    ################################################################
    Tor hidden service published after 1832 ms. Socked=HiddenServiceSocket[addr=kzafzl4nu5kb5vj6tn4esthv5hnnflo6kw6utnsq5hfdjfsmatbhacad.onion,port=9999]
    ################################################################ 

• run script for each DUT at the same time, over a couple of hours

  Results

• hidden service publishing times dropped from around 33 seconds to < 5 seconds
• no significant difference in Tor startup times

  [raw data](https://github.com/bisq-network/projects/files/4408264/bisq-results.zip)

[cbeams]

@freimair, could you give a status update about this project?

[freimair]

Everything is done for Milestone 2 except the "wait and see how it goes over time" part which will be concluded by the upcoming report.

Overall, it seems that v3 onion services did the project no harm whatsoever. I have been monitoring support and issues and there is not a single hint that something does not work - quite the contrary, it feels like support issues regarding tor connection issues and/or startup-issues for people using system tor have vanished. Will do a proper report as scheduled.

[freimair]

Closing report on milestone: "Ship it"

There have been no reports that would indicate something not working properly or causing issues with Tor onion services v3. Quite the contrary, it seems like connection issues have gone down. It even seems that people have upgraded to v3 on their own - at least that is one explanation of the number of v3 hosts in the Bisq P2P network has reached the neighborhood of 40% already.

All in all, although we could and probably should have done it earlier, moving to Tor onion services v3 has been a success.

[ripcurlx]

Regarding the next steps for this project. I would only add the option to update to a new v3 address without taking the reputation over. I don't think it is worth the hassle to do this.

[wiz]

Well, now that V2 is officially deprecated, migrating to V3 is required and we will need to force all users to migrate to V3 onions at some point in the future. Some kind of migration plan like:

• After X date, disable taking offers using V2 onions
• After Y date, disable making offers and all active offers using V2 onions
• After Z date, disable V2 onion entirely and force users to upgrade

[freimair]

basically:

1. do we want to just move users to v3 once or
2. do we want to allow them to change their address at will?

both need similar preparations. in order to safely change the hidden service host users need to

• make sure they have no ongoing trades
• make sure they have no ongoing disputes
• (already fixed that a while ago https://github.com/bisq-network/bisq/pull/4021) make sure they have no open offers

violating these preconditions will result in unrecoverable trade states and lost funds.

Once we hit wiz's third step, we need to do these checks automatically.

My suggestion is to reactivate this one year old PR: https://github.com/bisq-network/bisq/pull/3044. It provides the tools to follow wiz's proposed timeline.

Your opinion?

[chimp1984]

@ripcurlx @cbeams Can we close that project?

[cbeams]

Closing as complete given that we have in fact migrated to Tor v3. The milestone on allowing clients to upgrade was not completed, but it appears to be a separate concern at this point.

[cbeams]

I've also closed the associated project board at https://github.com/orgs/bisq-network/projects/15.