[Design] feat: UTXO Consolidation

[msbrogli]

If we include an utxo consolidation during wallet initialization (with user's approval, of course), we can considerably improve wallet's loading time using the design proposed on this issue in hathor-core. The proposed solution would not be useful if the addresses have few UTXOs (which is not the case for most miners).

The basic idea is that the wallet can suggest a consolidation based on the number of utxos when it is loading. The flow must first suggest an UTXO consolidation and wait for the user's approval.

Even though it would be probably simple to detect consolidated transactions on the blockchain, I don't think it is a problem because many users are already consolidating their UTXOs as we can see in transactions with many inputs (usually more than one hundred) and few outputs (usually one).

We must consolidate UTXOs of the same address because it wouldn't affect the user's privacy.

Consolidated Transaction

The consolidation can be done creating new transactions with many inputs and a few outputs.

Let's say a wallet has 40 transactions (txA1, txA2, ..., txA40) for addr1 that sums X and 50 transactions (txB1, txB2, ..., txB50) for addr2 that sums Y. So, we would have to create two consolidated transactions like this:

tx1

• Inputs: [txA1, ..., txA40]
• Outputs: [(addr1, X)]

tx2

• Inputs: [txB1, ..., txB50]
• Outputs: [(addr2, Y)]

Notice that X and Y are balances that can be of more than one token. So, the token list must include all tokens of the inputs, and each address will have one output for each token (unless the balance for a given token is zero).

We cannot use only one transaction because we may expose the user's real balance.

UX Considerations

These considerations should be applied to all wallets implementing this design.

1. User must approve all consolidations individually before executing them.
2. Wallets can have an option to always consolidate transactions where all inputs and outputs have the same address.
3. Wallets should suggest consolidation both during its initialization and after a new transaction has arrived.

Drawbacks

Timing issue for privacy

If there too many UTXO and multiple transactions are created during the consolidation, the user's privacy many be exposed when the number of transaction in the network is low. Basically, two consolidation transactions with a small timestamp difference them are very likely to be part of the same wallet.

Future work

Maybe we should also add support for another type of UTXO consolidation that creates transactions using inputs from multiple addresses. Even though it can expose user's balance, it may be useful in specific cases.

[obiyankenobi]

My main question is when we'd display this option to the user:

1. During load. We stop loading new txs after we reach a threshold of utxos and ask him this. If he denies, we continue loading;
2. After loading is done. Only after everything is loaded we ask about utxo compression.

[jansegre]

The term "compression" gives me the wrong impression of what this is about (I immediately think of data compression). I think "UTXO aggregation" might be better.

Let me see if I understand correctly. This proposal is about automatically aggregating UTXOs in a manner that does not reveal any more information than if the UTXOs were scatter (not aggregated).

One issue that comes to mind is how this would work with a hardware wallet, where the only safe practice is having the user approve every the signing of every transaction. Because of this I wonder if it isn't better start by not doing this automatically, and instead providing a "button" (under some advanced section or only shown in certain conditions) so the user can do this manually.

The only exception to this "signing every transaction" I can think of is if the input address and output address are the same, so the hardware wallet can agree to sign that kind of transaction silently, be even though I'm not sure what the implications would be, and for that reason I'd suggest that we start with a "manual step" instead. I mean, this is supposed to be useful mostly for miners, which right now may be the majority of active Hathor users, but that shouldn't be the case in the future I think. Miners are usually power users, so I think it's fine to have this as an advanced tool.

[msbrogli]

I agree that we must request user's approval. Answering both @obiyankenobi and @jansegre, I think we should have a set of tools and compression will be available there, and we should also detect that there are too many UTXOs which are compressible and suggest compression while the wallet is loading. In the latter, we would stop loading the wallet and ask user's permission to compress their UTXOs.

About hardware wallet, I don't think the flow would be a little different because the user would have to accept our suggestion of compressing UTXOs and then would have to approve using their hardware wallet.

I agree that UTXO compression may be misleading. What about UTXO Squeeze? Or UTXO Shrink? I prefer UTXO Squeeze.

[obiyankenobi]

I think aggregation, as Jan suggested, is better. Or consolidation. Squeeze sounds weird to me.

[jansegre]

I like "consolidation". It literally means what we want: combining many UTXOs into a single UTXO. While "aggregation" means bundling them together (but not necessarily in the sense of replacing many by one), and "compression" means making it smaller/lighter (we aren't making individual UTXOs smaller).

[obiyankenobi]

Making a link with the PR about improving wallet loading time (https://github.com/HathorNetwork/hathor-core/issues/54), we return the complete utxo list at once, so there's no intermediate step when we can stop and ask the user about consolidation. It would be only in the end, I guess.

[pedroferreira1]

I am still think about how much faster this would load with this compression. If I understood correctly, this would not be replacing the history, it would be replacing the utxo_list, right?

Thinking about the design here: https://github.com/HathorNetwork/hathor-core/issues/54.

We would load

utxo_list: [
    {tx_id1, index1, is_authority1, value1},
    {tx_id2, index2, is_authority2, value2},
    ....
]

With this aggregation (supposing that all outputs are to the same address, then we would aggregate to only one 'tx'), we would have

utxo_list: [
    {
        inputs: [
            {tx_id1, index1},
            {tx_id2, index2}
       ],
       output: {address, sum_value}
    }
]

I feel that, if we want to use this aggregation data to send transactions also, we still need the tx_id and index of each utxo (to calculate the tx inputs) and each of the utxo values also (to calculate a change output, if needed). So I don't know how we would do it without those informations.

[obiyankenobi]

@pedroferreira1 the second utxo_list object doesn't seem right. There shouldn't be any inputs or outputs in it.

But I think I understand your point. For the cases we've been having problem right now (miners with lots of txs to the same address) we would still have the same problem. I guess the difference it that, after loading it once, they'd have the option to compress utxos so next time it'll be better.

[msbrogli]

I just renamed from Compression to Consolidation.

[msbrogli]

Making a link with the PR about improving wallet loading time (HathorNetwork/hathor-core#54), we return the complete utxo list at once, so there's no intermediate step when we can stop and ask the user about consolidation. It would be only in the end, I guess.

We can change the design to paginate the list of utxo. So, the wallet can decide when to stop requesting the next pages and suggest a consolidation.

[msbrogli]

I don't understand why it wouldn't solve the problems we've seen so far. As far as I understand, the problems are related to loading thousands of transactions in memory to consider that the wallet is loaded. If the UTXOs are consolidated in just a few outputs, we can load only those outputs and the first page of each address, reducing the number of requests and speeding up the overall load process. It would also require less memory to store all the data.

Let's say a user has 15,000 UTXOs. Using consolidated transactions with 100 inputs, we can quickly reduce the UTXOs to 150 in one step (or to 2 UTXOs in two steps). After that, the wallet would load after requesting the first page of the transaction history and these UTXOs. The remaining history wouldn't be loaded until the user clicks on the next pages of the history.

[pedroferreira1]

I misunderstood, I think. This consolidation would create new transactions in the network. At first I thought those "new transactions" would be created only in the wallet storage as an aggregation. Just to "compress" locally the user data. I guess the name compression confused me a little bit.

But yeah, now I understand and everything makes sense.

[obiyankenobi]

We cannot use only one transaction because we may expose the user's real balance.

That might already be the case if there are multiple consolidation txs on the blockchain, as the txs would happen almost at the same time. At least now, that there are not many txs going on, it's most likely these txs belong to the same wallet.

[pedroferreira1]

We just need to decide when we will execute the utxo consolidation. If we will require user approval and how. But for me it's good and approved the design.

[jansegre]

We cannot use only one transaction because we may expose the user's real balance.

That might already be the case if there are multiple consolidation txs on the blockchain, as the txs would happen almost at the same time. At least now, that there are not many txs going on, it's most likely these txs belong to the same wallet.

I agree that timing can be used to expose some privacy, but I'd rather we start with this and try to improve later, than use this as an excuse to make an even greater consolidation (one tx per wallet). I know that's not what you meant though.

[jansegre]

In general I'm on board with this. But like @pedroferreira1 said, I think we could flash out the UX details a little more.

I'd prefer user approval on every consolidation possibly with an option to always allow tx where the all input and output addresses are the same (and there are no time locks), so that could be a possible option for automatic silent consolidations.

[obiyankenobi]

I agree that timing can be used to expose some privacy, but I'd rather we start with this and try to improve later, than use this as an excuse to make an even greater consolidation (one tx per wallet). I know that's not what you meant though.

I agree, let's start with this and improve later.

[msbrogli]

I just updated with your considerations, including comments about UX and the timing issue. Can you check again, please? @obiyankenobi @jansegre @pedroferreira1