An option to send unspent amount to the same origin address

[vicepool]

I am writing a service that will make small transactions from the same wallet/address using cardano-wallet api. Initially, I have funds only on the one address. But once I make a small transaction, the unspent amount is sent to the next unused address. So after 10k TXs cardano-wallet is monitoring 10k addresses for incoming TXs. That slows it down and makes it impossible to use after the TX number grows.

It would be a useful feature if you could add a non-required option like unspent: "origin" that will send the unspent amount to the same origin address instead of sending it to the next unused address.

The same problem happens with the pool's pledge. Once you send any small amount from your pledge address, all funds will move to another unused address, and the pledge will not be met. It leads to huge problems with missing the whole epoch rewards once you forget about that behavior and do not make an additional TX for all wallet funds to the original address.

[olegandreyev]

Missed epoch rewards as well (248), after registering pledge wallet in voting (actual transaction). As a temporary solution I sent all funds back to the original address.

[KtorZ]

Thanks for your feedback on this. This is certainly something we've been considering for a while but hasn't got any major priority yet.

That slows it down and makes it impossible to use after the TX number grows.

Do you have any numbers backing this claim?

[vicepool]

Do you have any numbers backing this claim?

Yes. I do. I will attach some screenshots and errors I am getting performing the same actions on a wallet with used 6.3k addresses vs 50.

At the moment I was making those screenshots, cardano-wallet was not sending or making any tx, so the comparison should be correct. I left it just running without performing any actions so Load average could be compared.

This is the server load with 50 addresses:

Here the same server, same wallet, idle with 6.3k used addresses.

Compare "Load average". And it's just 6.3k addresses, not like 50k or 100k. I will do the testing with it, but there are some serious problems with cardano-wallet that do not allow me to get to that number.

Ok. Next, let's take a look at the latest Daedalus that used cardano-wallet.

50 addresses: and it's memory consumption:

And now 6.3k addresses: and the memory used 😱

😱😱😱😱😱

Now let's move to another problems that makes cardano-wallet unstable and unusable... I got to 6.3k used addresses by running tiny transactions to the same 1st address. I sent api requests in loop with the same size, but different metadata, so I could distinguish TXs. API server was working quite stable but when it got to ~5-6k used addresses it started throwing errors out of nothing like:

    [code] => not_enough_money
    [message] => I can't process this payment as there are not enough funds available in the wallet. I am only missing: coin: 1.000000 tokens: []

although the wallet had a couple of thousand ADA on it.

And:

    [code] => cannot_cover_fee
    [message] => I am unable to finalize the transaction, as there is not enough ada I can use to pay for fees, or to satisfy the minimum ada quantities of change outputs. I need about pretty 1.171089 ada to proceed; try increasing your wallet balance as such, or try sending a different, smaller payment.

And even:

    [code] => created_invalid_transaction
    [message] => That's embarrassing. It looks like I've created an invalid transaction that could not be parsed by the node. Here's an error message that may help with debugging: HardForkApplyTxErrFromEra S (S (S (Z (WrapApplyTxErr {unwrapApplyTxErr = ApplyTxError [LedgerFailure (UtxowFailure (UtxoFailure (BadInputsUTxO (fromList [TxInCompact (TxId {_unTxId = "e319e62b3e1419b870cbd4edf37135ca91325149b6463512105b7a2ce0dabe67"}) 1])))),LedgerFailure (UtxowFailure (UtxoFailure (ValueNotConservedUTxO (Value 2000000 (fromList [])) (Value 1995196917 (fromList [])))))]}))))

The more addresses were used, the more often it was throwing such errors and when I finished testing getting to 6.3k used addresses, it was around 90% of time that API server threw such errors instead of making a TX. Before that, going up to 5k I have almost 0% rate of any problems.

Cardano-wallet started having problems like: Feb 26 16:54:56 web bash[252081]: SQLite3 returned ErrorConstraint while attempting to perform step: UNIQUE constraint failed: seq_state_pending.wallet_id, seq_state_pending.pending_ix and Feb 26 16:55:57 web bash[252081]: [cardano-wallet.api-server:Error:47810] [2021-02-26 13:55:57.51 UTC] [RequestId 24520] 500 Internal Server Error in 1.975678921s

And that was only 6.3k used addresses, not like 50k or 100k. I guess it will just die on such numbers. I will try to get there, but it's not running smoothly already and fails.

So I guess the issue is there and using new addresses for the unspent amount may be an illusory choice for keeping privacy (it's really not because it can all be tracked back to the original address) but it's making more problems than any good.

I really hope that this post will receive more attention from contributors and an option for using less addresses will be implemented.

PS: cardano-wallet is still my choice to work with, although atm I will probably have to switch wallets occasionally with moving all funds to a new a wallet and deleting the old one. Not a good solution, but at least something.

[vicepool]

Seed to this testnet wallet is: journey combine topple sand rescue tray knife title differ flock essence brown slush hedgehog outdoor verify pistol bulb year color tenant track orange glad

Only rare tries to make a TX will go through. cardano-wallet even thinks that it has 0 balance most of the time and fails with not_enough_money a lot. While testing it I also detected another bugs related to Daedalus. I will open an issue there.

[KtorZ]

@vicepool thanks for the detailed reports, that is useful. Some numbers above aren't right. I do have a few extra questions to understand the problem a bit better.

• I reckon your wallet is a Shelley wallet (i.e. created via the /v2/wallets endpoint) ?

• You say you've been making "tiny transactions". At what pace? Why am I asking this is because, in order to make transactions, you need to have available UTxO on your wallet. When you send a transaction, its associated UTxOs are locked until the transaction is inserted and seen in the ledger. So, even if your total balance is large, your available may not be, resulting in the error you've reported above.

• From the error message above, I can tell that you haven't been using any of the two latest releases. So, what release are you using?

using new addresses for the unspent amount may be an illusory choice for keeping privacy (it's really not because it can all be tracked back to the original address) but it's making more problems than any good.

Yes. Unfortunately, the current scheme for delegation makes the whole privacy argument pointless. However, using new addresses shouldn't be a major problem for most users. We've exchanges with 100k+ addresses in their wallet working correctly. Although they're using a different type of wallet, I am a bit surprised about your report but be sure we'll address it nonetheless.

[KtorZ]

@vicepool I've just tried restoring your wallet. It's doing quite fine on my side. The wallet contains ~6.7K addresses, ~16.8K transactions. The CPU usage once restored is basically close to 0 and constant and it demands about 1.3GB of RAM (that sounds a little bit high, but nothing quite alarming at this stage).

I've tried estimating fees and sending a transaction, it went through pretty much instantaneously. May I ask an extra question:

• How did you start the wallet (what command, options and runtime flags) ?

[vicepool]

* How did you start the wallet (what command, options and runtime flags) ?

It runs as a daemon with: cardano-wallet serve --testnet /home/cardano/cardano-node/testnet-byron-genesis.json --node-socket /home/cardano/cardano-node/db/socket --database /home/cardano/cardano-node/wallets --log-level notice