Haveno's bundled build of monero-wallet-rpc on ARM64 is incompatible with official builds

[haveno-user]

(Original issue title: Unable to mark Payment Sent in Pay-By-Mail trades)

System: Linux ARM64, using the workaround of connecting to an already running system tor process. Everything else is working fine so far.

I currently have two Pay-By-Mail trades I've created, one initiated on version 1.0.5 and one on 1.0.6, and both are running into the same problem. When I try to mark them paid, it fails with,

May-31 01:37:55.239 [Thread-20946] ERROR haveno.core.trade.protocol.TradeProtocol: An error occurred at task: BuyerPreparePaymentSentMessage
Exception message: Error calling import_multisig: Multisig info is for a different account

The specific error message comes from https://github.com/monero-project/monero/blob/master/src/wallet/wallet2.cpp

    THROW_WALLET_EXCEPTION_IF(public_spend_key != keys.m_spend_public_key || public_view_key != keys.m_view_public_key,
        error::wallet_internal_error, "Multisig info is for a different account");

In Haveno's Funds > Transactions pane, the Multisig deposits for both trades show as Sent to: unavailable https://github.com/haveno-dex/haveno/blob/master/desktop/src/main/java/haveno/desktop/main/funds/transactions/TransactionsListItem.java

            MoneroOutgoingTransfer transfer = (MoneroOutgoingTransfer) tx.getTransfers().get(0);
            if (transfer.getDestinations() != null) addressString = transfer.getDestinations().get(0).getAddress();
            else addressString = "unavailable";

So it looks like my client somehow lost the multisig addresses associated with the trades. But I can find the multisig addresses in haveno.log from when the trades were initially created, and there were no errors logged at the time.

I already sent the payment parcel for one of the two trades. I have the other one packaged and ready to go, and I'd like to be able to complete both trades. Is it going to be feasible to just open a dispute at the end and have the arbitrator manually pay out the XMR with the counterparty's agreement, or would more fixing up be needed to get the trades back into a valid state?

[woodser]

Hm I haven't seen the "Multisig info is for a different account" error before.

Perhaps your multisig wallet cache files were moved, deleted, or corrupted somehow? Unlikely without manually tinkering with the application folder.

Sent to: unavailable is expected now, though it will indicate "Multisig deposit" in the Details column.

Do you mind sharing your logs here or DM?

Is it going to be feasible to just open a dispute at the end and have the arbitrator manually pay out the XMR with the counterparty's agreement

Yes it would normally, but I don't know if this error will interfere.

[woodser]

Did you encrypt your Haveno instance with a password by chance?

[haveno-user]

I did set a password on the Haveno wallet. I just tried removing it, it didn't change the error.

I'm pretty sure I didn't tinker with the trade files in the wallet directory.

I did have two payment accounts differing only in case - Pay By Mail: Anon and Pay By Mail: anon. If anything in the java code can grab accounts in a case-insensitive fashion, that might have confused it.

My haveno.log file is currently 20,387 lines. Is there a subset of it that I could post or send? That's a lot to check for anything potentially sensitive.

So far I've had this happen on 2 of 2 trades initiated. I'm going to try to reproduce it on 1.0.6 stagenet. If I can get it to happen there, then I can post a minimum log file from startup through the bug occurring, and also test what happens when it goes to arbitration.

[woodser]

I haven't been able to recreate the error under a variety of circumstances by applying a password.

I'd look for the error "Multisig info is for a different account" in your logs and include as much preceding text about the creation of the trade as you can, denoted by its trade ID.

You could also look for any weirdness around the log "Done changing all wallet passwords" from when you applied the password. Otherwise I'd just be eyeballing for any other errors.

That it happened to both trades suggests a general state the application is in, or bug when the password was applied. Happy to debug further with more logs or even sharing your application instance when you're done.

[haveno-user]

We've taken it to arbitration, update from the arbitrator:

Jun 3, 2024 11:07:52 PM (from Agent) Hello, thank you for the thorough report. The maker has already asked me to release the funds. I am also running into issues when trying to release them. I will now look into the emergency multisig tool that I supposedly have access to.

Jun 3, 2024 11:17:36 PM (from Agent) It turns out that the emergency multisig tool does exactly nothing.

Jun 3, 2024 11:25:23 PM (from Agent) I am afraid that I am currently unable to help you resolve this issue. I hope you'll understand that critical bugs are a bit above my paygrade and I would ask you to please bring this issue up with woodser. Usually I would be around to try out any possible solutions but I will actually not be available for a period of about 5 days starting 24 hours from now. I really hope this isn't too big of an inconvenience to you.

Jun 3, 2024 11:26:38 PM (from Agent) In any case I believe that we should be able to recover the funds, even if that takes manually importing the multisig into the monero CLI and getting them out that way.

I am able to reproduce the issue on stagenet. It does not require multiple payment accounts or password protecting the wallet. It does show up even earlier than attempting payment sent. I will be preparing stagenet logs.

[woodser]

I will be preparing stagenet logs.

Great, or if you can provide a set of steps to reproduce, that would be ideal.

[haveno-user]

For me, the step to reproduce is to take a trade, it happens every time.

monero-wallet-rpc.log haveno.log

[woodser]

I don't see anything wrong in the logs until the error occurs. Can you please archive and share the whole application directory? Then I can debug the wallet in detail.

It would also be good to know the exact error the arbitrator is encountering. You can open a dispute on the stagenet trade and I should be able to see it as the arbitrator.

[woodser]

Linux ARM64 isn't officially supported. Running on Ubuntu ARM64, I get the error:

Jun-04 10:38:26.044 [ERR] ERROR org.berndpruenster.netlayer.tor.Tor: /home/woodser/.local/share/Haveno/xmr_stagenet/tor/tor: 4: Syntax error: Unterminated quoted string 
Jun-04 10:38:26.044 [StartTor] ERROR h.network.p2p.network.TorNetworkNode: Starting tor node failed org.berndpruenster.netlayer.tor.TorCtlException: Could not setup Tor
    at org.berndpruenster.netlayer.tor.NativeTor.<init>(NativeTor.kt:106)
    at org.berndpruenster.netlayer.tor.NativeTor.<init>(NativeTor.kt:55)
    at org.berndpruenster.netlayer.tor.NativeTor.<init>(NativeTor.kt)
    at haveno.network.p2p.network.NewTor.getTor(NewTor.java:107)
    at haveno.network.p2p.network.TorNetworkNode.lambda$createTorAndHiddenService$10(TorNetworkNode.java:172)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: java.io.IOException: Tor exited with value 2
    at org.berndpruenster.netlayer.tor.TorContext.installAndStartTorOp(TorContext.kt:392)
    at org.berndpruenster.netlayer.tor.NativeTor.<init>(NativeTor.kt:66)
    ... 8 common frames omitted
Caused by: java.io.IOException: Tor exited with value 2
    at org.berndpruenster.netlayer.tor.TorContext.installAndStartTorOp(TorContext.kt:354)
    ... 9 common frames omitted

org.berndpruenster.netlayer.tor.TorCtlException: Could not setup Tor
    at org.berndpruenster.netlayer.tor.NativeTor.<init>(NativeTor.kt:106)
    at org.berndpruenster.netlayer.tor.NativeTor.<init>(NativeTor.kt:55)
    at org.berndpruenster.netlayer.tor.NativeTor.<init>(NativeTor.kt)
    at haveno.network.p2p.network.NewTor.getTor(NewTor.java:107)
    at haveno.network.p2p.network.TorNetworkNode.lambda$createTorAndHiddenService$10(TorNetworkNode.java:172)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: java.io.IOException: Tor exited with value 2
    at org.berndpruenster.netlayer.tor.TorContext.installAndStartTorOp(TorContext.kt:392)
    at org.berndpruenster.netlayer.tor.NativeTor.<init>(NativeTor.kt:66)
    ... 8 common frames omitted
Caused by: java.io.IOException: Tor exited with value 2
    at org.berndpruenster.netlayer.tor.TorContext.installAndStartTorOp(TorContext.kt:354)
    ... 9 common frames omitted

Trades are working for me locally, but I'm wondering if it's related to the OS environment.

[haveno-user]

Quite possibly the steps to reproduce would be to install Manjaro or Arch on an SD card (or other available storage, but that's commonly used for trying out other distros on laptops), boot from that, then build and run haveno under that. I've seen other projects that worked on Debian derivatives but had bugs that only occurred on Arch derivatives.

Getting tor to work is straightforward. After installing tor via the package manager, you need to add to your torrc,

ControlPort 9051

CookieAuthentication 1
CookieAuthFileGroupReadable 1
DataDirectoryGroupReadable 1
CookieAuthFile /var/lib/tor/control.authcookie

Start or reload the daemon, add the user you will run haveno as to the 'tor' group, log out and back in so that group assignment is active, then run haveno with $ ./haveno-desktop --torControlPort 9051 --torControlCookieFile=/var/lib/tor/control.authcookie --torControlUseSafeCookieAuth appended to the command line.

I have opened a dispute on stagenet so you can see it as arbitrator.

Is there a way I can send the data directory just to you? Even though it's a stagenet wallet, if someone else got into it first that could interfere with debugging. You could run OnionShare in dropbox mode and post the .onion link, and I could upload it to you there.

[woodser]

I'm seeing the same error with the arbitrator: "Multisig info is for a different account".

It seems the multisig wallet is not generated correctly from your environment, which should use the corresponding monero binaries for aarch64.

There may be a bug in the core monero-project binaries which is appearing only in your environment.

If you have the means to do some digging, I'd try creating multisig wallets in your environment, using either the official Monero CLI, or by setting up a local haveno test network and creating trades.

You may uncover a bug that can be reported and fixed in monero-project for other users.

[haveno-user]

OK, so it may be a monero bug rather than a haveno bug. I'll test creating multisig wallets with monero-wallet-cli.

Looks like the address got cut off, .onion service names should be 56 characters before the dot, and I get 'Problem Loading Onionsite'.

[haveno-user]

Now I realize I should have provided a PGP key so you don't have to post the OnionShare information publicly.

Since I have your public key in the project data, here is a PGP message with my own OnionShare address so you can just grab the file from there: https://gist.github.com/haveno-user/a2131c6959a6df8e6464ba3f94449fc8

PGP key for me for future reference

[woodser]

I've decrypted your onion and private key, but getting an error that the onion service can't be reached from the tor browser.

[haveno-user]

The current revision of the gist with the onion address starting with ct- and ending with -id? It works for me. Can you also post your full onionshare address encrypted with my key so we have two possibilities to make it work?

[edit] The file is 228MB. Test downloading projects 3-4 hours to finish over tor. But gpg encrypting and decrypting files of that size turns out to be very quick, so here is an alternate download link: https://gist.github.com/haveno-user/19d8a6708b4be12ca63ba452a834d0c4

[haveno-user]

I have successfully created 2/2 and 2/3 multisig wallets in my environment (Manjaro aarch64) using monero-wallet-cli. No problems with creating wallets, receiving funds, or sending funds. So it's not just a bug in Monero.

For trade multisig wallets like xmr_trade_SEERA6O_c156f57c, if we haven't set a wallet password in Haveno, what should we use as the password when trying to load them in monero-wallet-cli?

monero-wallet-cli warns about using multisig:

Error: Multisig is an experimental feature and may have bugs. Things that could go wrong include: funds sent to a multisig wallet can't be spent at all, can only be spent with the participation of a malicious group member, or can be stolen by a malicious group member.

[haveno751]

Please expedite a fix for this bug. The arbitrator cannot release my funds due to this bug so I cannot open a new trade nor withdraw. Other users could fall into this trap as well.

[woodser]

@haveno-user

if we haven't set a wallet password in Haveno, what should we use as the password when trying to load them in monero-wallet-cli?

Please try "password", which is the default unless you set your own password.

[woodser]

@haveno-user

Thanks for testing with the official CLI. Since that works, it sounds like the issue is isolated to the monero-wallet-rpc binary, which is automatically included with Haveno on your platform.

It would be very helpful if you could confirm, by:

1. Manually replacing the monero-wallet-rpc binary in Haveno's application directory (~/.local/share/Haveno/) with the official binary for your OS.
2. Retrying to create a trade on stagenet to see if the issue still occurs.

Since this issue has the potential to lose money, your debugging efforts are really appreciated. Thanks.

At least, we need to ensure that if the wallet is in an invalid state, it's detected and the trade fails before funds are deposited.

It would be nice if we could direct message for further debugging, and I still need to get ahold of one of those corrupt wallets. Going to try to copy the arbitrator stagenet wallet from VPS.

The current revision of the gist with the onion address starting with ct- and ending with -id?

That's correct.

[haveno-user]

Opening haveno_XMR or the trade wallets in monero-wallet-cli using password as the password fails saying it's an invalid password.

If I replace Haveno's monero-wallet-rpc with the one from the monero-aarch64-linux-gnu-v0.18.3.3 archive and sticky it so it can't be overwritten at Haveno startup, opening the main wallet fails: 2024-06-11 20:21:50.967 [RPC0] ERROR wallet.wallet2 src/wallet/wallet2.cpp:5023 !r. THROW EXCEPTION: error::invalid_password

-1: Failed to open wallet
RPC request: 'open_wallet' with params: {"password":"password","filename":"haveno_XMR"}                                                                                                  
        at monero.common.MoneroRpcConnection.validateRpcResponse(MoneroRpcConnection.java:634)                                                                                           
        at monero.common.MoneroRpcConnection.sendJsonRequest(MoneroRpcConnection.java:393)                                                                                               
        at monero.common.MoneroRpcConnection.sendJsonRequest(MoneroRpcConnection.java:335)                                                                                                       at monero.wallet.MoneroWalletRpc.openWallet(MoneroWalletRpc.java:300)                                                                                                            
        at haveno.core.xmr.wallet.XmrWalletService.openWalletRpc(XmrWalletService.java:1609)                                                                                             
        at haveno.core.xmr.wallet.XmrWalletService.openWallet(XmrWalletService.java:356)                                                                                                 
        at haveno.core.xmr.wallet.XmrWalletService.maybeInitMainWallet(XmrWalletService.java:1338)                                                                                               at haveno.core.xmr.wallet.XmrWalletService.maybeInitMainWallet(XmrWalletService.java:1320)                                                                                       
        at haveno.core.xmr.wallet.XmrWalletService.lambda$initMainWalletIfConnected$37(XmrWalletService.java:1311)                                                                       
        at haveno.common.ThreadUtils.lambda$execute$0(ThreadUtils.java:50)                                                                                                                       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) 
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Possibly there is some difference in the password hashing functions linked into the two binaries so they can only open wallets they've created.

I thought you'd already gotten the data directory. I've encrypted it and uploaded it to a file hosting service. Link: https://gist.github.com/haveno-user/3ffcd2c6394a01a0b81bcc8fe39f21d0

[haveno-user]

I made a new data directory, haveno-XMR_STAGENET_user2a, that uses the official monero-wallet-rpc binary from the start so it can read its own wallet.

Now I'm back to the arbitrator being unavailable on stagenet so I can't create offers.

Jun-11 22:38:35.319 [JavaFX Application Thread] WARN h.c.o.p.tasks.MakerSendSignOfferRequest: Arbitrator unavailable: address=ywoag7j4ot4kpmtiv274y34vqbkrhsvi3bdrb2j2vxcjzxkolsbziwad.onion:7777, error=java.util.concurrent.TimeoutException

This is with user1 which is still using the bundled monero-wallet-rpc.

[woodser]

Sorry just resumed the stagenet arbitrator. I'd shut it down earlier in order to transfer locally and analyze the broken wallet.

I'm working on a PR which adds logging while the multisig is created, in order to debug further. There's a small chance the issue will resolve itself with the PR, in case I previously created the ARM64 binaries incorrectly. But otherwise, we'll be able to see where it diverges from expected.

Will follow up shortly.

In the meantime it's helpful to confirm that the official binaries work as expected.

[haveno-user]

Thanks. Currently have user1 (maker) running the bundled monero-wallet-rpc, and user2a (taker) running the official. This combination is producing the familiar error in both consoles. Jun-11 23:59:15.152 [pool-14-thread-1] WARN haveno.core.trade.Trade: Failed to import multisig hex, attempt=5/5, tradeId=numzftgu, error=Error calling import_multisig: Multisig info is for a different account

Will next try with user1a X user2a, maker also running the official binary.

[woodser]

Ok the PR is ready for testing: https://github.com/haveno-dex/haveno/pull/1028

It adds trace logging when multisig wallets are created, in order to see the seed, public keys, and exchanged hex at each step.

I've updated the stagenet arbitrator and posted offers from a client which also has the changes applied, so we'll be able to see the debug info for every instance.

Please first apply the PR, e.g.:

git fetch origin
git pull --rebase origin pull/1028/head
./gradlew build --refresh-keys --refresh-dependencies -x test -x checkstyleMain -x checkstyleTest
make haveno-desktop-stagenet

Then take one of my trades on the stagenet network:

And finally pass along your log file so I can see the debug output. I'll have these offers posted for the next 12 hours or so before traveling.

Thanks again for your help!

[haveno-user]

I'll try that in a minute. I just finished testing combinations of monero-wallet-rpc. When both maker & taker are using the official binary, the trade is successful. Importing multisig hexes succeeds. When either one or both is using the bundled binary, the trade fails as described.

[haveno-user]

Client built. Trade accepted. I see the trace logging in the console, and the import multisig error occurring. I can't test marking payment sent because you created maker-as-xmr-buyer trades, so you will need to try clicking that in your client.

haveno.log monero-wallet-rpc.log RPC error message.txt

The detailed dump of the RPC request in the third file is something that is printed to the console, but not saved to either log file, at least at the default log level.

[woodser]

Thanks for testing. The logs confirm that all wallets have correct public keys after initial creation.

I've added more logs to isolate when your wallet state changes with invalid keys. Maybe it does have something to do with encrypting and saving the wallet?

If you can please apply new updates, e.g.:

git fetch origin
git reset --hard origin/master
git pull --rebase origin pull/1028/head
make haveno-apps
make haveno-desktop-stagenet

Then you can take one of my offers or one of your own. No need to be the buyer, since we should see the problem before confirming payment sent or received.

[haveno-user]

user1 as maker, user2 as taker, both using bundled monero-wallet-rpc. user1.txt user2.txt

[haveno751]

Please continue to expedite this issue. My funds are still locked. This can still trap other users. Thanks to haveno-user and @woodser for your continued efforts.

[haveno-user]

https://www.getmonero.org/resources/developer-guides/wallet-rpc.html

It's possible to talk to monero-wallet-rpc from the console using curl. I am able to load one of the wonky stagenet trade wallets, poke at it interactively, export the multisig seed, and restore from seed in monero-wallet-cli for more convenient testing.

So far, either program shows it's stuck at the stage where it's just received the two deposits for the bulk of the trade plus the security deposit, and needs to complete a round of multisig info exchange to finish computing the key images.

I suppose my next step is to run a full local testnet so I can be the arbitrator too and manually exchange multisig info between all three wallets.

[woodser]

I suspect there's an issue with the monero-wallet-rpc binary which I generated from Ubuntu 22.04.2 ARM64, where export_multisig_info exports with different public keys for some reason.

In this case there can be several fixes:

1. Generate the monero-wallet-rpc binary from Ubuntu 20.04 in order to fix the root cause.
2. The arbitrator and other peer should be able to resolve the dispute manually, being tolerant of a failure to update from the broken peer. I'll create a patch within a few days which the arbitrator and other peer can apply.
3. Fix whatever is broken when generating this binary from Ubuntu 22.04.2.

I'm not seeing a way to detect the broken state before funds are deposited, since the key mismatch appears after the first confirmation, on export_multisig_info.

[haveno-user]

That the bundled binary produces wallets which are incompatible with the standard binaries is also a bug, and it might be the same bug.

I ran a full testnet. When all three copies of monero-wallet-rpc (maker, taker, and arbitrator) are the bundled binary, the multisig exchange works, and the trade can be completed. It's just any combination of the two builds that doesn't work.

I also successfully repaired & completed one of the broken trades on stagenet. I used the variant monero-wallet-rpc to dump the seeds for the main wallet and trade wallet, restored them from seed in monero-wallet-cli, then switched to using the official binary with that node, and the trade could be completed. I'll try fixing my two mainnet trades next.

[haveno-user]

Looks like that fixed my mainnet trades. I am able to mark them payment sent, trading partner on the first trade marked it payment received, and that trade is completed.

It was necessary to use a local xmr node. Marking payment sent when using remote nodes over tor still always timed out or errored out.

Haveno apparently doesn't recognize private IP addresses and tries to connect to them with tor, which causes a SOCKS error on startup. To use a local node on your LAN, it's necessary to use both the --xmrNode="http://" option and --useTorForXmr=OFF .

It would be more user friendly if it did recognize private address ranges.

[haveno-user]

Unlike Localmonero, the trade chat can't be used to keep in touch after trades complete, so here is a PGP message for the counterparty of our already completed trade: https://gist.github.com/haveno-user/bc4e1227623bdc070c97d55c4afcd49d

[woodser]

Glad that you were able to settle the trade.

I was finally able to recreate the error. As suspected, the binaries built for ARM64 were incompatible.

I've implemented two main fixes: https://github.com/haveno-dex/haveno/pull/1045

1) Fixed the ARM64 binaries by building on Ubuntu 20.04 instead of Ubuntu 22.04. 2) The arbitrator is able to resolve the dispute by being tolerant of a peer with corrupted state.

I've also opened an issue on monero-project to hopefully prevent others from creating corrupted binaries: https://github.com/monero-project/monero/issues/9371

Feel free to try it out. Thanks again for your effort and patience to get this fixed!

[woodser]

@haveno-user

Haveno apparently doesn't recognize private IP addresses and tries to connect to them with tor, which causes a SOCKS error on startup. To use a local node on your LAN, it's necessary to use both the --xmrNode="http://address:port" option and --useTorForXmr=OFF .

I opened this PR to fix this issue, which disables tor for private IP addresses: https://github.com/haveno-dex/haveno/pull/1058

Hopefully you can test, since I don't have a node running on non-localhost private IP.

[woodser]

The fix for private IP addresses was tested and confirmed by gavinbarnard here: https://github.com/haveno-dex/haveno/issues/1026#issuecomment-2181871292