Recovery from seed not working

[steepdawn974]

Moving the seed of any existing node with balance and channels to a different machine does not restore the channels, or the balance

Steps to reproduce:

1. Initialize a phoenixd node on server1, and back up the seed word from ~/.phoenix/seed.dat
2. Fund the node with enough sats to open a channel. getbalance and listchannels must return non-zero/null results
3. Shut down phoenixd node on server1
4. On server2, create ~/.phoenix/seed.dat . Populate it with the seed words from step 1: word1 word2 wordN
5. Start phoenixd node on server2 --> The original NodeId is restored so the seed is picked up correctly, but the balance is empty, and node has no channels.

./phoenix-cli getbalance
{
    "balanceSat": 0,
    "feeCreditSat": 0
}

./phoenix-cli listchannels
[
]

EDIT: Version 0.2.0-6eb0940

[pm47]

Cannot reproduce with a simpler variant where I just manually delete db files from ~/.phoenix and restart.

You're stopping server1 before starting server2 right? Please send server2's logs to phoenix@acinq.co for investigation.

[steepdawn974]

Logs sent!

Yes, server1 is stopped for sure.

In fact, originally I ran into this problem when trying to update a docker container to the latest 0.2.0 , and orig container was stopped& then restarted.

[steepdawn974]

Resolved. Looks like a bad user backup

[steepdawn974]

Reopening this issue, having analyzed this further.

1) the seed.dat file does not only consider the twelve BIP-39 seed words, but produces different nodeId is there are additional characters in there. In fact, anything can be put into this file and phoenixd still derives a nodeId from it. That's a bug; it should validate the content of seed.datfor valid BIP-39 seed word, and fail otherwise.

2) The content of seed.dat that phoenixd intially creates seems to contain some extra character beyond the visible twelve words. And thus when you try to restore a node by creating an empty seed.dat file and typing in the backed up twelve words, you get a different nodeId. But if you kept a physical copy of the original seed.dat file, and restored from there, it works.

Here's what I've done:

1. Initialize a fresh phoenixd, and generate a new seed. Take note of the nodeId.
2. Open ~/.phoenix/seed.dat in an editor. Ctrl+A, Ctrl+C to copy the content of seed.dat to clipboard
3. Shutdown phoenixd, and clear seed & db files: rm -f ~/.phoenix/*
4. Create an new empty seed.dat , and paste the content from clipboard: nano ~/.phoenix/seed.dat (then paste content and save)
5. Start phoenixd --> it will recover a different nodeId!!

Steps

./phoenixd --agree-to-terms-of-service
Generating new seed...done
Generating default api password...done
Generating webhook secret...done
2024-07-14 21:50:50 datadir: /home/myuser/.phoenix
2024-07-14 21:50:50 chain: Mainnet
2024-07-14 21:50:50 autoLiquidity: 2000000 sat
2024-07-14 21:50:50 nodeid: 0246c3...
2024-07-14 21:50:50 offer: lno1zrxq8...
2024-07-14 21:50:50 connecting to lightning peer...
2024-07-14 21:50:50 connected to lightning peer
2024-07-14 21:50:50 listening on http://127.0.0.1:9740

#now copy the seed

^C2024-07-14 21:51:14 shutting down...
^C^C2024-07-14 21:51:15 http server stopped

 ~/Downloads/test  rm -f ~/.phoenix/*                                                                                                                                                                                   sh: sure you want to delete all 6 files in /home/myuser/.phoenix [yn]? y
 ~/Downloads/test  nano  ~/.phoenix/seed.dat    
# paste the seed and save file

~/Downloads/test  ./phoenixd --agree-to-terms-of-service                                                                                                                                                                
Generating default api password...done
Generating webhook secret...done
2024-07-14 21:51:57 datadir: /home/myuser/.phoenix
2024-07-14 21:51:57 chain: Mainnet
2024-07-14 21:51:57 autoLiquidity: 2000000 sat
2024-07-14 21:51:57 nodeid: 02d1f1....
2024-07-14 21:51:57 offer: lno1zrxq...
2024-07-14 21:51:57 connecting to lightning peer...
2024-07-14 21:51:57 connected to lightning peer
2024-07-14 21:51:57 listening on http://127.0.0.1:9740

Test seed.dat with some garbage

phoenixd still starts . This should fail, or at least issue a warning.

~/Downloads/test  cat ~/.phoenix/seed.dat
I AM NOT A BIP-39 SEED!!!

~/Downloads/test  ./phoenixd --agree-to-terms-of-service                                                                                                                                                                       2024-07-14 22:04:47 datadir: /home/myuser/.phoenix
2024-07-14 22:04:47 chain: Mainnet
2024-07-14 22:04:47 autoLiquidity: 2000000 sat
2024-07-14 22:04:47 nodeid: 021ab2a6128c4c0a6e8677476d62e6880f02834ede7bb4b34f6410da76017b6d99
2024-07-14 22:04:47 offer: lno1zrxq8pjw7qjlm68mtp7e3yvxee4y5xrgjhhyf2fxhlphpckrvevh50u0q0jnfev43d0fenlvq2g7fhsrr47ztz2hvw8q5f5eeddf7nn3g9krcqsza2wqzvyefkl65qhwlnydd5pm0yzgpnzt7742htfyl3vkpu352rgqqvagn2xc7dr9rh8ckyhnx7atgpt5f6ztspa0rty7uu07n2a49la775xds5swt25tzn336dnuqufu0xyalav7qw88x6vslq0s8c9va3g35dxk2qw8y83wk9wlcnw28mefkt4v0fy7wqqs4y2d836j3r3yc5ymy8s6m7lmpu
2024-07-14 22:04:47 connecting to lightning peer...
2024-07-14 22:04:47 connected to lightning peer
2024-07-14 22:04:47 listening on http://127.0.0.1:9740

[pm47]

Thanks for the detailed report!

Point 1. is addressed in #73.

I wasn't able to reproduce 2. though. Phoenixd doesn't add any special characters, line breaks or anything. Just 12 words, space-separated. edit: Most likely you added a line break when editing with nano, will add trimming.

[steepdawn974]

I couldn't tell for sure if it adds anything, but the original seed.dat file phoenixd definitely derives a different nodeId than when I create an empty  seed.dat file myself and type in the twelve words (which is what you'd expect from a user trying to restore his wallet).

Try to do a cat ~/.phoenix/seed.dat on a phoenixd-generated file, and it'll show a '%'  at the end. I dont know what that is, but I suspect that this the reason it restores to a different nodeId.

[pm47]

This is because zsh automatically adds a newline character if there is none (https://unix.stackexchange.com/questions/167582/why-zsh-ends-a-line-with-a-highlighted-percent-symbol).

[steepdawn974]

Even when I cat the content from seed.dat into an env var, and then echo it back into seed.dat , this results into a different nodeId. Although it must be a 1:1 copy of the content.

[pm47]

This should all be fixed in #73. Are you able to build yourself to verify? Otherwise I can attach a binary

[steepdawn974]

This should all be fixed in #73. Are you able to build yourself to verify? Otherwise I can attach a binary

I can build it myself. But first, will also test if I can/can't reproduce this on Ubuntu/bash.

[steepdawn974]

@pm47 Could you attach a binary after all? somethings wrong my local build env, and I don't have the nerve to troubleshoot it now..

[pm47]

For macos right?

[steepdawn974]

No, x86_64 linux

But I got something to work in a Docker container in the meantime and built off e480167ebd060e961754c212fdfebb3550176c5d

When I edit the seed.dat file (after it got created by the intial start of phoenixd) , it always complains that the last word is not a valid mnemonic word:

5362d6f44847:~/bin$ vi /phoenix/.phoenix/seed.dat
5362d6f44847:~/bin$ ./phoenixd --agree-to-terms-of-service
Usage: phoenixd [<options>]

Error: invalid mnemonic word daughter

[pm47]

What's the output of hexdump -C .phoenix/seed.dat ?

[pm47]

Wait you are missing two commits , you need 194cb2b

[steepdawn974]

Wait you are missing two commits , you need 194cb2b

Yeah that was it.. copied the wrong commit hash.

Now everything is working. I was even able to restore my original node with the funds :partying_face: