Host unlock hash is not updated when wallet seed changes

[MaxMaximus]

Hello.

I have finished my full cycle of testing as a host on sia network (from July to mid of October) and results just awful. It not only have lot of minor to moderate bug and problems (i already reported some of them here) but test ended with such unacceptable critical situation:

• all contract already expired, but host did not get ANY of storage revenues (despite fact it successfully post some storage proofs to blockchain) and also had host all collateral locked in contracts. Not just risked collateral (it can be happening if host fail storage proof part), but all collateral sent to contract.

At first i thought it is a just wallet display bug as reported there: https://github.com/NebulousLabs/Sia/issues/2420 So host actually getting collateral back and this is just not reflected properly as incoming transaction in wallet. But later i set up wallet balance monitoring - and found out that available balance only goes down. There are no additions to balance after contract expire - only additional small deduction for fees each time a next contract expire.

I do not find anything interesting in logs what can lead to situation with full collateral lost. There are some messages about successful storage proofs, some errors with missed storage proofs as described in that issue: https://github.com/NebulousLabs/Sia/issues/1862 (data folder is intact, host online, wallet unlocked, but still somehow miss some of storage proofs). No other major errors.

Here are details: windows host (window 7 64 bit Pro) running siad. ver. 1.22 initially, later host was updated to ver. 1.3.0 via built-in updater (siac update ) . 100 % hosting, never used for renting/uploading files or as finance wallet (i have a separate SIA installation for this on another computer)

Here full wallet detail from test host: wallet-tx.txt As you can see there are only 2 incoming tx (50+350 SC). It was my own tx - loading this test wallet by external transaction (from my other wallet). All other txs were made by host automatically. And there is no any incoming tx despite all contract already ended (last one about a week ago). Final balance: 48 SC from initial 400 SC. So actual storage revenue is a negative value.

Here stats from host (siac host -v outputs i have taken manually from time to time) host-v.txt

Stat shows that all contracted already ended. Note additional minor bug: host stats reports some of Locked/risked Collateral and potential revenues still remaining despite fact that all contract already expired and amount of data stored is zero at last report.

And another one minor bug: "Lost Collateral" and "Transaction Fee Expenses" counter seems broken - they always show zero, despite fact that this host had lost collateral and had paid some tx fees.

Here is a host.log: host.log Some errors at contract formation stage in July (some of them caused by one more issue with siad - i have changed my prices and collateral few times and looks like siad tries to force new setting on already existing contract which lead to errors in host-renter communication):

Error at 9 September - it was my mistake: i forgot to unlock wallet after one of client restarts and host failed storage proofs on 1 or 2 contract (but there were 17 of them, and even if storage proof missing host should lose only risked collateral in proportion to amount of data actually was stored under this contract, not full locked collateral).

Errors in October with Missed storage proof - looks like one more bug. Host was running 24/7 online in October, with port open, wallet unlocked, data folder intact, etc. But somehow still missed some storage proofs (while successfully doing others).

Here is a wallet.log: wallet.log I did not find anything interesting here, but there are more details about wallet tx after update to v. 1.3.0.

I can provide additional log, but think there is nothing useful there: i have checked ALL other logs already and found out only routine records like start/stop event at client restarts and other standard expected entries.

P.S. I saw few similar report on your discord server about collateral not return back after contract expire. There is something in common: windows host, acceptingcontracts set to NO before contract expire. So may be it related to previous issue (acceptingcontracts=NO prevent use of existing contract, not only new): https://github.com/NebulousLabs/Sia/issues/2415

[DavidVorick]

Hi, the collateral will not be returned for a full 24 hours. If you can link to the transaction on the blockchain, we can show the outputs and prove that the balance was in fact returned to the wallet.

[Cinerar]

I found that collateral in fact return to balance, but there are no transactions or whatever that can help you to confirm that. The only fact that helps - balance just increase one day. There are some serious bugs that just make all finance unclear #2144 #2138 and this is definitelly not good. I believe #2420 should clear things out.

[MaxMaximus]

Almost two weeks have passed since the last contract was over. And more than 3 weeks from the first contract expiration. Still do not see any increases in wallet balance. 400 SC at beginning of contract formation in July and 48 SC now after all contracts already expired.

There is no any incoming tx, so i can not provide a link. But all outgoing tx ids/hashes (tx at contract creation time and additional small txs at contracts expiration which looks like storage proofs) already included in wallet output i have posted before in first post: https://github.com/NebulousLabs/Sia/files/1412620/wallet-tx.txt

And here is my host unlock hash "f72f0b8e1d061b493b31ecead7346f084821a7b1e3c92221c431e7d895ce6c2fbc7f85f13d0c". You can scan blockchain for all contracts created with it. For example there is one of contracts(found it by hash above) from my host: https://explorer.siahub.info/hash/26eb90a947cd8a27861564d18f1586def22d6a6e4d4c3f10e82ffba3f81bdc73 Contact conditions shows that my host should get 13.20 SC (if storage proof OK) or 13.04 SC (if host fail proof) after contract expiration. But it did not get nor 13.2 nor 13.04. Where i can find corresponding tx of collateral return from this contact?

It will be nice to check if all correct in the blockchain too. If contracts in the blockchain is OK (very likely), then there must be some bugs with local wallet logic implementation in the current (1.3.0) siad client. May be it has lost corresponding private keys somehow and now can not redeem coins from finished contracts. Or it does not scan blockchain correctly and simply does not "see" that there is some money belonging to it just waiting for use.

But fact is simple - there are no coins returns in host wallet after > 1 week since all contracts has expired. Nor in the wallet tx list nor in the wallet available (spendable) balance. Only outgoing txs and only decreasing of balance.

[DavidVorick]

We could clear this up if we had some way to display the outputs that the wallet can see.

The contract you posted is here: http://explore.sia.tech/hash.html?hash=26eb90a947cd8a27861564d18f1586def22d6a6e4d4c3f10e82ffba3f81bdc73

As you can see, the storage proof was successful, meaning you should be seeing the coins added to your wallet. From what I can tell, those coins have never been spent, so I think the easiest move to make here to help with debugging would be for you to spend all the coins in your wallet (all 48 of them) and send them to a new address.

That should put a transaction into the blockchain spending the 13 coins. It's possible, though unlikely, that the wallet is not properly seeing the file contract payments that it's supposed to see. But I'm pretty sure it does check them correctly.

[MaxMaximus]

So i should send all coins to myself just to a new generated address? And related question - what is an unlock hash (in my example "f72f0b8e1d061b493b31ecead7346f084821a7b1e3c92221c431e7d895ce6c2fbc7f85f13d0c") used by host to form contract with renters? Its format looks like regular sea wallet address. Is it just a standard address for receiving contract payouts? Or is it some other entity with common format?

If it just a regular address for receiving coin transfers - should it be listed in wallet among other addresses? If so - it can be the source of problem: if I ask siad to list all my wallet addresses ("siac wallet addresses" command) - there is no contract unlock hash among them. I see this hash only in host.json

[lukechampine]

@MaxMaximus: "Unlock hash" is the technical term for an address. An address is the hash of a set of "unlock conditions" which specify who can spend the output.

One thing I want to check, just be sure: please run this program to confirm whether your host's unlock hash belongs to your current wallet seed. If it doesn't, that could explain the issue.

[MaxMaximus]

So this hash is just a regular address for receiving coins -exactly same as any other wallet addresses?

That program returns error with any given address: seed failed checksum verification While I am sure that I use correct seed - I have it written down and also have checked it out via "siac wallet seeds" command from host. It shows same seed as my primary.

But i think we on the right track to the root of problem. I have remembered that I re-Initialized wallet once after installing the host (but before the formation of contracts). I set up client, created wallet with default settings. Later I decided to change wallet password (it = primary seed by default, that very inconvenient to use that huge phrase when you need to unlock wallet after each restart). But there is no option to change password on existing wallet in sia client. So I re-initialize wallet from scratch (anyway old/first wallet was empty) with new custom encryption password. And actual bug probably lies there - host might be still using hash from very first wallet seed for contract formation. So "wallet init" and "wallet init-seed" are the first candidates to check - those procedures must change host unlock hash if wallet primary seed changes.

P.S. On another thought: if host uses unlock hash from another seed (which was replaced/deleted) - how does it sign contracts with renters without private key for this unlock hash?

[lukechampine]

that does seem quite possible to me. If you do not have the original wallet seed, you may not be able to recover the coins. The program will confirm this if you are able to get it running successfully. I have tested it myself with newly-generated seeds+addresses, and it works for those. Make sure you pass the address as the only argument, and enter the seed (with no leading or trailing whitespace) when prompted.

I agree that the host needs to switch its unlock hash if the wallet changes. However, I don't know if there's an easy way for the host to detect that this has occurred. We may just have to advise users to do it themselves. @DavidVorick may have thoughts.

As for the P.S. -- the signing key is different from the key that controls the unlock hash. The host generates a separate public/private keypair for this purpose. It announces the public key in the blockchain and uses the private key to sign contracts.

[MaxMaximus]

Good news.

1. Program actually works fine. You was right - error was caused by excessive whitespace. But not leading or trailing (i have checked for it in first place) but one tricky space in the middle - it accidentally split one word into two separate correct words ("uphill" ==> "up hill").

2. Program shows that my current host unlock hash does not belong to current wallet primary seed. So we confirmed why wallet did not see any coins from ended contracts.

3. I have dug up deeply into my daily incremental backup of critical data and found out the initial seed generated at sia setup. Backup shows what I replaced it 2 days later to new current seed before i loaded any coins into wallet and before I announced my host to network. So i just deleted old seed thinking there is no any use for it any more. And your program has found host unlock hash at index 3 for that old seed. So host has been using unlock hash from old deleted seed all along.

4. I have added that old seed to my current wallet as additional (auxiliary) seed ("siac wallet load seed" command) and did a full blockchain re-scan with 2 seeds. And recover all lost funds!. There is still no any incoming transactions in wallet history but available balance now ~413 SC. 400 SC initially loaded to this wallet and about 13 SC as hosting revenues from successfully ended contracts (which even slightly bigger compared with siac host -v stats. but i think there are errors in stats, not in actual transactions )

Looks like @avahowell already proposed one of variant how to track and avoid such problems in future. May be it is overkill to check for it after each block. May be once at each host start would be enough. And at least client build-in functions like "wallet init" and "wallet init-seed" which replace primary wallet seed must also update host unlock hash after replacing seed. If user call one of those functions - client always know and 100% sure that wallet seed was changed.

I used "siac wallet init --force --password" command to replace seed(and wallet encryption password - in ver. 1.2.x there was no option to change password on existing wallet, only recreate new one with custom pass) And I could not even think that the client would continue to use the keys from the old (destroyed --force option) wallet after that. This is clearly an incorrect program behavior.

[tbenz9]

@MaxMaximus This issue has a lot of history and complexity making it a little difficult to understand. I would love to close this issue and open a new issue with a single simple task. Would you be willing to open a new issue with clear steps on how to recreate your host unlock hash is not updated when wallet seed changes bug?