Recan wallet faield

[lvshuang]

My dcrwallet config:

pass=${DCR_PASS}
rpccert=/root/rpc.cert
rpckey=/root/rpc.key
cafile=/root/rpc.cert
rpcconnect=dcr-daemon-01.prod.com:9109
username=DCR_RPC
password=${DCR_PASS}
grpclisten=0.0.0.0:9111
rpclisten=0.0.0.0:9110
gaplimit=1000000

I create 400 addresses, and send 0.02dcr to the Fourth hundred address, the transaction is confirmed on chain, but i can't get transction throgh listtranaction interface.

Then i rescan the wallet with command:

curl --insecure --request POST   --url https://127.0.0.1:9110/   --header 'authorization: Basic base64 password'   --header 'content-type: application/json'   --data '{"jsonrpc": "2.0", "method": "rescanwallet", "params": [],  "id": 1}'

And it return error:

curl: (16) Error in the HTTP2 framing layer
{"jsonrpc":"2.0","result":null,"error":{"code":-4,"message":"wallet.RescanFromHeight:: dcrd.jsonrpc.rescan: -1: Transaction filter must be loaded before rescanning"},"id":1}

I have a look at the dcrd log, i see some error like below

2018-09-05 20:32:11.256 [ERR] RPCS: Websocket receive error from 172.18.0.3:38476: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 20:32:35.031 [ERR] RPCS: Websocket receive error from 172.18.0.5:39414: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 20:32:41.738 [ERR] RPCS: Websocket receive error from 172.18.0.4:50870: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 20:37:38.253 [ERR] RPCS: Websocket receive error from 172.18.0.6:49460: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 20:42:38.560 [ERR] RPCS: Websocket receive error from 172.18.0.6:49620: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 20:47:38.914 [ERR] RPCS: Websocket receive error from 172.18.0.6:49704: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 20:52:39.345 [ERR] RPCS: Websocket receive error from 172.18.0.6:49776: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 20:57:39.620 [ERR] RPCS: Websocket receive error from 172.18.0.6:49842: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:00:17.413 [ERR] RPCS: Websocket receive error from 172.18.0.3:38686: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:02:39.885 [ERR] RPCS: Websocket receive error from 172.18.0.6:49916: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:07:40.221 [ERR] RPCS: Websocket receive error from 172.18.0.6:49998: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:12:40.669 [ERR] RPCS: Websocket receive error from 172.18.0.6:50070: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:17:40.990 [ERR] RPCS: Websocket receive error from 172.18.0.6:50138: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:22:41.334 [ERR] RPCS: Websocket receive error from 172.18.0.6:50210: websocket: close 1006 (abnormal closure): unexpected EOF#015

How can I fix this? Thanks very much!!!

[jrick]

Please provide the wallet logs as well (redacting them of any sensitive info).

[lvshuang]

@jrick The dcrwallet log like below:

2018-09-05 21:00:18.040 [INF] DCRW: Version 1.4.0-pre+dev (Go version go1.11 linux/amd64)#015
 2018-09-05 21:00:18.969 [INF] WLLT: Opened wallet#015
 2018-09-05 21:00:19.940 [INF] WLLT: The wallet has been unlocked without a time limit#015
 2018-09-05 21:00:19.952 [INF] DCRW: Attempting RPC client connection to dcr-daemon-01.prod.com:9109#015
 2018-09-05 21:00:19.952 [INF] DCRW: gRPC server listening on 0.0.0.0:9111#015
 2018-09-05 21:00:19.953 [INF] RPCS: Listening on 0.0.0.0:9110#015
 2018-09-05 21:00:19.994 [INF] SYNC: Established connection to RPC server dcr-daemon-01.prod.com:9109#015
 2018-09-05 21:00:19.995 [INF] SYNC: Headers synced through block 00000000000000007aa227d2c28437682970b0af1d393aaa66e9ed75ea3f5cd1 height 271787#015
 2018-09-05 21:00:19.995 [INF] SYNC: Transactions synced through block 298e5cc3d985bfe7f81dc135f360abe089edd4396b86d2de66b0cef42b21d980 height 0#015
 2018-09-05 21:00:20.025 [INF] SYNC: Fetched 5 new header(s) ending at height 271792 from wss://dcr-daemon-01.prod.com:9109/ws#015
 2018-09-05 21:00:20.388 [INF] SYNC: Connected 5 blocks, new tip block 000000000000000020cccd4ca77464469411198f239ce131e9e6519b464484af, height 271792, date 2018-09-05 20:57:41 +0800 CST#015
 2018-09-05 21:00:20.389 [INF] WLLT: Discovering used accounts#015
 2018-09-05 21:00:22.655 [INF] RPCS: New websocket client 172.18.0.6:55028#015
 2018-09-05 21:01:52.659 [INF] RPCS: RPC method getblockcount invoked by 172.18.0.6:55028#015
 2018-09-05 21:01:53.272 [INF] RPCS: RPC method listtransactions invoked by 172.18.0.6:55028#015
 2018-09-05 21:02:44.006 [INF] RPCS: RPC method rescanwallet invoked by 127.0.0.1:46914#015
 2018-09-05 21:02:44.009 [INF] WLLT: Rescanning block range [1, 1999]...#015
 2018-09-05 21:06:54.910 [WRN] RPCS: Websocket receive failed from client 172.18.0.6:55028: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:06:54.911 [INF] RPCS: Disconnected websocket client 172.18.0.6:55028#015
 2018-09-05 21:06:55.548 [INF] RPCS: New websocket client 172.18.0.6:55124#015
 2018-09-05 21:11:54.993 [INF] RPCS: RPC method rescanwallet invoked by 127.0.0.1:47034#015
 2018-09-05 21:11:54.995 [INF] WLLT: Rescanning block range [1, 1999]...#015
 2018-09-05 21:11:55.898 [WRN] RPCS: Websocket receive failed from client 172.18.0.6:55124: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:11:55.898 [INF] RPCS: Disconnected websocket client 172.18.0.6:55124#015
 2018-09-05 21:11:56.383 [INF] RPCS: New websocket client 172.18.0.6:55192#015
 2018-09-05 21:16:56.982 [WRN] RPCS: Websocket receive failed from client 172.18.0.6:55192: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:16:57.023 [INF] RPCS: Disconnected websocket client 172.18.0.6:55192#015
 2018-09-05 21:16:57.207 [INF] RPCS: New websocket client 172.18.0.6:55258#015
 2018-09-05 21:19:14.264 [INF] RPCS: RPC method rescanwallet invoked by 127.0.0.1:47144#015
 2018-09-05 21:19:14.267 [INF] WLLT: Rescanning block range [1, 1999]...#015
 2018-09-05 21:20:27.435 [INF] RPCS: RPC method rescanwallet invoked by 127.0.0.1:47164#015
 2018-09-05 21:20:27.437 [INF] WLLT: Rescanning block range [1, 1999]...#015
 2018-09-05 21:21:57.748 [WRN] RPCS: Websocket receive failed from client 172.18.0.6:55258: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:21:57.748 [INF] RPCS: Disconnected websocket client 172.18.0.6:55258#015
 2018-09-05 21:21:57.871 [INF] RPCS: New websocket client 172.18.0.6:55334#015
 2018-09-05 21:26:58.347 [WRN] RPCS: Websocket receive failed from client 172.18.0.6:55334: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:26:58.349 [INF] RPCS: Disconnected websocket client 172.18.0.6:55334#015
 2018-09-05 21:26:58.644 [INF] RPCS: New websocket client 172.18.0.6:55408#015
 2018-09-05 21:31:58.965 [WRN] RPCS: Websocket receive failed from client 172.18.0.6:55408: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:31:58.965 [INF] RPCS: Disconnected websocket client 172.18.0.6:55408#015
 2018-09-05 21:31:59.953 [INF] RPCS: New websocket client 172.18.0.6:55474#015
 2018-09-05 21:37:00.426 [WRN] RPCS: Websocket receive failed from client 172.18.0.6:55474: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:37:00.426 [INF] RPCS: Disconnected websocket client 172.18.0.6:55474#015
 2018-09-05 21:37:00.940 [INF] RPCS: New websocket client 172.18.0.6:55540#015
 2018-09-05 21:42:01.437 [WRN] RPCS: Websocket receive failed from client 172.18.0.6:55540: websocket: close 1006 (abnormal closure): unexpected EOF#015
 2018-09-05 21:42:01.438 [INF] RPCS: Disconnected websocket client 172.18.0.6:55540#015
 2018-09-05 21:42:01.969 [INF] RPCS: New websocket client 172.18.0.6:55612#015

[jrick]

What RPC errors are returned, and what is logged by dcrd/wallet, if you make a dcrd RPC such as getchaintips to the wallet?

[lvshuang]

@jrick I make a rpc getchaintips throgh shell command

curl --insecure --request POST   --url https://127.0.0.1:9110/   --header 'authorization: Basic bas64'   --header 'content-type: application/json'   --data '{"jsonrpc": "2.0", "method": "getchaintips", "params": [],  "id": 1}'

And it return

{"jsonrpc":"2.0","result":[{"height":271802,"hash":"00000000000000001270f0336850b8ed696b03d6901daeef8fddf0f6749859e9","branchlen":0,"status":"active"},{"height":271786,"hash":"00000000000000003a93b1e8c0e391e1a8c24df4a17afb078c4a9682278b9d29","branchlen":1,"status":"valid-headers"}],"error":null,"id":1}

There no error

[jrick]

You just leaked your RPC password. Better change that.

Is there another application talking to the wallet besides curl? I'm confused why it's being logged as a new websocket client, since it should only be making HTTP POST calls. edit: that's the client to dcrd, which is correct, so nevermind. Still investigating.

[lvshuang]

@jrick Yes, there has a another application talking to wallet besides curl

[jrick]

Please rebuild dcrd and dcrwallet with the race detector enabled (pass -race to go build or go install) and paste any data races that it outputs when run.

Once new binaries with the race detector are ready, try the test again and check if any data races are being reported by either dcrd or dcrwallet.

Finally, try to sync the wallet without the additional application talking to it and see if it is able to sync then (HTTP POST with curl is still fine).

[lvshuang]

@jrick We run 3 dcrwallet and them connent to the same dcrd. Does this method produce errors?

[jrick]

It shouldn't, but please check using the race detector for anything.

[lvshuang]

@jrick We are exchange, and we need to create unqiue address per time for every user. We should not set the param gaplimit. Keep it default. And pass ignore to create address api when create new address. Is it right? If it is right, some user send dcr to the address which exceed the default gap limit, It seems can get the transaction with listtransaction api.

[jrick]

This:

2018-09-05 21:00:19.995 [INF] SYNC: Transactions synced through block 298e5cc3d985bfe7f81dc135f360abe089edd4396b86d2de66b0cef42b21d980 height 0

is your problem. Your wallet has not synced any transactions, so of course listtransactions won't show anything. It doesn't matter if the address is within or beyond the gap limit.

Have you run the race detector in your environment yet? This is now the third time I've asked you to...

[lvshuang]

@jrick Yes, i runed the race detector. But I got nothing

[jrick]

And have you run the wallet without the other websocket client talking to it?

[lvshuang]

I fixed the rescan return error

{"code":-4,"message":"wallet.RescanFromHeight:: dcrd.jsonrpc.rescan: -1: Transaction filter must be loaded before rescanning"}

It because I set gaplimit 1000000, I get 400 addresses, is less than gaplimit. Only if the addresses count beyond the gap limit, the wallet will call dcrd to load filter.

for {
        if alb.cursor >= gapLimit {
            switch opts.policy {
            case gapPolicyError:
                return nil, errors.E(op, errors.Policy,
                    "generating next address violates the unused address gap limit policy")

            case gapPolicyIgnore:
                // Addresses beyond the last used child + gap limit are not
                // already watched, so this must be done now if the wallet is
                // connected to a consensus RPC server.  Watch addresses in
                // batches of the gap limit at a time to avoid introducing many
                // RPCs from repeated new address calls.
                if alb.cursor%uint32(w.gapLimit) != 0 {
                    break
                }
                n, err := w.NetworkBackend()
                if err != nil {
                    break
                }
                addrs, err := deriveChildAddresses(alb.branchXpub,
                    alb.lastUsed+1+alb.cursor, gapLimit, w.chainParams)
                if err != nil {
                    return nil, errors.E(op, err)
                }
                err = n.LoadTxFilter(context.TODO(), false, addrs, nil)
                if err != nil {
                    return nil, err
                }

            case gapPolicyWrap:
                alb.cursor = 0
            }
        }

And rescan need dcrd loaded transaction filter.

[jrick]

The filter is loaded as part of the initial sync operations:

https://github.com/decred/dcrwallet/blob/42b5e82/chain/sync.go#L454 and L463

Even if the filter is not including all addresses of the gap limit (it won't, and that's by design and correct -- the gap limit is used for discovering addresses), it will be loaded, and that is the only required precondition to perform the rescan RPC.