Client.TrackAsync() doesn't throw any error but derivation is not tracked

sigHash commented 5 years ago

Hi! I upgraded to latest versions of the client and NBXplorer (last update august 20th) and all the sudden none of my new derivations are being tracked.

I don't get any error by calling TrackAsync but no TransactionEvent happens when I send money to an address. I haven't changed any of my code and It used to work before without any problem.

NicolasDorier commented 5 years ago

Can you try downgrade to nbxplorer:1.0.2.31?

NicolasDorier commented 5 years ago

I could not reproduce the issue. Please let me know if the downgrade works.

NicolasDorier commented 5 years ago

Have you a way for me to reproduce it?

NicolasDorier commented 5 years ago

Ok I could not reproduce, can you udpate to latest NBXplorer. I released a version with better logs.

sigHash commented 5 years ago

I actually checked out master yesterday, implemented AWSBroker and built a new docker image. Let me make a pull request to make it more clear.

sigHash commented 5 years ago

I built a new image using docker-compose.testnet.yml and there is no transaction event on any of my derivation schemes. When I call ..../derivations/{my derivation}/addresses/unused, it returns derivation not found. I even explicitly POST derivation to track route (not using the client) and still no change.

NicolasDorier commented 5 years ago

I don't manage to replicate this. Have you a piece of code I can run on regtest?

sigHash commented 5 years ago

I tried to call the endpoint using a simple function:

derivationScheme = "2-of-tpubD9iq7nGnyzLFNr4Qs1CTbYQx6SeV2VLLeyexknK7cAzDp8REbQuWUN5DaCfpoMRt65QP2V3arh2LPAsWbammxEpHUPZKELzWMjzRewYvBuL-tpubD9iq7nGnyzLFTKWiyCWGFqwQTNb7RtqJbLRNsGZHd9F4Xj5KHehoqzGV4Mu7oP2EJ6RChCs1ououZ55DTwNkey98nBq5LiER4n1bjbwvuWt-tpubDECKqjCqDySTFgnHjq2936HU1LQuyAn8kvAoDdqbs76zdcQhJQ7HYm8178nr6uTVKa25DTwuc7QYoKydjS4DdRfhzHhPoS5uB36JvQYC9aF-[legacy]-[keeporder]"

 public async Task<string> GetAddress(string derivationScheme)
{
     string _nbxplorerEndpoint = "http://xx.xx.xx.xxx:pp";
     var _nbxplorerNetwork = new NBXplorerNetworkProvider(NetworkType.Testnet).GetBTC();
     var client = new ExplorerClient(_nbxplorerNetwork, new Uri(_nbxplorerEndpoint));

     var factory = new DerivationStrategyFactory(Network.TestNet);
     var newDerivation = factory.Parse(derivationScheme);
     await client.TrackAsync(newDerivation);

    return newDerivation.Derive(new KeyPath("1")).ScriptPubKey.GetScriptAddress(Network.TestNet).ToString();
}

Packages I'm using:

    <PackageReference Include="NBitcoin" Version="4.1.1.51" />
    <PackageReference Include="NBXplorer.Client" Version="1.0.3" />

Then I send money to the new address and no new transaction event.

NicolasDorier commented 5 years ago

That is strange, and with the latest version I pushed today of NBXplorer, you don't see any log? What if you remove the AWSBroker

sigHash commented 5 years ago

I just tried to build a new docker image from master using the docker file you provided. (Excluding all my changes). No logs besides new block events,

commit ec02fd91a1a5b372343c293a7721bf7b438be0d8 (HEAD -> master, tag: v1.0.3.2, tag: stable, origin/master, origin/HEAD)

Dockerfile:

FROM microsoft/dotnet:2.1.402-sdk-alpine3.7 AS builder
WORKDIR /source
COPY NBXplorer/NBXplorer.csproj NBXplorer/NBXplorer.csproj
# Cache some dependencies
RUN cd NBXplorer && dotnet restore && cd ..
COPY . .
RUN cd NBXplorer && \
    dotnet publish --output /app/ --configuration Release

FROM microsoft/dotnet:2.1.4-aspnetcore-runtime-alpine3.7
WORKDIR /app

RUN mkdir /datadir
ENV NBXPLORER_DATADIR=/datadir
VOLUME /datadir

COPY --from=builder "/app" .
ENTRYPOINT ["dotnet", "NBXplorer.dll"]

docker build -t nbxplorer .

Then I run it:

docker run  -it --rm  -p 8000:32838 --env-file nb.env nbxplorer:latest

Env file:

NBXPLORER_NETWORK=testnet
NBXPLORER_BIND=0.0.0.0:32838
NBXPLORER_NOAUTH=1
NBXPLORER_BTCRPCURL=http://x.x.x.x:18332/
NBXPLORER_BTCRPCUSER=btc
NBXPLORER_BTCRPCPASSWORD=btc123
NBXPLORER_BTCNODEENDPOINT=x.x.x.x:18333
NBXPLORER_BTCSTARTHEIGHT=1415186
NBXPLORER_BTCRESCAN=1

logs:

info: Configuration:  Data Directory: /datadir
info: Configuration:  Configuration File: /datadir/TestNet/settings.config
info: Configuration:  Creating configuration file
info: Configuration:  Network: Testnet
info: Configuration:  Supported chains: BTC
info: Configuration:  Network: Testnet
info: Configuration:  Supported chains: BTC
info: Configuration:  Rescanning the BTC chain...
info: Configuration:  BTC: Testing RPC connection to http://x.x.x.x:18332/
info: Configuration:  Starting Azure Service Bus Message Broker
Hosting environment: Production
Content root path: /app/
Now listening on: http://0.0.0.0:32838
Application started. Press Ctrl+C to shut down.
info: Configuration:  BTC: RPC connection successfull
info: Configuration:  BTC: Full node version detected: 160300
info: Configuration:  BTC: Loading chain from node...

The image is running on my local machine and successfully connected to my full node. I use postman to do:

POST http://localhost:8000/v1/cryptos/BTC/derivations/2-of-tpubD9iq7nGnyzLFNr4Qs1CTbYQx6SeV2VLLeyexknK7cAzDp8REbQuWUN5DaCfpoMRt65QP2V3arh2LPAsWbammxEpHUPZKELzWMjzRewYvBuL-tpubD9iq7nGnyzLFTKWiyCWGFqwQTNb7RtqJbLRNsGZHd9F4Xj5KHehoqzGV4Mu7oP2EJ6RChCs1ououZ55DTwNkey98nBq5LiER4n1bjbwvuWt-tpubDECKqjCqDySTFgnHjq2936HU1LQuyAn8kvAoDdqbs76zdcQhJQ7HYm8178nr6uTVKa25DTwuc7QYoKydjS4DdRfhzHhPoS5uB36JvQYC9aF-[legacy]-[keeporder]

It returns 200 OK.

Then I do:

GET http://localhost:8000/v1/cryptos/BTC/derivations/2-of-tpubD9iq7nGnyzLFNr4Qs1CTbYQx6SeV2VLLeyexknK7cAzDp8REbQuWUN5DaCfpoMRt65QP2V3arh2LPAsWbammxEpHUPZKELzWMjzRewYvBuL-tpubD9iq7nGnyzLFTKWiyCWGFqwQTNb7RtqJbLRNsGZHd9F4Xj5KHehoqzGV4Mu7oP2EJ6RChCs1ououZ55DTwNkey98nBq5LiER4n1bjbwvuWt-tpubDECKqjCqDySTFgnHjq2936HU1LQuyAn8kvAoDdqbs76zdcQhJQ7HYm8178nr6uTVKa25DTwuc7QYoKydjS4DdRfhzHhPoS5uB36JvQYC9aF-[legacy]-[keeporder]/addresses/unused

It returns:

{
    "httpCode": 404,
    "code": "strategy-not-found",
    "message": "This strategy is not tracked, or you tried to skip too much unused addresses"
}

Address on path 0/1:

2NAeXsKhp7cb4uU7uz8pgHvcoy5taAmMCkJ

Testnet transaction:

blockcypher

Full node blockchaininfo:

{
  "chain": "test",
  "blocks": 1433558,
  "headers": 1433558,
  "bestblockhash": "0000000000000f4208c4e27694abff1bba2a98612c02f8c810dea9231d0c049d",
  "difficulty": 262144,
  "mediantime": 1539026508,
  "verificationprogress": 0.9999988441713114,
  "initialblockdownload": false,
  "chainwork": "0000000000000000000000000000000000000000000000c6f5c048b5e42fded3",
  "size_on_disk": 22899370548,
  "pruned": false,
  "softforks": [
    {
      "id": "bip34",
      "version": 2,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip66",
      "version": 3,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip65",
      "version": 4,
      "reject": {
        "status": true
      }
    }
  ],
  "bip9_softforks": {
    "csv": {
      "status": "active",
      "startTime": 1456790400,
      "timeout": 1493596800,
      "since": 770112
    },
    "segwit": {
      "status": "active",
      "startTime": 1462060800,
      "timeout": 1493596800,
      "since": 834624
    }
  },
  "warnings": "Warning: unknown new rules activated (versionbit 28)"
}

It still doesn't track my derivation :/

NicolasDorier commented 5 years ago

Damn. I just tested the same thing on regtest and it works. Can you find a way to share NBXplorer data directory?

sigHash commented 5 years ago

!!! This is a link to datadir (zip). I am running a container using nicolasdorier/nbxplorer:latest from docker hub.

NicolasDorier commented 5 years ago

Thanks a lot, thanks for your patience, last days has been quite busy for me so there was lot's of latency to this discussion.

NicolasDorier commented 5 years ago

I am querying http://localhost:52113/v1/cryptos/BTC/derivations/2-of-tpubD9iq7nGnyzLFNr4Qs1CTbYQx6SeV2VLLeyexknK7cAzDp8REbQuWUN5DaCfpoMRt65QP2V3arh2LPAsWbammxEpHUPZKELzWMjzRewYvBuL-tpubD9iq7nGnyzLFTKWiyCWGFqwQTNb7RtqJbLRNsGZHd9F4Xj5KHehoqzGV4Mu7oP2EJ6RChCs1ououZ55DTwNkey98nBq5LiER4n1bjbwvuWt-tpubDECKqjCqDySTFgnHjq2936HU1LQuyAn8kvAoDdqbs76zdcQhJQ7HYm8178nr6uTVKa25DTwuc7QYoKydjS4DdRfhzHhPoS5uB36JvQYC9aF-[legacy]-[keeporder]/addresses/unused and I receive a result by running under windows... trying through the container.

NicolasDorier commented 5 years ago

Ok so even with the container, it keeps working... I don't know what more I can try.

NicolasDorier commented 5 years ago

Address on 0/1 is 2N1KWwhjk3SMoiAqPsUfxW7hv5mfoxjndG8

NicolasDorier commented 5 years ago

actually something is weird

NicolasDorier commented 5 years ago

So I managed to reproduce something on alpine. After a while the derivation scheme "disappear"

NicolasDorier commented 5 years ago

Fixed by 9af604834d7bae89e6789b9c1f9f3009a6fc861d . This was pretty bad. I wonder why nobody saw this issue earlier, as it should have impacted BTCPay users. Update to 1.0.3.3.

sigHash commented 5 years ago

It was indeed scary! Thanks for fixing it. Maybe I should go on a coding strike and make PostgreSQL happen. Let me know if you have time to review IRepository

NicolasDorier commented 5 years ago

Well Postgres would have the same issue. The problem is that I was not committing the changes.

dgarage / NBXplorer

Client.TrackAsync() doesn't throw any error but derivation is not tracked #112