Desirability and apparent_performance 0 on Linux build inconsistent with MacOS cardano-wallet

[Straightpool]

Context

Cardano-wallet returns `0` on `apparent_performance` and `desirability` on Linux builds when MacOS (and most likely Windows) builds of cardano-wallet return valid positive data for both. | Information | - | | --- | --- | | Version Ubunto | 2020.1.27 (git revision: 23e12d1a26d282bf4acf9d8df3de9803e7e9baf5) | | Version MacOS | 2020.1.27 (git revision: a6df56ed91132e344925df9736ffaba6922b256f) | | Platform | Ubunto and MacOS | | Installation | From Github Release | # Steps to Reproduce

1. Update Daedalus ITN binaries on MacOS for Jormungandr, JCLI and cardano-wallet-jormungandr to latest builds
2. Sync Daedalus ITN wallet on MacOS
3. Check rank of 10 or abouts for pool STR8
4. Find port for cardano-wallet with netstat -a | grep -i "listen"
5. Query cardano-wallet for desirability for pool on MacOS within Daedalus ITN wallet folder

./cardano-wallet-jormungandr stake-pool list --port 63215 | grep 3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6 -B30 -A2

returns

Ok. }, { "saturation": 2.685978906636558e-3, "metrics": { "controlled_stake": { "quantity": 305918895633, "unit": "lovelace" }, "produced_blocks": { "quantity": 4, "unit": "block" } }, "cost": { "quantity": 0, "unit": "lovelace" }, "margin": { "quantity": 4, "unit": "percent" }, "apparent_performance": 1.7508770049058773, "metadata": { "homepage": "https://straightpool.github.io/about/", "owner": "ed25519_pk12rdfk8a29tdnmu85nh7g39hj0spsr70nyg5vzvuxat0d6qpjaqhs480msg", "name": "Straight Pool", "ticker": "STR8", "pledge_address": "addr1s3ypzx7z3la27lmlsfxk5rqx7rjdlne2ce9qr2l076nlk5uv3rhdhz8xpq0h2xflzcxamrs97kl60eyspex84l8yrc62nr99s9xdv790nahhrv", "description": "Competitive and sustainable 24/7 pool on decentralized infrastructure maintained by IT professionals." }, "id": "3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6", "desirability": 5.8023584275117905e10 }, `

The performance and desirability shown here is in line with Daedalus UI on MacOS showing rank 10 and a performance of 175,48%.

6. Query desirability for same pool on Ubunto with Linux build of cardano-wallet at the same time with a freshly synced cardano-wallet

./cardano-wallet stake-pool list | grep 3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6 -B30 -A2

returns

Ok. }, { "saturation": 2.685978906636558e-3, "metrics": { "controlled_stake": { "quantity": 305918895633, "unit": "lovelace" }, "produced_blocks": { "quantity": 4, "unit": "block" } }, "cost": { "quantity": 0, "unit": "lovelace" }, "margin": { "quantity": 4, "unit": "percent" }, "apparent_performance": 0, "metadata": { "homepage": "https://straightpool.github.io/about/", "owner": "ed25519_pk12rdfk8a29tdnmu85nh7g39hj0spsr70nyg5vzvuxat0d6qpjaqhs480msg", "name": "Straight Pool", "ticker": "STR8", "pledge_address": "addr1s3ypzx7z3la27lmlsfxk5rqx7rjdlne2ce9qr2l076nlk5uv3rhdhz8xpq0h2xflzcxamrs97kl60eyspex84l8yrc62nr99s9xdv790nahhrv", "description": "Competitive and sustainable 24/7 pool on decentralized infrastructure maintained by IT professionals." }, "id": "3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6", "desirability": 0 }, `

7. Compare desirability and apparent_performance
8. Try not to cry

Expected behavior

Desirability and apparent_performance on Linux build cardano-wallet queries should match what MacOS and Windows Daedalus clients show. This is especially bad since services like Pegasuspool ranking use data from Linux build cardano-wallet queries which is not corresponding to the wallets most people use.

Actual behavior

Linux build cardano-wallet returns 0 for both apparent_performance and desirability while MacOS cardano-wallet returns expected and valid positive values for both.

---

Resolution

---

QA

[KtorZ]

@Straightpool that is an interesting behavior though there's not particular difference between the MacOS (resp. Windows) and Linux builds. Same source code, same logic, especially regarding the performance calculation. Are you sure of the versions used in both cases? If you can still reproduce on Linux, I'd be interested in the logs if you could provide them. If possible, may you also increase the minimal severity on the relevant tracers to increase the log verbosity: --trace-pools-engine DEBUG --trace-pools-db DEBUG

Thanks.

[KtorZ]

Note that, at least as of today, the pool's performance look plausible to me on Linux. I don't have a Windows machine at hand right now to compare but..

[
  {
    "saturation": 0.0040219698185849145,
    "metrics": {
      "controlled_stake": {
        "quantity": 425802159397,
        "unit": "lovelace"
      },
      "produced_blocks": {
        "quantity": 1,
        "unit": "block"
      }
    },
    "cost": {
      "quantity": 0,
      "unit": "lovelace"
    },
    "margin": {
      "quantity": 4,
      "unit": "percent"
    },
    "apparent_performance": 0.4700528335618468,
    "metadata": {
      "homepage": "https://straightpool.github.io/about/",
      "owner": "ed25519_pk12rdfk8a29tdnmu85nh7g39hj0spsr70nyg5vzvuxat0d6qpjaqhs480msg",
      "name": "Straight Pool",
      "ticker": "STR8",
      "pledge_address": "addr1s3ypzx7z3la27lmlsfxk5rqx7rjdlne2ce9qr2l076nlk5uv3rhdhz8xpq0h2xflzcxamrs97kl60eyspex84l8yrc62nr99s9xdv790nahhrv",
      "description": "Competitive and sustainable 24/7 pool on decentralized infrastructure maintained by IT professionals."
    },
    "id": "3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6",
    "desirability": 15577422129.317404
  }
]

There are several possible explanations for this discrepancy, so bear with me a minute:

1. An actual bug in cardano-wallet.

2. The apparent performance depends on what the wallet has seen. Being only a node client there are a few historical metrics that we can't access nor can't compute without maintaining a whole ledger state. So, if you wallet server wasn't up and connected to a network at, let's say epoch E, then you won't have data about epoch E and, won't be able to compute performance reliably. Trying to cope with this, the wallet computes the performance using data up to 14 epochs in the past. So, different wallets may have different data and ends up with slightly different performance. This is especially true for small pools that don't necessarily produce the same amount of blocks each epoch. For others, it's roughly averaged out.

3. There's no single source of truth on a blockchain. There are possibly an infinite amount of branches possible and, different nodes on different machines might as well be on very different branches. Yet, in your scenario, it seems like your pool production matches in both case so I tend to think that 2. is more likely than 3. here.

[Straightpool]

KtorZ, thank you for looking into this.

On to your points:

• Yes I am very confident the versions I pasted for both versions are accurate, I double checked. I have a bit of an unusual Daedalus installation on MacOS at the moment, I downloaded the much newer cardano-wallet from https://github.com/input-output-hk/cardano-wallet/releases/tag/v2020-01-27 and replaced cardano-wallet-jormungandr binary with this one to get Daedalus to sync in the Applications folder

• I just synced Daeadalus on MacOS again. In the UI my pool dropped to spot 19. I got a bit more stake from delegation since last epoch but still no assigned slots, so this is in line with expectations.

The UI in Daedalus is again in line with a repeated query on MacOS of the cardano-wallet:

Ok. }, { "saturation": 3.8142502215246753e-3, "metrics": { "controlled_stake": { "quantity": 434747953873, "unit": "lovelace" }, "produced_blocks": { "quantity": 4, "unit": "block" } }, "cost": { "quantity": 0, "unit": "lovelace" }, "margin": { "quantity": 4, "unit": "percent" }, "apparent_performance": 1.4859441067241674, "metadata": { "homepage": "https://straightpool.github.io/about/", "owner": "ed25519_pk12rdfk8a29tdnmu85nh7g39hj0spsr70nyg5vzvuxat0d6qpjaqhs480msg", "name": "Straight Pool", "ticker": "STR8", "pledge_address": "addr1s3ypzx7z3la27lmlsfxk5rqx7rjdlne2ce9qr2l076nlk5uv3rhdhz8xpq0h2xflzcxamrs97kl60eyspex84l8yrc62nr99s9xdv790nahhrv", "description": "Competitive and sustainable 24/7 pool on decentralized infrastructure maintained by IT professionals." }, "id": "3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6", "desirability": 4.924378060996894e10 },

apparent_performance dropped since yesterday from 1.7508770049058773 to 1.4859441067241674 and desirability dropped from 5.8023584275117905e10 to 4.924378060996894e10.

The numbers you quoted for apparent_performance are 0.4700528335618468 and for desirability a wopping 15577422129.317404. Does this really seem plausible to you?

I also just now repeated the query on Ubunto by syncing cardano-wallet again, again I did reproduce output 0 on both:

Ok. }, { "saturation": 3.8142502215246753e-3, "metrics": { "controlled_stake": { "quantity": 434747953873, "unit": "lovelace" }, "produced_blocks": { "quantity": 4, "unit": "block" } }, "cost": { "quantity": 0, "unit": "lovelace" }, "margin": { "quantity": 4, "unit": "percent" }, "apparent_performance": 0, "metadata": { "homepage": "https://straightpool.github.io/about/", "owner": "ed25519_pk12rdfk8a29tdnmu85nh7g39hj0spsr70nyg5vzvuxat0d6qpjaqhs480msg", "name": "Straight Pool", "ticker": "STR8", "pledge_address": "addr1s3ypzx7z3la27lmlsfxk5rqx7rjdlne2ce9qr2l076nlk5uv3rhdhz8xpq0h2xflzcxamrs97kl60eyspex84l8yrc62nr99s9xdv790nahhrv", "description": "Competitive and sustainable 24/7 pool on decentralized infrastructure maintained by IT professionals." }, "id": "3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6", "desirability": 0 },

Saturation and controlled stake is identical on both MacOS and Linux, so the data seems uptodate otherwise.

I have run the 0 Linux query with higher verbosity as requested, the relevant logging output is in the attached file: cardano-wallet-ubunto-debug.txt.zip

I understand the values are dependant on what a wallet has seen, but having seen the whole blockchain and being on the same fork, I do still expect to see the same data more or less. I have started and synced the cardano-wallet on Ubunto specifically for this test, so it has had minimal uptime (twice fully synced, query run, node stopped)

My pool has a horrific ranking on Pegasuspool.info with 0 performance. This app relies on the Ubunto output of cardano-wallet. On follow-up with Pegasuspool a screenshot of the 0 output was shared with me. This wallet should have seen a lot, as the site is online for weeks now.

This is when I decided to follow-up on this.

Let me know if I can provide any more debugging output.

[KtorZ]

@Straightpool thanks for the details. One more thing, may you run the following query directly on your sqlite database on Linux (the database should be located in ~/.local/share/cardano-wallet/jormungandr/testnet/wallets/stake-pools.sqlite3 if you used the default location for the state directory).

sqlite> SELECT pool_id,epoch FROM stake_distribution WHERE pool_id LIKE "3a192bc7%";

[Straightpool]

Sure, the filename has extension .sqlite not .sqlite3.

sqlite> SELECT pool_id,epoch FROM stake_distribution WHERE pool_id LIKE "3a192bc7%";
pool_id                                                           epoch     
----------------------------------------------------------------  ----------
3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6  57        
3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6  58        

My small pool so far successfully produced 4 blocks in epoch 37, 45 and 48 see https://shelleyexplorer.cardano.org/en/stake-pool/3a192bc7285d7ddfd77c0b854b4008bbf4c5fe53d1ad42770f6ffdf9b3d4c7d6

Seems cardano-wallet does only look at the epochs it has seen during uptime and does not look back 14 epochs from today as it should.

[KtorZ]

@Straightpool Sooo.. I am afraid this is "normal". The software does look back to 14 epochs in the past, but, it can only compute the performance of epochs for which it knows the stake distribution. There's currently no way to retrieve the stake distribution of past epochs from Jörmungandr, and, this is not something that can be computed easily from the blockchain data. The stake distribution goes through many implicit calculations and operations that are not recorded on-chain, but tracked by ledger nodes using on-chain data. In order to redo these calculations, one would need to keep track of all data regarding all delegation accounts and stake pools. Said differently, one would need to be a ledger.

Yet, we are a mere wallet here so we do with what's available. Only the stake distribution of the node's ongoing epoch is available. So, if your wallet wasn't able to query the node at epoch 45 and 48, then it's too late. We cannot get the corresponding data back.

This is a known limitation and issue (https://github.com/input-output-hk/jormungandr/issues/852). You may upvote this if you think that'd be useful but at this stage, there's not much we can do from cardano-wallet's perspective I am afraid :/

[Straightpool]

Alright, this is plausible then. What I still do not understand though is how PegasusPool still sees 0 with a cardano-wallet node which should be fully synced and running for weeks on end. I have asked him to provide debugging output from his end to append to this issue.

To confirm this is no bug his sqlite database must have no stake_distribution data on epoch 45 and 48. Today the calculation should look back 14 epochs to epoch 44.

[KtorZ]

Yes, if they indeed were online all the time, especially during epoch 45 and 48, then this is puzzling. Yet, they might also be on a fork, or simply have missed one or two epochs. Can't really tell.

[Straightpool]

Today I was shocked to see my Daedalus ranking plummet, by looking at the stake-pool.sqlite DB used by Daedalus I could again confirm that it missed the epochs criticial to calculate rank for my small pool.

For anyone else who wants to do so:

1. Open Daedalus Diagnostics to determine the "Daedalus state directory" within this directory is a wallet directory with the stake-pools.sqlite file
2. start "sqlite3" in a terminal
3. on the command line open the sqlite file with ".open "<PATH>/wallet/stake-pools.sqlite"
4. Query for your pool SELECT pool_id,epoch FROM stake_distribution WHERE pool_id LIKE "<First-8-chars-of-yourpoolID>%"
5. Notice that epochs you produced blocks in which would increase your rank are missing, which is why cardano-wallet cannot accurately calculate rank

I hereby close this ticket to follow up on the root cause.

[Straightpool]

I have decided to reopen this, until the underlying issue in input-output-hk/jormungandr#852 is fixed

[Straightpool]

This underlying issue fix was just merged and should be in the next Jormungandr build and hopefully used by the next cardano-wallet used by the next Daedalus release!

I will close this issue as soon as I can validate the fix, hopefully next week.

[KtorZ]

Should be fixed since #1444 . Note that this may take up to 14 epochs to stabilize as cardano-wallet will know record the right stake distribution at each epoch; but this won't go back in already existing data; thus if your wallet server is already synced, you'll have to wait for 14 epochs to see the full effect. Otherwise, removing the stake pool database could be an option; it'll get automatically re-created and after a few minutes be synced again with relevant data.

[Straightpool]

Ranking looks good and consistent with Daedalus 2.3.1 tested today. I am happy to close the issue. Thanks for the hard work!