Combine efforts with 0xB10C/known-mining-pools

[0xB10C]

I've been maintaining a similar repository, 0xB10C/known-mining-pools. I think it would make sense to merge these two repositories at some point.

I opted for a more maintainable structure than having a large pools.json file for the tags and addresses. I'm using a JSON file per pool where a pool can have multiple tags and addresses. This is makes it much less cluttered and greatly reduces the error surface of having e.g. duplicates with slightly different names. See, for example, mara-pool.json.

{
  "name": "MARA Pool",
  "addresses": [
    "3D72db1KMCnj7FL7MBsmxTw81z2bVu4UN5",
    "3LC8dDKyBsrWPfzhXyt7aAyjXxGYkfDdHu",
    "15MdAHnkxt9TMC2Rj595hsg8Hnv693pPBB"
  ],
  "tags": [
    "MARA Pool"
  ],
  "links": [
    "https://marathondh.com/"
  ]
}

Automatically generating a pool.json file, as is used by many projects (mempool.space, miningpool.observer, forkmonitor.info, rust-bitcoin-pool-identification, ...), is done with this script. I've also set up quite a bit of automated CI for quality assurance.

I've seen that the mempool project added a slugs and I think adding this in the per-pool JSON file should be easy. As well as generating a pools.json file with the slugs as mempool.space expects them.

Let me know what you think. Also happy to move the combined repo to e.g. the @bitcoin-data organization.

[0xB10C]

As discussed on Matrix, I've merged https://github.com/0xB10C/known-mining-pools/pull/69 which included some of the pool information from this repo. We should be synced now.

I've also came across e.g. a few addresses tagged as AntPool which are clearly not coinbase output addresses. I had previously cleaned these up in my repo. For example: 1D9jw3QHNankXxtcGVihsDK7Z7THN6j7Pg.

https://github.com/mempool/mining-pools/blob/e73a42541eea345eb7bd65c02fa418778f0a909c/pools.json#L1213-L1216

As a next step, I'd be interested in hearing which format works best for mempool-space (the same old pool.json with slugs? or a new format?).

[nymkappa]

Thanks @0xB10C.

Regarding the non-coinbase addresses, I don't know why they are here, they were already in the original repo before we forked it. If the file should be soly focused on mining pool (which I think should), then that would make sense to only keep coinbase output addresses. If in the future, if we want to tag large holders on mempool.space, this should not just include miners but also exchanges and other known entities, which would be done in a different file. So your cleanup is welcome.

Regarding the format, the one you proposed is great. We want it to be easy to parse (here a simple JSON.parse will do) and easy to map to a database (rows/columns). We need to add the slug (or id, as a more general purpose field name) field which represents an unique identifier for a mining pool.

Regarding the export tool, which is a bit out of scope for this issue For mempool version <= 2.4.0, I believe we should not provide any update to the mining pool definitions. The reason is that even if you provide an export tool to convert from the new format to the old format, my parser will not be able to spot all changes, and as a result, user's database will tag some blocks improperly. The only workaround is to ask user to re-index everything, which of course can only be done manually since they are not upgrading.

Todo on mempool repo:

I will rewrite the mining pool parser from scratch to support this new format once validated, which should not be very complicated with that new syntax.

[0xB10C]

We need to add the slug (or id, as a more general purpose field name) field which represents an unique identifier for a mining pool.

Adding IDs in https://github.com/0xB10C/known-mining-pools/pull/74.

I've also changed the format a bit to only allow a single URL as link instead of a list of URLs as links in https://github.com/0xB10C/known-mining-pools/pull/71.

I will rewrite the mining pool parser from scratch to support this new format once validated, which should not be very complicated with that new syntax.

Let me know I you'd like a generated JSON-file with the separate per-pool-JSON-files as a list. I could easily generate something like this automatically once the pool data changes. Should make it easier to download it for mempool-space than having to load all JSON files separatly.

[
  {
    "id": "f2pool",
    "name": "F2Pool",
    "addresses": [
      "1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY",
      "bc1q7wedv4zdu5smt3shljpu5mgns48jn299mukymc",
      "bc1qf274x7penhcd8hsv3jcmwa5xxzjl2a6pa9pxwm"
    ],
    "tags": [
      "七彩神仙鱼",
      "🐟"
    ],
    "link": "https://www.f2pool.com"
  },
  {
    "id": "slush",
    "name": "SlushPool",
    "addresses": [
      "1AqTMY7kmHZxBuLUR5wJjPFUvqGs23sesr",
      "1CK6KHY6MHgYvmRQ4PAafKYDrg1ejbH1cE"
    ],
    "tags": [
      "/slush/"
    ],
    "link": "https://slushpool.com"
  },
...
]

[nymkappa]

Let me know I you'd like a generated JSON-file with the separate per-pool-JSON-files as a list

Yes we definitely need one single file with a list of JSON objects because we detect file changes using the sha sum of the file using github API https://github.com/mempool/mempool/blob/343a48818b9057b7fd0b428fef1e46205acca8c8/backend/src/tasks/pools-updater.ts#L102-L118

[0xB10C]

Here's a file you can test with: https://github.com/0xB10C/known-mining-pools/blob/d966997c35d068eb8f4cccc32acd65f07f3c72f0/generated/pool-list.json

Feel free to comment in https://github.com/0xB10C/known-mining-pools/pull/75 too if you run into any issues.

I'm planning to set up a CI job that generates and pushes the file automatically, which removes the burden of having to run the generate scripts every time someone touches the data.

[0xB10C]

There's now a new file to test with https://github.com/0xB10C/known-mining-pools/blob/master/generated/pool-list.json

~https://github.com/0xB10C/known-mining-pools/issues/77 is still a todo.~

[0xB10C]

Renamed Slush to Braiins in https://github.com/0xB10C/known-mining-pools/pull/78

[nymkappa]

Closing. Once we're done refactoring the mining pool process on our end, I will fetch updates manually from https://github.com/bitcoin-data/mining-pools.

Thanks for your help on this!