These two different ways to estimate total chain work by the lowest hashes. It doesn't require knowing any difficulties or timestamps, or even max_target in the simplest form.
Potential Uses
This might be useful for "pruning the consensus layer" for "decentralized, trustless checkpoints" for things like SPV wallets and DAGs (where the number of block headers can be burdensome). "Checkpoints" including a "Proof of PoW" can be created by including all headers with the lowest hashes ever seen. The size of the proof increases logarithmically with chain work. See "Mining in Logarithmic Space".
Given the N lowest hashes ever seen:
*chain_work = (2^256 -1) (N-1) / Nth_lowest_hash** (eq 1)
StdDev of the error = 1/SQRT(N-1) for N > 10
on the condition that difficulty target was never < Nth_lowest_hash
This equation comes from viewing Nth_lowest_hash as a target that was met by N-1 hashes. As a review:
difficulty = (2^256-1)/target = 2^32 BTC_difficulty
Chain_work = number_of_hashes_below_target difficulty
The StdDev equation is my guess, supported by experiment (see code below). The equation assumes all the lowest hashes ever hashed are on the blockchain as a block hash which is the case if the PoW target never fell below Nth_lowest_hash (and assuming no orphans were lower than the Nth, otherwise they need to be included in N). In other words, difficulty must not have ever been > chain_work/(N-1). Unknown hashes smaller than the Nth (such as orphans) would silently increase N which would increase the estimate. N can be large to get good accuracy: up to height/14 =~ 60,000 for today's Bitcoin.
For the case of N=1, the statistics behind above equation have no solution (small hashes greatly increase the result and you could get a divide by zero) but experiment indicates the following where the "6" could be anything from 1 to infinity in any particular million runs that calculate an average. Usually it's from 2 to 12:
The equation can be simplified. We replace the Nth_lowest_hash target with a number of leading zeros ("u") that has M hashes below it (M = N-1). The target for u leading zeros is 2^(256-u). Eq 1 simplifies (with an error of 10^(-78)) to:
chain_work = (2^256-1) M / 2^(256-u) = M 2^u (eq 2)
Stdev err = 1/sqrt(M)
As before, this equation is chain_work = M * difficulty, i.e. M hashes were found at an imaginary difficulty that has been in place since genesis. It's still on the condition that the blockchain's difficulties never exceeded 2^u, i.e. u must be > log(highest_difficulty) / log(2). Bitcoin's highest-ever difficulty is currently 85 T (times 2^32) which shows u must be > 78 to meet that condition. There are about 60,000 blocks with hashes < 2^79, which has about 0.4% expected & actual error.
It's interesting and important to note that eq 2 isn't valid if you choose M and then count the leading zeros (u). Adam Back made this error. This is because the actual u (the 2^u imaginary difficulty) for that M could have been u, u-1, u-2, but not u+1, u+2, etc. In other words, you're taking into account M's that were lucky but not the unlucky ones. This would overestimate but not underestimate the total work, so the mean of many such observations will be an overestimate (by experiment: ~10x for M=1, 3.7x for M=2, 2.2x for M=3, and 1.6x for M=4). If you use it correctly (choose u first) and expect (via knowing the actual number of hashes) a mean of M=1, many tests runs will have M=0 which reduces the mean to the correct mean.
In the notation of "Mining in Logarithmic Space" M = 2m = 6k = 36. I started working on this because I couldn't see their estimate for total chain work, just a way to determine which of two chains is longer.
If leading zeros are in hex:
chain_work = M * 16^u
As of block height 826532, there are 96 block hashes with 22 or more leading zeros when expressed in hex. Current chain work is 3.17E28 and the equation gives 96 * 16^22 = 2.97E28, a 6.3% error.
Using eq 1 for Proof of PoW
The estimation might be used to prune the proof of work in the following scheme: If a block finds one of the N=100 lowest hashes ever seen, then the block that mines on top of him declares himself the newest checkpoint and includes the 100 headers that have the lowest hashes ever seen. He gets the other 99 from the prior checkpoint. Then every block after his will reference his block hash & height as the most recent checkpoint that everyone must store. Total chain work is the work implied in the headers in the most recent checkpoint as determined by equation 1 plus the sum of difficulties (times 2^32 if used in BTC) after the checkpoint. This prevents the need to refer to earlier checkpoints to get PoPoW. An attacker would need to mine about 1/N the total chain work to subvert the most recent checkpoint. The number of blocks back to the most recent checkpoint increase as total work increases if N is a constant. Increasing N with chain work to prevent this "increase back to prior checkpoint" requires knowing none of the N hashes were ever above the lowest target (which may require removing some of the earliest N's) and making an assumption about (or an enforcement of) how much difficulty changes between checkpoints. An attacker may find a new low hash and not disclose it for a while, enabling double-spending, so the majority of hashrate needs to be honest and to stay online since the most recent checkpoint to ignore new checkpoints that don't include all blocks since the previous checkpoint.
Bitcoin Observations
Real world example for the 1st equation: 10th lowest hash for BTC as of height 720,441 (when chainwork was 1.23E28) was in block 696,345 who's hash was 8.8E49.
000000000000000000036a76070bdfc971182b2a7430b1cbd99297fc9a6b513f
chain_work = (2^256-1) * (10-1) / 8.8E49 = 1.3E28 (6% error).
A more complete example from this data. The data is also at the end of this article.
Here's an example of using eq 2 as of January 24, 2024.
The above M values was collected using the following SQL after registering at dune.com. The example below is for 20 leading hex zeros (80 in binary) because it's 19 hex zeros following by "1". Change the 1 to 2 for 79 leading binary zeros, and change the 1 to 80 for 81 leading binary zeros (the 0 after the 8 is needed due to a quirk in their SQL wanting an even number of characters).
SELECT
COUNT(*)
FROM bitcoin.blocks
WHERE
hash < 0x00000000000000000001
The following is experimental tests to show 1st equation is correct.
# equation 1, chain_work = (2^256 -1) * (N-1) / Nth_lowest_hash
import random
import statistics
runs=100 # number of times to do the experiment
hashes=100000 # number of hashes to do in each experiment
N=5 # number of lowest hashes to use in the calculation
max_target = pow(2,256)
avg_max = 0
max_all = []
for i in range(1, runs):
# generate hash targets with values 0 to max_target.
hash_list = [random.randint(1,max_target) for _ in range(hashes)]
lowest_hashes = sorted(hash_list)[:N]
max_of_low_hashes = max(lowest_hashes)
if N==1:
max_all.append(max_target/6/max_of_low_hashes)
avg_max = avg_max + max_target/6/max_of_low_hashes/runs
else:
max_all.append(max_target*(N-1)/max_of_low_hashes)
avg_max = avg_max + max_target*(N-1)/max_of_low_hashes/runs
print("N = " + str(N))
print("actual chain work (hashes) = " + str(hashes))
print(f"Estimated CW, avg of runs = max_target * (N-1) / max_of_N_lowest_targets: {avg_max:,.0f}")
print(f"Stdev: {statistics.stdev(max_all):,.0f}")
Output to supports the 1st equation:
Total hashes (actual chain work): 10,000
Runs: 30,000
Here's the code for the leadings zeros method for measuring chain work (identical to the other method, but in a different math route).
# equation 2, chain_work = M * 2^leading_zeros
import random
import statistics
runs = 1000 # number of times to do the experiment
hashes = 10000 # number of hashes to do in each experiment
u = 13
difficulty = pow(2,u)
max_target = pow(2,256)
all_run_counts = []
for i in range(1, runs+1):
count=0
for j in range(1,hashes+1):
hash=max_target*random.random()
if hash < max_target/difficulty:
count+=1
all_run_counts.append(count)
print(f"Selected u-level (leading zeros): {u:,.0f}")
print(f"{hashes:,.0f} hashes in each of {runs:,.0f} runs.")
print(f"Mean of estimated hashes: {statistics.mean(all_run_counts)*difficulty:,.0f}")
print(f"Stdev of estimated hashes: {statistics.stdev(all_run_counts)*difficulty:,.0f}")
print(f"Mean of hashes below 2^(256-u): {statistics.mean(all_run_counts):,.2f}")
print(f"Stdev of hashes below 2^(256-u): {statistics.stdev(all_run_counts):,.2f}")
Lowest 100 hashes as of bitcoin block height 826532 (chain work 3.17E28). To get these, I registered at dune.com and asked it's A.I. "What are the 100 lowest block hashes?"
These two different ways to estimate total chain work by the lowest hashes. It doesn't require knowing any difficulties or timestamps, or even max_target in the simplest form.
Potential Uses This might be useful for "pruning the consensus layer" for "decentralized, trustless checkpoints" for things like SPV wallets and DAGs (where the number of block headers can be burdensome). "Checkpoints" including a "Proof of PoW" can be created by including all headers with the lowest hashes ever seen. The size of the proof increases logarithmically with chain work. See "Mining in Logarithmic Space".
Given the N lowest hashes ever seen:
*chain_work = (2^256 -1) (N-1) / Nth_lowest_hash** (eq 1) StdDev of the error = 1/SQRT(N-1) for N > 10 on the condition that difficulty target was never < Nth_lowest_hash
This equation comes from viewing Nth_lowest_hash as a target that was met by N-1 hashes. As a review: difficulty = (2^256-1)/target = 2^32 BTC_difficulty
Chain_work = number_of_hashes_below_target difficulty
The StdDev equation is my guess, supported by experiment (see code below). The equation assumes all the lowest hashes ever hashed are on the blockchain as a block hash which is the case if the PoW target never fell below Nth_lowest_hash (and assuming no orphans were lower than the Nth, otherwise they need to be included in N). In other words, difficulty must not have ever been > chain_work/(N-1). Unknown hashes smaller than the Nth (such as orphans) would silently increase N which would increase the estimate. N can be large to get good accuracy: up to height/14 =~ 60,000 for today's Bitcoin.
For the case of N=1, the statistics behind above equation have no solution (small hashes greatly increase the result and you could get a divide by zero) but experiment indicates the following where the "6" could be anything from 1 to infinity in any particular million runs that calculate an average. Usually it's from 2 to 12:
chain_work = (2^256 -1) / 6 / lowest_hash
StdDev =~ 5000%
The equation can be simplified. We replace the Nth_lowest_hash target with a number of leading zeros ("u") that has M hashes below it (M = N-1). The target for u leading zeros is 2^(256-u). Eq 1 simplifies (with an error of 10^(-78)) to:
chain_work = (2^256-1) M / 2^(256-u) = M 2^u (eq 2) Stdev err = 1/sqrt(M)
As before, this equation is chain_work = M * difficulty, i.e. M hashes were found at an imaginary difficulty that has been in place since genesis. It's still on the condition that the blockchain's difficulties never exceeded 2^u, i.e. u must be > log(highest_difficulty) / log(2). Bitcoin's highest-ever difficulty is currently 85 T (times 2^32) which shows u must be > 78 to meet that condition. There are about 60,000 blocks with hashes < 2^79, which has about 0.4% expected & actual error.
It's interesting and important to note that eq 2 isn't valid if you choose M and then count the leading zeros (u). Adam Back made this error. This is because the actual u (the 2^u imaginary difficulty) for that M could have been u, u-1, u-2, but not u+1, u+2, etc. In other words, you're taking into account M's that were lucky but not the unlucky ones. This would overestimate but not underestimate the total work, so the mean of many such observations will be an overestimate (by experiment: ~10x for M=1, 3.7x for M=2, 2.2x for M=3, and 1.6x for M=4). If you use it correctly (choose u first) and expect (via knowing the actual number of hashes) a mean of M=1, many tests runs will have M=0 which reduces the mean to the correct mean.
In the notation of "Mining in Logarithmic Space" M = 2m = 6k = 36. I started working on this because I couldn't see their estimate for total chain work, just a way to determine which of two chains is longer.
If leading zeros are in hex: chain_work = M * 16^u
As of block height 826532, there are 96 block hashes with 22 or more leading zeros when expressed in hex. Current chain work is 3.17E28 and the equation gives 96 * 16^22 = 2.97E28, a 6.3% error.
Using eq 1 for Proof of PoW
The estimation might be used to prune the proof of work in the following scheme: If a block finds one of the N=100 lowest hashes ever seen, then the block that mines on top of him declares himself the newest checkpoint and includes the 100 headers that have the lowest hashes ever seen. He gets the other 99 from the prior checkpoint. Then every block after his will reference his block hash & height as the most recent checkpoint that everyone must store. Total chain work is the work implied in the headers in the most recent checkpoint as determined by equation 1 plus the sum of difficulties (times 2^32 if used in BTC) after the checkpoint. This prevents the need to refer to earlier checkpoints to get PoPoW. An attacker would need to mine about 1/N the total chain work to subvert the most recent checkpoint. The number of blocks back to the most recent checkpoint increase as total work increases if N is a constant. Increasing N with chain work to prevent this "increase back to prior checkpoint" requires knowing none of the N hashes were ever above the lowest target (which may require removing some of the earliest N's) and making an assumption about (or an enforcement of) how much difficulty changes between checkpoints. An attacker may find a new low hash and not disclose it for a while, enabling double-spending, so the majority of hashrate needs to be honest and to stay online since the most recent checkpoint to ignore new checkpoints that don't include all blocks since the previous checkpoint.
Bitcoin Observations Real world example for the 1st equation: 10th lowest hash for BTC as of height 720,441 (when chainwork was 1.23E28) was in block 696,345 who's hash was 8.8E49. 000000000000000000036a76070bdfc971182b2a7430b1cbd99297fc9a6b513f chain_work = (2^256-1) * (10-1) / 8.8E49 = 1.3E28 (6% error).
A more complete example from this data. The data is also at the end of this article.
Here's an example of using eq 2 as of January 24, 2024.
The above M values was collected using the following SQL after registering at dune.com. The example below is for 20 leading hex zeros (80 in binary) because it's 19 hex zeros following by "1". Change the 1 to 2 for 79 leading binary zeros, and change the 1 to 80 for 81 leading binary zeros (the 0 after the 8 is needed due to a quirk in their SQL wanting an even number of characters).
The following is experimental tests to show 1st equation is correct.
Lowest 12 block hashes as of 773048. Chain work as of then was 1.93E28.
Here's the code for the leadings zeros method for measuring chain work (identical to the other method, but in a different math route).
Lowest 100 hashes as of bitcoin block height 826532 (chain work 3.17E28). To get these, I registered at dune.com and asked it's A.I. "What are the 100 lowest block hashes?"