Question about the max output length

[lj-k]

Hello, in $2.6 of this spec, "BLAKE3 can produce outputs of any byte length 0 < 2^64."

I know the output length could be extended according the 64bit counter t, so the output could have 2^64's block for maximum. But the output of one block is 512 bit (64 byte), why not the max output is 64 * 2^64 byte?

Thanks

[oconnor663]

You're absolutely right. The way we've defined things, the upper 6 bits of t are never used. (And on the input side, the upper 10 bits are never used.) The benefit of specifying things this way is that it gives implementations "permission" to use a u64 as a byte index for input and output. This matters when we consider types like OutputReader that support seeking. Here's a Rust program that reads 100 bytes of BLAKE3 output starting at index 2⁶⁴-1:

use std::io::prelude::*;
use std::io::SeekFrom;

fn main() -> anyhow::Result<()> {
    let mut output = blake3::Hasher::new().finalize_xof();
    output.seek(SeekFrom::Start(u64::MAX))?;
    let mut buf = [0u8; 100];
    output.read(&mut buf)?;
    dbg!(output.position());
    Ok(())
}

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/scratch`
thread 'main' panicked at 'attempt to multiply with overflow', /home/jacko/.cargo/registry/src/github.com-1ecc6299db9ec823/blake3-1.3.3/src/lib.rs:1417:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It crashes! As it turns out, the call to read actually succeeds, because we store the position as a 64-bit count of blocks (not bytes), and incrementing that count doesn't overflow in this case. But the call to position fails, because it multiplies by 64 to compute the byte offset, and it's that multiplication that overflows. If we run the same example in --release mode, the overflow checks are removed, and we get something worse than a crash...we get the wrong answer!

$ cargo run --release
   Compiling scratch v0.1.0 (/tmp/tmp.WaRua5GnAo/scratch)
    Finished release [optimized] target(s) in 0.15s
     Running `target/release/scratch`
[src/main.rs:9] output.position() = 99

So, is this implementation of BLAKE3 incomplete or broken? No, because BLAKE3 is only defined to provide 2⁶⁴-1 bytes of output. If you ask for something in that range, a correct implementation must give you the correct answer. But if you go outside that range, a correct implementation is within its rights to crash or to give you a garbage answer.

Of course this is mostly just a theoretical question. Most applications will never look at the far end of the output range like this. Also, even if we said that inputs longer than 2⁶⁴-1 bytes were supposed to be supported, real world implementations would mostly ignore that requirement and do something similar to what our implementation does above. So another way to think about our choice here in the spec, is that we want to specify behavior that people are likely to implement correctly, and we want to avoid specifying behavior that's likely to be broken or incomplete in the real world.

[lj-k]

Thanks very mush. I got it and perceive the circumspection from designers. For avoiding the bug/wrong answer caused by overflow in some environment/situations, limiting the input and output length will be useful and will never going wrong in real word applications. Thanks again.