So far as I can tell this code path is completely unreachable in any practical scenario, and is at worst able to cause denial-of-service. On 64-bit systems I think you use more RAM than has ever been manufactured (and certainly more than you can fit in most processors' physical address spaces) before it triggers.
The only case I can imagine is a system with a 32-bit usize which will still let you allocate a (2^32 - 63)-length Vec (unlikely unless sizeof(void*) != sizeof(size_t)), plus an input to squeeze_and_ratchet where ((outlen/HASHLEN)+1)*HASHLEN >= (1<<32).
The
while i * HASHLEN < outlenpattern insqueeze_and_ratchetfor bothShoSha256andShoHmacSha256can overflow if given an extremely largeoutlen, leading to a panic indebugmode or an infinite memory-eating loop inreleasemode. A tweaked implementation usingu32and a test case that trips the overflow is available at https://github.com/jpdoyle/libsignal/blob/fb6a53de6c7b26863d1feab1d83c62291ac1be0d/rust/poksho/src/shosha256.rs#L131So far as I can tell this code path is completely unreachable in any practical scenario, and is at worst able to cause denial-of-service. On 64-bit systems I think you use more RAM than has ever been manufactured (and certainly more than you can fit in most processors' physical address spaces) before it triggers.
The only case I can imagine is a system with a 32-bit
usizewhich will still let you allocate a(2^32 - 63)-lengthVec(unlikely unlesssizeof(void*) != sizeof(size_t)), plus an input tosqueeze_and_ratchetwhere((outlen/HASHLEN)+1)*HASHLEN >= (1<<32).I initially thought that if you have a poksho statement with 67108863 scalar parameters and all these other things line up, you can potentially trigger the overflow via https://github.com/signalapp/libsignal/blob/aa5f7c2b14315e94fa54fbdbda02b3938b218925/rust/poksho/src/statement.rs#L206 -- but it turns out that by including the encoded statement into the transcript,
to_bytesgets called and that will fail an assert if there are more than 256 scalars.