Draw a clear distinction between multi-part messages and streams

[kirelagin]

Everything is easy with standard “block”-level cryptography: you have a sequence of bytes, you sign/encrypt/mac it and you get your result as a new sequence of bytes (which might or might not include the original one in encrypted or plaintext form).

When we start talking about streaming data, things become more complicated. In particular, we can do two things with a stream:

• Fold it into a single value (e.g. a signature or a mac)
• Map something over it (e.g. encryption)

However note that if the data is processed in a streaming fashion, it is likely to be consumed in a streaming fashion as well. If the signature/mac is only placed at the end of the stream, this brings the danger of the data being maliciously modified, but still consumed eagerly, even though the signature/mac is no longer valid – but we discover this only at the very end of the stream.

As an example, imagine that there is an encrypted .sql file. We do something like:

$ decrypt foo.sql.enc > psql dbname

The file will be encrypted chunk-by-chunk and fed to psql, which will execute each complete statement one-by-one. But if each individual chunk is not authenticated, this means we will discover that the file was tampered with only at the very end, when most of the statements are already executed.

For this reason we propose to make the distinction between multi-part messages (treated as a single message, processed only as a whole) and streams (processed as soon as the data becomes available).

Encryption

• Multi-part encryption does not really make much sense, because we need to “process” (encrypt/decrypt) each part of the message anyway, so just adding an authentication tag to each part is not a big deal.
• Streaming encryption is simply encryption that adds an authentication tag to each chunk of the stream. Very easy, very convenient.

Signature / MAC

• Multi-part signatures/macs make sense when data is handled in-place, e.g. when we want to sign a file on the disc – we read it lazily and compute the signature part-by-part. It is very important to verify the signature on the whole file then rather before processing any part of it. This is actually pretty dangerous, so there should be a warning, but this can be useful.
• Streaming signatures/macs are useful when one wants to send data somewhere but not encrypt it. In this case we are copying it anyway, so attaching an authentication tag to each chunk is not a problem. In this case each chunk is properly authenticated and thus can be processed safely as soon as it arrives.

[kirelagin]

I think we should not provide multi-part functions, since they are potentially dangerous and it is not really clear how they can be used in practice: if a message fits in memory, read it and use the single-message interface; if a message does not fit in memory, then there is no easy way to guarantee that it will not change as we are verifying it.