The format only supports seeking within the decompressed stream. With better native compression, the Brotli "wrapper" would not be necessary and the format could seek instead within the compressed stream. I believe the primary problem with sizes today is the verbose encoding of code tables.
Quoted from @dominiccooney
(I don't understand the last sentence)