search

tsani / apollo

HTTP MPD integration for Labcoders Radio
MIT License
4 stars 0 forks source link

The /archive endpoint is slow #11

Closed tsani closed 7 years ago

tsani commented 7 years ago

First, the zip archive constructed from the audio files doesn't need to be compressed, since the audio files themselves are compressed. Compressing each file is slow, whereas simply storing it would be faster.

Second, the construction of the zip archive does not operate in a nicely bounded amount of space. Ideally, we would want the space usage of the archival process to be constant. I think that we can get it to operate in constant space, since the CRC32 checksum of the files can be computed in constant space by streaming the file into the checksum function, and the file can be read again when it is to be written into the archive, uncompressed. Both of those operations run in constant space, so overall it should run in constant space. This probably means that the zip-archive library is doing something silly.

tsani commented 7 years ago

It's so slow that nginx times out reading from upstream, because no headers are sent until the archive is fully processed.

tsani commented 7 years ago

Async archiving is implemented in Apollo HEAD, and used in apollo-export.