search

google-deepmind / language_modeling_is_compression

Apache License 2.0
101 stars 14 forks source link

Questions regarding the mask_fn #10

Closed hui-po-wang closed 9 months ago

hui-po-wang commented 9 months ago

Dear authors,

Thanks so much for the amazing work!

I have some questions regarding the mask function. During compression, you used a mask function to pre-process data. For instance, a right_shift_bytes_by_one function is used to shift the chunked data by 1 bit.

I wonder what's the purpose of it, and is it still a lossless compression even if the LSB is discarded?

Thanks for your time.

anianruoss commented 9 months ago

We describe this in detail in appendix B.2 of our ICLR 2024 paper. Please let me know if you have any further questions.

anianruoss commented 9 months ago

Also, see https://x.com/anianruoss/status/1704615421280649436

anianruoss commented 9 months ago

But you are right: If you want to implement a truly lossless compressor, you'll have to account for the masked bit. We do this when computing the compression rate.