Support `zstd` compression in the `file` source

[zamazan4ik]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Right now Vector supports reading files compressed only with GZip - https://vector.dev/docs/reference/configuration/sources/file/#compressed-files . But other algorithms (like Zstd or Snappy) could be used nowadays to store compressed logs.

Attempted Solutions

Try to unpack logs externally before passing them to Vector but it's too inconvenient solution.

Proposal

Add support for zstd compression.

References

No response

Version

No response

[jszwedko]

@zamazan4ik do you have a specific compression algorithm you are interested in seeing support for? I think we should have one issue per algorithm.

[zamazan4ik]

Especially me interested in the following algorithms:

• Zstd
• Bzip
• Snappy (but in this one less interested compared to Zstd and Bzip)

Feel free to create an issue per requested algorithm :)

[jszwedko]

Thanks @zamazan4ik ! I'll repurpose this one for zstd since I think that is likely to be most popular. We can let users open issues for the others with their use-cases.

[zamazan4ik]

In my practice, additionally to GZip, I've seen logs in Zstd and Bzip formats, stored in files. Never seen logs in Snappy but could be useful to support it as well.

[dermetfan]

This looks like a duplicate of #13500

[jszwedko]

Agreed, I'll close the other one since this is more focused. Thanks for flagging @dermetfan

[mutexed]

Hello, any updates in this ticket? Interested in zstd compression for file source, may be there is some workaround?

@jszwedko

[kvlas]

Any update? I'm also interested in zstd for file source.