search

pinterest / secor

Secor is a service implementing Kafka log persistence
Apache License 2.0
1.85k stars 540 forks source link

Unable to set Compression config when using AvroWriter #1725

Open richiesgr opened 3 years ago

richiesgr commented 3 years ago

Hi I would like to add more on this because I check it now The problem is that Avro doesn't support Gzip compression by default. The problem is that Avro and the message writer both use the same configuration params: secor.compression.codec So if you put org.apache.hadoop.io.compress.GzipCodec you get an exception because Avro writer try to use it and fail If you put null (no compression) so the MessageWriter fail with an exception because it try to instance a class with that If put empty it fails because there you must put value

So I don't know what to put here because I try to Big query to read the files and don't thing it support something else than Gzip or uncompressed. and you can't set neither !!

Originally posted by @richiesgr in https://github.com/pinterest/secor/issues/482#issuecomment-735748818

HenryCaiHaiying commented 3 years ago

Richard,

If you need to set those two params separately, you can modify the secor code to introduce two separate params.

On Mon, Nov 30, 2020 at 4:14 AM Richard Grossman notifications@github.com wrote:

Hi I would like to add more on this because I check it now The problem is that Avro doesn't support Gzip compression by default. The problem is that Avro and the message writer both use the same configuration params: secor.compression.codec So if you put org.apache.hadoop.io.compress.GzipCodec you get an exception because Avro writer try to use it and fail If you put null (no compression) so the MessageWriter fail with an exception because it try to instance a class with that If put empty it fails because there you must put value

So I don't know what to put here because I try to Big query to read the files and don't thing it support something else than Gzip or uncompressed. and you can't set neither !!

Originally posted by @richiesgr https://github.com/richiesgr in #482 (comment) https://github.com/pinterest/secor/issues/482#issuecomment-735748818

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1725, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP74PTN7GEZNZSGCGKXTSSOECVANCNFSM4UHPPU4Q .

richiesgr commented 3 years ago

Hi Yes I've already made the modification. Don't know if it's something that can interest someone for a PR If yes I can push a PR

HenryCaiHaiying commented 3 years ago

I don't think it will hurt to introduce that parameter, feel free to submit a PR.

On Tue, Dec 1, 2020 at 1:01 AM Richard Grossman notifications@github.com wrote:

Hi Yes I've already made the modification. Don't know if it's something that can interest someone for a PR If yes I can push a PR

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1725#issuecomment-736326329, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP74AZISWO3HE2JTZGR3SSSWINANCNFSM4UHPPU4Q .