io.eels.component.parquet.ParquetWriterConfig should use ConfigLoader extensions

[hannesmiller]

For example I want to turn off compression for Parquet writing and the only way I could get it to work is by creating a reference.conf with the following contents:

eel.parquet.compressionCodec=UNCOMPRESSED

[sksamuel]

You want a programmatic option too?

On 5 Jun 2017 7:03 pm, "hannesmiller" notifications@github.com wrote:

For example I want to turn off compression for Parquet writing and the only way I could get it to work is by creating a reference.conf with the following contents:

eel.parquet.compressionCodec=UNCOMPRESSED

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/51zero/eel-sdk/issues/274, or mute the thread https://github.com/notifications/unsubscribe-auth/AAtZGkpgsODg-SYEUoZS1uwXmgDymZSEks5sBELhgaJpZM4NwXpn .

[hannesmiller]

Yes that would be cool to have at a per sink level

[sksamuel]

I don't understand your original request. What's the error/issue with using eel.parquet.compressionCodec=UNCOMPRESSED

[hannesmiller]

I can override the compression property by having my own "reference.conf" but "env.conf" for example doesn't work?

I have checked this through the debugger and think the WriterConfig class uses just the stars typesafe config without your extensions?

[hannesmiller]

Sorry meant standard typesafe config

[sksamuel]

I don't want to update eel to use my config extensions library as that's a library I don't mind using on closed source projects, but I don't want to force anyone using eel to also have to use that library.