search

kite-sdk / kite

Kite SDK
http://kitesdk.org/docs/current/
Apache License 2.0
394 stars 263 forks source link

CDK-991: Add size-based and time-based file rolling. #386

Closed rdblue closed 9 years ago

rdblue commented 9 years ago

File size-based rolling works and is passing a new test for Avro, but is disabled for Parquet because the appender has no reliable size estimate for Parquet.

Time-based rolling uses a new SPI interface, ClockReady, which exposes a method for passing time signals to implementing classes. This removes the need for Kite to provide a thread-based check.

mkwhitacre commented 9 years ago

Changes look good. +1

rdblue commented 9 years ago

Thanks @mkwhitacre!

I saw your question about when something wouldn't be ClockReady, which I think has been removed... but just in case the idea was that not all of the writers will be time-based. HBase, for example, doesn't need to be right now.

mkwhitacre commented 9 years ago

yeah some of it seemed to be answered with the second commit (which I hadn't gotten to yet) :smile: