Open lksvenoy-r7 opened 2 years ago
Hi @lksvenoy-r7 - thanks for opening this issue, sounds very promising! A few comments & questions...
Hello @kkrugler
Some additional information regarding segment creation:
I was speaking with @mayankshriv regarding extending the segment index creation driver to allow in-memory segment creation, but unfortunately the interface implements File getOutputDirectory(), meaning we are bound to using a file.
One approach which could work is to abstract away File, and use the Pinot filesystem to allow the output to be elsewhere.
A general improvement to the architecture would be to concretely return types which have to pass through PinotFS. As it stands, since PinotFS takes a generic java URI, there's no way of knowing if a specific URI has to pass through this pattern. A good inspiration for this would be to look at the way flink handles abstracting away the file system.
The current pinot flink connector does not gracefully handle errors. Due to the way the connector works, if it errors in the middle of adding segments to a table, the table ends up with an inconsistent view. Additionally, the connector does not currently support refresh tables. Refresh tables require atomic segment replacement, but the connector currently naively uploads segments as they are built.
From testing the connector in production, I've also identified a few performance issues. These have a few different causes; The AVRO serialization is not configurable, nor is the file writing configurable (for example for different block sizes).
I have written a flink connector based on this one, but with some heavy amendments. First of all, it implements WithPostCommitTopology<GenericRecord, PinotSinkCommittable> from flink, implementing a global committer. It does work in a few different stages:
This sink is currently only compatible with REFRESH type tables that want to replace all segments on every single job execution. It takes care of atomically replacing the segments for the table, and performs well due to the way it does the hard work upfront. I am open to sharing this code so that it can be merged into the pinot repository, but it does have some limitations.