search

ptgoetz / storm-hdfs

Storm components for interacting with HDFS file systems
Apache License 2.0
61 stars 59 forks source link

Adding Time Rotation Policy #11

Closed evidanary closed 10 years ago

evidanary commented 10 years ago

This is my first time contributing to an open source project. Please do let me know if something seems off.

ptgoetz commented 10 years ago

@yash-ranadive Thank you for the contribution! What you implemented was exactly how I was initially going to do it.

However, after giving it some thought, there's a problem with how FileRotationPolicys work with respect to time. If you have a slow data source (i.e. you only get tuples every now and then), the file rotation may not occur at the intended time (since it is triggered by the arrival of a tuple).

So I implemented it a little bit differently, in a way that doesn't rely on the arrival of a tuple to trigger the rotation.

I'm going to close this PR, but thank you for taking the time to contribute!

evidanary commented 10 years ago

Great! How did you implement it? When can we see it merged?

ptgoetz commented 10 years ago

With a dirty use of instanceof. ;)

I also added some synchronization since the rotation could be triggered in a different thread.

Check the latest commit.