search

brianmhess / cassandra-loader

Delimited file loader for Cassandra
Apache License 2.0
197 stars 93 forks source link

Need to load data using TTL option #67

Closed xqchen1 closed 7 years ago

xqchen1 commented 7 years ago

Our application needs to maintain 18 months history data. When I perform one-time history data migration to Cassandra using cassandra-loaded, all the 18 months data will be loaded together and expire at almost same time since I can only set default time_to_live at table level. This will be a huge issue .

Can we add a USING TTL option? So when I load data by month, I can use a different TTL and data will expire by month instead of 18 months.

We're in the process of migrating production data to Cassandra. Could you please let me know if TTL option can be added to cassandra-loader.

Thanks Brian.

Jane

venkatayaparla commented 7 years ago

Do the data migration and then use Spark to apply TTL.

Regards Venkata Yaparla.

On Thu, Feb 9, 2017 at 8:43 PM, xqchen1 notifications@github.com wrote:

Our application needs to maintain 18 months history data. When I perform one-time history data migration to Cassandra using cassandra-loaded, all the 18 months data will be loaded together and expire at almost same time since I can only set default time_to_live at table level. This will be a huge issue .

Can we add a USING TTL option? So when I load data by month, I can use a different TTL and data will expire by month instead of 18 months.

We're in the process of migrating production data to Cassandra. Could you please let me know if TTL option can be added to cassandra-loader.

Thanks Brian.

Jane

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/brianmhess/cassandra-loader/issues/67, or mute the thread https://github.com/notifications/unsubscribe-auth/AOiUdWJ8bUWxny4EEjuk0kudouDPplQHks5ra860gaJpZM4L87u- .

brianmhess commented 7 years ago

So - you want to add a flag to provide the same TTL to all rows loaded in that invocation of cassandra-loader?

xqchen1 commented 7 years ago

Yes Brian. The application requires keeping 18 months of history data . So we have to load 18 months of client claim data to Cassandra for one time data migration. The data is extracted from Oracle by clientid and month. For example, claims_clientid_201705.csv, claims_clientid_201704.csv and so on. We want to expire the initial data each month - NOT 18 months all together. If there is a TTL flag, I can load all client files with same year/month in one load. I would have 18 different TTLs.

What will happen if all 18 months data expire on the same day?

Cassandra-loader is the best tool I've ever used to load csv files. It's easy to set up, flexible and has great performance. It's so much better than Cassandra provided tools. So please keep making it better!

Thanks for you help, Jane

brianmhess commented 7 years ago

Fixed with v0.0.27