scylladb / kafka-connect-scylladb

Kafka Connect Scylladb Sink
Apache License 2.0
42 stars 22 forks source link

support for record to table mapping config #6

Closed patitapaban19 closed 4 years ago

patitapaban19 commented 4 years ago

Added support for the below configs:

  1. "topic.my_topic.my_ks.my_table.mapping": "col1=key.f1, col2=value.f1, ttl=value.f2, timestamp=value.f3, col3=header.f1"
  2. "topic.my_topic.my_ks.my_table.consistencyLevel": "LOCAL_ONE"
  3. "topic.my_topic.my_ks.my_table.ttlSeconds": 1
  4. "topic.my_topic.my_ks.my_table.deletesEnabled": "true"

Which will address the below:

Note: We are supporting only primitive data type to be mapped to table while extracting values from Kafka record header.

tarzanek commented 4 years ago

lgtm

flucchese commented 4 years ago

I'm missing a more detailed description of what this PR is about. What problem/gap is it addressing? What is the strategy?

Notice also that the commited code uses a distinct tabbing standard, so many lines iwth no real modification are modified. Please stick to the original standard (4 spaces) and/or submit another PR with a global standard change.

patitapaban19 commented 4 years ago

I'm missing a more detailed description of what this PR is about. What problem/gap is it addressing? What is the strategy?

Notice also that the commited code uses a distinct tabbing standard, so many lines iwth no real modification are modified. Please stick to the original standard (4 spaces) and/or submit another PR with a global standard change.

@flucchese Have added the description for the pr. Please have a look. Have used spacing as 2 for the connector but will raise a new pr with standard spacing after the above pr is merged.

patitapaban19 commented 4 years ago

My difficulty here is that there are several somehow unrelated changes squeezed into a single PR. Normally I would prefer incremental changes only, focused on one aspect of the code. Since I'm planning to conduct a major revision in the near future, i'm ok with these changes for now, making it clear that I haven't yet worked deeply into finding bugs or inconsistencies.

@flucchese This pr only contains the topic to mapping related configs support. We are raising pr after related configs support is provided.