search

a0x8o / kafka

A high-throughput, distributed, publish-subscribe messaging system
Apache License 2.0
67 stars 171 forks source link

Cast$Value doesn't transform String to Float64 #41

Open mmowris opened 1 year ago

mmowris commented 1 year ago

Problem

I'm using Kafka to take data from SQL Server and insert into Postgres - the issue is with billinglatitude [nvarchar] to billinglatitude [double precision].

I've tried to use Cast$Value to turn SQL Server '26.116184' to float64, but the column still comes across as a string. I can correctly cast SQL Server booleans [0, 1] to postgres [false, true].. how can I accomplish this for string -> double precision??

Attempt to Solve

I've tried to use billinglatitude:float64 and billinglatitude:float32, but the value still comes across as a string:

# Connector excerpt
  "transforms.castColumnsAccount.type": "org.apache.kafka.connect.transforms.Cast$Value",
  "transforms.castColumnsAccount.spec": "accepts_pos__c:boolean,billinglatitude:float32,billinglongitude:float64,biz_indicator__c:float64",
  "transforms.castColumnsAccount.predicates": "account",

# boolean cast, but nothing else cast...
"accepts_pos__c": false, # before cast: 0
"billinglatitude": "39.946713",
"billinglongitude": "-74.352108",
"biz_indicator__c": "2",

Reference docs

The Cast$Value documentation shows you can work with primitive types - string is a primitive type, so how can I take the string and cast it to a float??