search

scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra/parquet files. Alt. from DynamoDB to Scylla Alternator.
https://migrator.docs.scylladb.com/stable/
Apache License 2.0
55 stars 34 forks source link

decimal type migrations showing different results #33

Closed sangeethdba closed 3 years ago

sangeethdba commented 3 years ago

we see issues with scylla-migrator while handling decimals. for example my source columns with type decimal has data as 11 and after migration the target table data showing as 11.00000000000000

table defination has 

availability decimal,

source :

availability

11

target:

availability

11.000000000000000000

what need to done ,to see same data in both source and target

sangeethdba commented 3 years ago

any thing to do with floatingPointTolerance ?

sangeethdba commented 3 years ago

@iravid

sangeethdba commented 3 years ago

CREATE TABLE qa2.iav ( ktu int, lid text, lt text, av decimal, adsq decimal, bq bigint, mts timestamp, mu text, uav decimal, PRIMARY KEY ((ktu, lid, lt)) ); source ** ktu | lid | lt | av | adsq | bq | mts | mu | uav ------------+-------------+---------------+--------------+-----------------------------+-----------------+---------------------------------+-------------------------+------------------------- 123456 | 321 | xyz | 0 | null | null | 2020-06-23 13:43:16.092000-0400 | test | 0 ktu | lid | lt | av | adsq | bq | mts | mu | uav --------+-------------+---------------+--------------+------------------------------+-----------------+---------------------------------+------------------------0-+------------------------- 34567 | 543 | qwer | 11 | 0.176871 | 0 | 2020-08-14 17:55:15.208000-0400 | loader | 12

target*

ktu | lid | lt | av | adsq | bq | mts | mu | uav ------------+-------------+---------------+--------------+------------------------------+-----------------+---------------------------------+-------------------------+------------------------- 123456 | 321 | xyz | 0E-18 | null | null | 2020-06-23 17:43:16.092000+0000 | test | 0E-18

ktu | lid | lt | av | adsq | bq | mts | mu | uav --------+-------------+---------------+-----------------------+------------------------------+-----------------+---------------------------------+-------------------------+------------------------- 34567 | 543 | qwer | 11.000000000000000000 | 0.176871000000000000 | 0 | 2020-08-14 21:55:15.208000+0000 | loader| 12.000000000000000000

these columns has the data changed as below

av decimal, source 0 target 0E-18 source 11 target 11.000000000000000000 adsq decimal. source null target null source 0.176871 target 0.176871000000000000 uav decimal, source 0 target 0E-18 source12 target 12.000000000000000000

sangeethdba commented 3 years ago

@iravid FYI

sangeethdba commented 3 years ago

@iravid can you hep

sangeethdba commented 3 years ago

any help is apprecaited

iravid commented 3 years ago

Hi @sangeethdba - please pull the latest master version and add stripTrailingZerosForDecimals: true to the target section of your config.yaml . That should solve the issue.