search

erezsh / reladiff

High-performance diffing of large datasets across databases
https://reladiff.readthedocs.io/en/latest/index.html#
Other
366 stars 9 forks source link

[CLI] Regression: SQL Compilation Error with Quoted Column Names in Snowflake in Version 0.5.2 #30

Closed alex-mirkin closed 2 months ago

alex-mirkin commented 2 months ago

In version 0.5.1, any database comparisons with Snowflake worked correctly. However, after upgrading to version 0.5.2, I encountered a SQL compilation error when comparing any database to Snowflake. Error Message:

ERROR - 000904 (42000): SQL compilation error: error line 1 at position 21
invalid identifier '"id"'
erezsh commented 2 months ago

Hi @alex-mirkin , sorry to hear there has been a regression!

Reladiff does test snowflake as part of its CI, and looking through the changes between 0.5.1 and 0.5.2, it's not obvious to me what could cause this error to arise.

So, it would be helpful if you could give me more information: How do you run reladiff when you get this error? Can you run it with -d -v and paste the output here? (or at least whatever part of it that you can share)

alex-mirkin commented 2 months ago

Running with defaults, only adding extra columns parameters and not specifying the pk column (which is id).

The query that produced the error in 0.5.2:

SELECT cast(cast(min("id") as decimal(38, 0)) as string), cast(cast(max("id") as decimal(38, 0)) as string) FROM "FOO"

Same query on 0.5.1 that works:

SELECT cast(cast(min("ID") as decimal(38, 0)) as string), cast(cast(max("ID") as decimal(38, 0)) as string) FROM "FOO"

Looks like a column name case issue.

erezsh commented 2 months ago

It does look like a name case issue. However, I'm not able to reproduce it here. Can you give me more information on what you're doing? Maybe a minimal example that reproduces the issue?

Are you running it from the CLI or through the Python API? If you're using the Python API, are you using case_sensitive=False option?

p.s. is it happening when you diff mysql<->mysql or snowflake<->snowflake, or only when doing mysql<->snowflake?

erezsh commented 2 months ago

Nevermind, I managed to reproduce it. I will update soon.

erezsh commented 2 months ago

It was a bug I accidentally introduced in the recent version. I apologize for the inconvenience!

I added tests so this particular regression won't happen again.

I will make new release later today with these fixes. Please test the latest master if you can, and make sure everything is now working as you expect.

alex-mirkin commented 2 months ago

Works on my end, thanks for the quick response!

erezsh commented 2 months ago

Happy to help. Let me know if you run into any more bugs!

P.S. v0.5.3 is now released, including these fixes.