GoogleCloudPlatform / professional-services-data-validator

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match
Apache License 2.0
408 stars 119 forks source link

Teradata hash_sha256 UDF expects varchar but receives varbyte #1314

Open Deseao opened 2 weeks ago

Deseao commented 2 weeks ago

The DVT documentation says to use this UDF for row validations on Teradata. The documentation on the UDF indicates that it should be given a varchar.

However, the ibis_addon/operations.py RawSQL#format_hashbytes_teradata function is hardcoded to always invoke TransUnicodeToUTF8 before calling the UDF:

if op.how == "sha256":
        return f"rtrim(hash_sha256(TransUnicodeToUTF8({arg})))"

The Teradata documentation shows that TransUnicodeToUTF8 returns a VARBYTE(64000), not a varchar.

When I run a simple row validation against Teradata for a single column, it throws this error: [Version 20.0.0.20] [Session 240884] [Teradata Database] [Error 9881] Function 'hash_sha256' called with an invalid number or type of parameters

Is there a way to make the conversion to UTF8 happen as a calculated field that could be removed from the config file or to add a cast back to a varchar after performing it? If the conversion and hashing weren't tightly bound I could work around it.

sundar-mudupalli-work commented 1 week ago

Daseo (Alex),

Yes, you are correct. We have a fix to this issue, it was not documented. I have created a new branch issue-1314-teradata-udf-update to update the documentation and provide supplementary code for the udf. This code has been tested with another customer - but we missed including it and updating the documentation in the original pull request.

If you want to cut to the chase and go directly to what you need to do, please see README.md.

Please let us know if the instructions are correct. We will then promote these to the develop branch. Also let us know the specific Teradata version you are using.

Happy Validating!

Thank you.

Sundar Mudupalli