search

awslabs / simple-database-archival-solution

Simple Database Archival Solution (SDAS) is an open source solution which you can deploy in your AWS account to archive data to AWS. SDAS provides organizations with an efficient, easy and cost-effective solution for archiving Oracle, Microsoft SQL, PostgreSQL, and MySQL databases.
Apache License 2.0
39 stars 8 forks source link

Archived data validation process issues #53

Open marsoy opened 2 months ago

marsoy commented 2 months ago

In data validation observed following issues

ruiscosta commented 2 months ago

Thank you for reaching out. Could you please confirm which source database you are using where the issue "Archived even though archive jobs failed for some tables" is occurring?

marsoy commented 2 months ago

I have tried with MySQL and PostgreSQL source databases. validation_completed is having higher count than validation_count in most of the cases.

ruiscosta commented 2 months ago

fixed: Archive status stuck in validating even though all validation job are marked successful and vice versa where its marked Archived when few validation jobs are still running

ruiscosta commented 2 months ago

Can you share any errors for Archive status marked Archived even though archive jobs failed for some tables?

marsoy commented 2 months ago

Can you share any errors for Archive status marked Archived even though archive jobs failed for some tables?

The issue was related to Glue job failure due to incorrect schema conversion. The issue that was faced with integer values having 8 byte value getting converted into Large for Glue but SDAS schema conversion was marking them int.

Error: Unsupported case of DataType: com.amazonaws.services.glue.schema.types.IntegerType@438e8695 and DynamicNode: longnode