Snowflake-Labs / schemachange

A Database Change Management tool for Snowflake
Apache License 2.0
502 stars 224 forks source link

How do I make schemachange skip failures during deployment and proceed to the next script? #230

Open zroytman opened 7 months ago

zroytman commented 7 months ago

What are you trying to accomplish? I'm using SchemaChange as part of Azure DevOps Pipeline where i'm trying to deploy several sql scripts. In most of my cases the scripts have no dependencies between each other, but SchemaChange stops deployment on the first failure and doesn't proceed with the other scripts. Is there any option/flag/configuration that can provide me such a behavior?

Example: 3 scripts changed and are being deployed by schemachange. script 1 successfully deployed - schemachange tries the next one (2). script 2 failed on some sql error - i expect from schemachange to move to the next script (3) but it stops here.

What options have you tried so far ? Tried to find some flag or option of configuration in SchemaChange - nothing found.

sfc-gh-tmathew commented 5 months ago

@zroytman each file represents a change to the database and hence it is not recommended to support skip errors for Versioned files. The skip for R and A files makes sense but we want to be consistent with completing the run without errors.

The right approach is to avoid publishing the DDL files that contain errors and pushing them in a new iteration.

zroytman commented 5 months ago

@zroytman each file represents a change to the database and hence it is not recommended to support skip errors for Versioned files. The skip for R and A files makes sense but we want to be consistent with completing the run without errors.

The right approach is to avoid publishing the DDL files that contain errors and pushing them in a new iteration.

That's why I'm asking to add this optional parameter "Skip Errors - Y/N", so the decision will be upon us. In our case I'm deploying three views: v1, v2, v3. In case of failure in v2, the v3 will not be deployed. I prefer to deploy as much as possible. At the end of the deployment process I'll fix the issues regarding v2 and rerun the deployment, so the only v2 will be deployed at this time. There is no dependency between the views - all of them, v1, v2, v3 are standalone views.

This optional parameter "Skip Errors - Y/N" will help very much in those cases. And again, not all of the schemachange users should use it - for them it will be N as a default value. In our case I will define it as Y.

sfc-gh-tmathew commented 5 months ago

@zroytman Thank you for clarifying.

We are in the process of improving stability of the code base and would like to seek community to help out. We will have to make a few iterations before it makes it to a release.

Would you be able to do the following to add this feature.

  1. The feature should apply to R and A files only.
  2. Failed R and A files will be recorded in Change History
  3. When rerun, the failed files will remain in the change history for tracking purposes but will have a new entry for successful execution.
  4. Default for this switch will be set to N to behave as before.
  5. Documentation needs to be updated to reflect the new switch
  6. Add test cases to cover for this feature request

Open a PR with these changes and we can review and include it an appropriate release version.

zroytman commented 5 months ago

@sfc-gh-tmathew Thank you so much. Additional clarification: In case there was at least one failure during the deployment, the deployment should proceed till the last script, but the whole deployment batch should be marked as "Failure" in order to make us, users, aware about some error occured during the deployment, If all the scripts successfully deployed, the batch will be marked as "Success".

Unfortunately I cannot help in developing of this feature as this is out of my knowledge and skills, so dear community, please help.

By the way, previously I've opened the dedicated enhancement ticket for this purpose, #239 .

zroytman commented 3 months ago

Hello dear community! Any updates regarding this enhancement? Thanks a lot!

zroytman commented 1 month ago

Dear community, any updates regarding this enhancement? Thanks a lot!