Snowflake-Labs / schemachange

A Database Change Management tool for Snowflake
Apache License 2.0
481 stars 218 forks source link

Adding Skip Failures parameter for proceeding to the next script even if the previous script failed #239

Open zroytman opened 3 months ago

zroytman commented 3 months ago

Is your feature request related to a problem? Please describe. I'm using SchemaChange as part of Azure DevOps Pipeline where I'm trying to deploy several sql scripts. In most of my cases the scripts have no dependencies between each other, but SchemaChange stops deployment on the first failure and doesn't proceed with the other scripts.

Describe the solution you'd like Adding new parameter "Skip Failures" Y/N (True/False) will be helpful in that case, so the deployment process will deploy all the possible scripts without stopping after the first failure.

Thanks.

opodoprigora commented 2 months ago

Are these V scripts? Issue with this potential parameter will be that if you run the scripts like so:

And later edit V1.2.0__file.sql to pass it will not be executed since its not the max version in the change history table.

In either case, parameter aside, a workaround solution can be to structure these scripts as tasks and then call them either manually or via an A script. Since tasks are executed asynchronously your calls should always be successful.

zroytman commented 2 months ago

Are these V scripts? Issue with this potential parameter will be that if you run the scripts like so:

  • V1.1.0__file.sql - pass
  • V1.2.0__file.sql - fail
  • V1.3.0__file.sql - pass

And later edit V1.2.0__file.sql to pass it will not be executed since its not the max version in the change history table.

In either case, parameter aside, a workaround solution can be to structure these scripts as tasks and then call them either manually or via an A script. Since tasks are executed asynchronously your calls should always be successful.

Hi, thanks for the response. No, these are R__ scripts. The reason I asked for such a parameter is to give us the possibility to deploy all the scripts with no stopping after the first failure - means, deploy as much as possible. After that I will fix the failed script and rerun the deployment - it will deploy the only non-deployed yet scripts (means, my fixed one).

sfc-gh-tmathew commented 2 months ago

239 and #230 are duplicates. So when the community decides to work on this enhancement, we will address both open issues. Thank you for raising the issue and helping us understand the problem.

We have other priority items we are trying to address and will get to this if time permits.

zroytman commented 2 months ago

@sfc-gh-tmathew Thank you so much. Actually the 230 was the question while the 239 is the request for enhancement. Of course you can merge them.

zroytman commented 1 month ago

239 and #230 are duplicates. So when the community decides to work on this enhancement, we will address both open issues. Thank you for raising the issue and helping us understand the problem.

We have other priority items we are trying to address and will get to this if time permits.

Dear @sfc-gh-tmathew , any news regarding this enhancement? As we're using the schemachange massively during the recent period, we're facing many issues due to the fact that the deployment stops running on the first failure. We are many developers that work in parallel on different objects. Once one of the developers causes error in some of his objects, all other developers cannot deploy their objects till the errored one is solved. But in real, there is no any dependency between this errored one and other objects of other developers. So this enhancement is very critical for us. Thanks a lot.

sfc-gh-tmathew commented 1 month ago

@zroytman we are working on streamlining the release and testing process so that community contributions don't break existing functionality. This is not something that I am working on right now.

FWIW, I worked around for a personal project like this in my devops pipeline using github actions.

# Logic before schemachange block
set +e
echo "Preventing Schemachange's exit code from exiting the shell"
schemachange --config-folder myproject
RESULT=$?
if [ $RESULT -eq 0 ]; then
    echo "Schemachange deployment completed successfully"
else
   echo "Schemachange deployment failed. Proceeding to exception handling"
fi
set -e
# logic after schemachange block

if [ $RESULT -eq 0 ]; then
   echo "Completed Successfully"
else
  echo "Deployment failed"
  exit 1 # do this if you want to control how you exit the pipeline. if you choose to exit 0 then the pipeline will see it as a success.
fi