Deployment via Jenkins not being assigned to an environment

[bglee]

This issues deals with using a Jenkins script to deploy a new version of an application with a new version of a component.

Currently this is what the environments Assigned Applications tab looks like:

You can see that the hlf-network application is assigned. (For some reason the hlf-network;0_1_0 is also assigned even though it is not a base version. It should be a version of hlf-network. I have tried removing it but it can not be removed.)

This is the processes used to manually deploy an application version:

On the application: Go to Assigned Environments -> Click + -> Click small-dev-1

This is the resulting Assigned Environments tab of the application version:

This is the resulting Assigned Applications tab of the environment:

This works. When I try to deploy it uses the right version of the ca (The first component of the app).

INFO: Starting deployment #1123
--
INFO: Deploying Application "hlf-network;1_4_1_25" into Environment "small-dev-1"
INFO: Deploying Component ca;1_4_1_100
INFO: Starting Sequential Deployment of Component ca;1_4_1_100
INFO: Deploying Component ca;1_4_1_100 using Custom Action helm_install_action
Creating Values TOML File

This is the script with envs set to the same environment. I am assuming it should set them like it did with the previous example:

But no environments are set on the application version that is currently deploying. Furthermore it tries to use the ca version in 0_1_0_25 while still having the build logs under the new application 1_4_1_26. The version of the ca for this build should be 1_4_1_101.

INFO: Starting deployment #1124
--
INFO: Deploying Application "hlf-network;1_4_1_26" into Environment "small-dev-1"
INFO: Component 'ca;1_4_1_100' is already present on server 'small-dev-1-hlf-org1'
INFO: Component 'ca;1_4_1_100' is already present on server 'small-dev-1-hlf-org2'
INFO: Starting Sequential Deployment of Component cli;1_4_1_117
INFO: Deploying Component cli;1_4_1_117 using Custom Action helm_install_action
Creating Values TOML File

Then when it completes it completely removes the application version from the environment so we are back where we started.

Please let me know if any more details are required.

CC: @piyush94 , @svisagan83

[sbtaylor15]

@bglee is the application associated to the Environment correctly after the Create HLF App Version stage of the job completes and before the deployment stage executes. Trying to narrow down where the environment association is getting lost. Also, if the app is associated to the environment correctly then do a deployment from the web UI and see if it deploys correctly.

CC: @piyush94 , @svisagan83

[bglee]

@sbtaylor15 - It is on application creation. I commented out the Deploy HLF Step and I even tried commenting out the addCompVer2AppVer. The new application still did not get assigned to the environment. CC: @piyush94 , @svisagan83

[sbtaylor15]

@bglee the environments get associated to the base version of the app. What is the base version name and the new version name?

CC: @piyush94 , @svisagan83

[bglee]

@sbtaylor15 the base version is hlf-network the new version is hlf-network;1_41$BUILD_NUMBER.

I can see the environment gets assigned under the base version:

And I can see in the environment that the base version is there:

And when I manually assign a version to an environment it shows up in both places as the current version:

CC: @piyush94 , @svisagan83

[sbtaylor15]

@bglee when you manually assign an application to an environment it tells DeployHub that the version has been deployed to that environment "outside" of DeployHub's process. This is used for the roll-forward rollback process. In your case, DeployHub will assigned the application version to the environment when a successful deployment has been performed. Remove the application version from the environment but keeping the base version associated and do a deployment from the UI. Let us know if that deploys correctly.

CC: @piyush94 , @svisagan83

[sbtaylor15]

@blgee it looks like the application is trying to roll forward from the base version. Deploy always is causing incorrect version to be deployed. We are in the process of reworking the logic in the Deployhub groovy to account for this.

[bglee]

@sbtaylor15 To try to reproduce this from DeployHub I decided to clean out the environment completely. I think I now understand the two different things that were happening that caused this to appear as an issue.

1) The Deployment tries to deploy the component version(s) of the previous application version that is assigned to the environment before deploying the new version. This may be intended behavior.

If I have always deploy set to "N" this is skipped and the new application version is deployed. My previous version of the application was broken, manually assigned, and had some components with Always Deploy "Y". This caused the new application version to never deploy. So I never saw the new component.

In my simpler example it succeeds:

INFO: Starting deployment #1147
--
INFO: Deploying Application "hlf-network;2" into Environment "small-dev-1"
INFO: Component 'ca;1' is already present on server 'small-dev-1-hlf-org1'
INFO: Component 'ca;1' is already present on server 'small-dev-1-hlf-org2'
INFO: Deploying Component ca;2
INFO: Starting Sequential Deployment of Component ca;2
INFO: Deploying Component ca;2 using Custom Action helm_install_action

2) The second issue was caused by the way I had manually created an earlier application version.

This caused me to have two base versions:

Even though I had removed the application version from the real base version it was still falling back to the other base version that I could not remove. That base version had components set with always deploy "Y" (see 1).

I am currently trying to recreate the the second issues circumstances.

CC: @piyush94 , @svisagan83

[sbtaylor15]

@bglee do you want us to skip over deploying the previous versions and only deploy the latest one? We can make this happen by assigning the latest version to the environment before doing the deployment. Deploy Always would need to be turned on to force a deployment of the latest version. Let us know how you would like to proceed.

CC: @piyush94 , @svisagan83

[piyush94]

@sbtaylor15 - I also did some testing around this, i found that component version roll forwarding is happening and i think it's part of DeployHub's design and makes sense if some previous versions have not been deployed. But my concern is, if due to some reason the previous component version(s) were defective and earlier deployments had failed, it still tries to deploy these versions before deploying the latest version which might have include fixes and causes this deployment to also fail. So, i feel there should be a way to prevent roll forward, in a case of defective older component versions and only deploy the latest version.

Even, if above case is not the scenario, it would be good to have a option to deploy only the latest application version from both the UI and Jenkins.

CC: @bglee , @svisagan83

[sbtaylor15]

@piyush94 ok, we will update the groovy to skip the roll forward. I think this should be fine since the components that you are working with don't rely on incremental updates.

CC: @bglee , @svisagan83

[bglee]

@sbtaylor15 - Yes with Helm Charts incremental updates are not necessary. Each Helm version should always be self contained.

The one thing I caught in your message earlier is that Always Deploy will have to be set to "Y." Is this due to the fundamental flow of DeployHub?

CC: @piyush94 , @svisagan83

[sbtaylor15]

@bglee the Always Deploy = 'Y' is basically enables DeployHub to "force" a deployment even in the version already exists on the end point. The process that we will change the DeployHub groovy code to do is, set the version being deployed to the environment to be the new version coming from the Jenkins pipeline. Then with Alway Deploy set to Y we will then deploy the new application version. DeployHub will skip any incremental processing and just deploy the new application version. Basically, we are turning off the incremental processing that DeployHub does by default.

CC: @piyush94 , @svisagan83

[bglee]

@piyush94 Do you think this will work for us? I want to make sure that it doesn't interfere with the current processes that may be counting on roll forward.

CC: @sbtaylor15 , @svisagan83

[sbtaylor15]

@piyush94 we will make it optional so things like database updates that are incremental can still be handled.

@bglee, @svisagan83

[sbtaylor15]

@bglee we changed the groovy code to assign the new component to the environment just like you would do manually in the UI. The deployment should just deploy the new version and skip over the previous versions (ie do a roll forward). @piyush94 there is a new method in the groovy library, setConfig which can be used to turnoff this new change. See the groovydoc. Please pull the latest image. The groovy master branch is up to date.

cc: @svisagan83

[bglee]

@sbtaylor15 - I have confirmed the fix works from my end. After a failed deployment the application becomes unassigned form the environment. The deploy process does not try to roll back through all the deployments. It only deploys the most recent application version. Further more with Always Deploy "N" is skips the components already deployed to the environment. This behavior is what I believe is appropriate for Helm deployments.

I would like @piyush94 to confirm the fix works for his case before I close the issue.

cc: @svisagan83

[piyush94]

@bglee - I have verified from my end also. You can close this issue. Thanks.

CC: @sbtaylor15 , @svisagan83

[piyush94]

@sbtaylor15 - Hi Steve, how can I prevent the roll-forward from the DeployHub UI for helm chart components?

[sbtaylor15]

@piyush94 are the components set to Always = Y or N?

[piyush94]

@sbtaylor15 Always Deploy is set to N. I was able to deploy using this script.

@Library('deployhub') _

def dhurl="";
def dh = new deployhub();

node {
stage('dh sid ras') {
    withCredentials([usernamePassword(credentialsId: '', passwordVariable: 'pw', usernameVariable: 'user')])
    {
      def String[] env = [""];
      def depenv = "";
      def app = "";
      def appVer = "";

      data = dh.newApplication(dhurl, user, pw, app, appVer, env);
      echo "$data";
      def appid = data[0].toString();
      data = dh.deployApplication(dhurl, user, pw, appid, depenv);
      if (data[0])
      {
        def deploymentid = data[1]['deploymentid'];
        echo "Deployment Logs for #$deploymentid";
        // get deployment logs
        data = dh.getLogs(dhurl,user,pw, "$deploymentid");
        echo data[1];
        // check if deployment was successful or not
        if(data[0]) {
          echo "Deployment was successful";
        }
        else {
          echo "Deployment was not successful";
          // this line is to set build status to failure
          error 'Deployment failed';
        }
      }
    }
  }
}

[sbtaylor15]

@piyush94 we looked at the code and both the groovy script and the UI use the same call to do the deployment. The logic is to put the component version being deployed on the endpoint, this will tell the deployment engine to skip the roll forward process. But this only gets turned on if the component is set to Always Deploy = Y.

Try assigning the component version to the endpoint in the UI and then do a deployment from the UI and see if you get the same result as the groovy script.

We are going to do some more debugging on this.

[piyush94]

@sbtaylor15 - We need Always Deploy = N. Is it possible? The scenario is that, there is a later application version that runs a few component versions behind from the application version from which it was created. I feel roll forward should happen if we deploy new component version, but it is happening with older component versions also.

[sbtaylor15]

@piyush94 in the UI assign the -1 version of the component to the end point and then do a deployment. This should enable the version you want to be deployed even with the Always Deploy = N.

Let us know if that solves your scenario.