Jenkins groovy issue

[piyush94]

jenkins_log.txt

Jenkinsfile.txt

[sbtaylor15]

@piyush94 - I simplified the groovy code to make the component variant optional. Here is a snippet of the code I was testing with. This is in line with the versioning format that you are currently using.

You will need to fully qualify the component name using a dot between each domain level. This is needed so we can place the component in the correct domain.

      def compname = "GLOBAL.American University.CSC589.chili.Uptime Dashboard XXX";
      def compvariant = "";
      def compversion = "v1.5.0";

      data = dh.newComponentVersion(url,user,pw, compname, compvariant, compversion);
      echo data.toString();
      def attrs = [buildnumber: env.BUILD_ID];
      data = dh.updateComponentAttrs(url,user,pw, compname, compvariant, compversion ,attrs);
      echo data.toString();

Please delete old components from the DeployHub UI and start fresh. I tested the following scenarios:

1. No component exists - results in new base version and version 1 with the job number associated to version 1
2. Rerun the same pipeline without any changes - results in version 1 having the job number updated
3. Use "July Release" as Component Variant and Component Version = "v1.5.0-" + env.BUILDID - results in new base version and version 1 with the job number associated to version 1 and version = v1_5_0_1260
4. Rerun test 3 - results in version 2 with the job number associated to version 2 and version = v1_5_0_1261

[sbtaylor15]

CC: @svisagan83

[piyush94]

@sbtaylor15 I am still not able to create version and update attributes.

Jenkinsfile

def comp="GLOBAL.Test_Project.Test.testapp"
// def app="AppUI_Helm"
def environment="ustr-k8s"
// def cmd=""
def url="https://us-deployhub.na.uis.unisys.com:7171"
def user="dhguest"
def pw=""
echo "${url}";
echo "${version}";

// create component version
// def newComponentVersion(String url, String userid, String pw, String compname, String compvariant, String compversion)
data = dh.newComponentVersion(url, user, pw, comp, "", version);
echo "Creation Done " + data.toString();

// // update attrs
def attrs = [Chart: "harbor-lib/"+imagename, buildnumber: env.BUILD_ID, chartversion: version];
echo "${attrs}";
// // def updateComponentAttrs(String url, String userid, String pw, String compname, String compvariant, String compversion, Map Attrs)
data = dh.updateComponentAttrs(url, user, pw, comp, "", version , attrs);
echo "Update Done " + data.toString();

Jenkins Log

[Pipeline] stage
[Pipeline] { (Deploy)
[Pipeline] echo
https://us-deployhub.na.uis.unisys.com:7171
[Pipeline] echo
0.1.0-103
[Pipeline] echo
Creation Done 141
[Pipeline] echo
[Chart:harbor-lib/app-ui-helm, buildnumber:103, chartversion:0.1.0-103]
[Pipeline] echo
Update Done null
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline

I have created a new component for the testing

Am i missing something?

CC: @svisagan83

[sbtaylor15]

Did testapp exist already before you ran your pipeline or did it get created?

[sbtaylor15]

@piyush94 - how do you have the security setup for dhguest?

[sbtaylor15]

@piyush94 - The chartversion was causing issues. We added that in as a special attribute to catch and push to the docker component item once we recognized that helm needed the chart version passed in as a parameter.

Please pull the latest image.

CC: @svisagan83

docker pull quay.io/deployhub/deployhub-pro@sha256:e556ab9441d46ef1067f8507b14a62633aaa1e24cc4ffd87cda7dd83382d5547

[sbtaylor15]

Delete the testapp component and let the Jenkins pipeline create it. The pipeline should create testapp and testapp;0_1_0_103. testapp;0_1_0_103 should have the lastbuild number set and the docker component item should have the chart and chart version listed.

[piyush94]

@sbtaylor15 It is working now. But i have some doubts:

1. Do we not have to set component attributes anymore for helm charts, if we have a docker item with details like this:
2. Is the above item working?
3. Does the create new version step applicable to existing components?
4. How do we set the new component version in the application and do deploy?

CC: @svisagan83

[sbtaylor15]

Need to inherit from base version or latest

• Build Job
• Component Type
• Actions
• Deploy Always

[sbtaylor15]

List of Attributes for component and component item

[sbtaylor15]

Attributes and properties for the component

• BuildId - build id from the build system such as Quay
• BuildUrl - build url for the build system
• Chart - helm chart for the component
• ChartVersion - version of the helm chart
• operator - Kubernetes operator
• DockerBuildDate - timestamp for the docker build
• DockerSha - sha for the docker image
• DockerRepo - url for the docker registry
• GitCommit - git commit that trigger the build
• GitRepo - git repo name
• GitTag - git tag such as master or v1.5.0
• GitUrl - url to the git repo
• buildnumber - build job number for Jenkins
• buildjob - build job name for Jenkins
• ComponentType - name of the component type
• ChangeRequestDS - name of the Change Request Datasource
• Category - name of the components category
• AlwaysDeploy - Y/N
• DeploySequentially - Y/N
• BaseDirectory - base directory for the component
• PreAction - name of the pre-action
• PostAction - name of the post-action
• CustomAction - name of the custom-action
• Summary - component summary

[sbtaylor15]

// // update attrs
def attrs = [
         BuildId: env.BUILD_ID,
         BuildUrl: env.BUILD_URL,
             Chart: "harbor-lib/"+imagename, 
         chartversion: version,
         operator: "Kubernetes operator",
         DockerBuildDate: "timestamp",
         DockerSha: "sha for the docker image",
         DockerRepo: "url for the docker registry",
         GitCommit: "git commit",
         GitRepo: "git repo",
         GitTag: "git tag",
         GitUrl: "git url",
         buildnumber: env.BUILD_ID, 
         buildjob: "GLOBAL.test-project",
         ComponentType: "Wildfly_EP",
         ChangeRequestDS: "GLOBAL.JiraUnisys",
         Category: "General",
         AlwaysDeploy: "N",
         DeploySequentially: "N",
         BaseDirectory: "tmp",
         PreAction: "GLOBAL.Run_SQL_Script_Postgres",
         PostAction: "GLOBAL.Run_SQL_Script_Postgres",
         CustomAction: "GLOBAL.Run_SQL_Script_Postgres",
         Summary: "Pipeline Comp"
        ];
echo "${attrs}";
// // def updateComponentAttrs(String url, String userid, String pw, String compname, String compvariant, String compversion, Map Attrs)
data = dh.updateComponentAttrs(url, user, pw, comp, "", version , attrs);

[sbtaylor15]

The latest image has the updated for setting all of attributes listed above. Next step is to add logic to the groovy library for passing in Component Item Type (docker/non-docker items).

docker pull quay.io/deployhub/deployhub-pro@sha256:2a086f6c36f63c73d2c810f59545a5a498c4dfbdf7ed4b3eb5dbd218c06090d6

[sbtaylor15]

@piyush94 chartnamespace can now be passed as an attribute.

[sbtaylor15]

@piyush94 here is an example that creates the application version, assigns the application to an environment, and adds component versions to the application.

https://github.com/DeployHubProject/Uptime-Jenkins-Pipeline/blob/master/Jenkinsfile

CC: @svisagan83

[piyush94]

@sbtaylor15 I am getting this error when i am making changes to the docker image.

Not sure if this is the reason.

CC: @svisagan83

[sbtaylor15]

@piyush94 it is trying to roll forward from 1_0_42 to 1_0_43 and the 2 versions are unrelated so it can't figure out how to do the roll forward. I believe there is a flag we can set to clear this up. The other way is to make 1_0_43 be a child of 1_0_42. Goto the base version of the application and then to the Application Versions tab and send a screen shot. I believe both versions come from the "start" instead of 1_0_43 being related to 1_0_42.

CC: @svisagan83

[piyush94]

@sbtaylor15 I don't have the screenshot now, because i deleted that application. But, yes, both versions came from "start". And i think it would be ok to prevent roll forward for helm charts. Let me know how to prevent this.

CC: @svisagan83

[sbtaylor15]

@piyush94 From the Jenkinsfile use forceDeployIfNeeded(String url, String userid, String pw, String Environment) before doing the deployment. This will tell DeployHub to forget the current version that is deployed to the environment enabling any version to be deployed.

Let me know if you need to reset via the UI since there are multiple steps that need to be done.

CC: @svisagan83

[piyush94]

@sbtaylor15 getting this error:

It's asking me to approve the script execution:

CC: @svisagan83

[piyush94]

@sbtaylor15 groovy is having one issue:

fix:

CC: @svisagan83

[piyush94]

@sbtaylor15 9/12/2019 updates issues/enhancements:

1. Inherit components from a successful latest version when doing newApplication().
2. Update component versions for existing components in new application version.
3. Report back error to Jenkins if any step fails for deployApplication().
4. Prevent deployment of already deployed components.

CC: @svisagan83

[piyush94]

@sbtaylor15 Hi Steve, is the creation of non docker component working from the groovy library. If it is, can you share a sample Jenkinsfile. Thanks.

CC: @svisagan83

[sbtaylor15]

@piyush94 we added the logic for the new application version to be derived from the latest. Make sure to set the Component Deploy Always to "N", this will tell DeployHub to skip the roll forward unless the components don't exist for the environment.

Need to discuss what is meant by "latest".

CC: @svisagan83

[piyush94]

@sbtaylor15 Hi, for now "latest" can be latest application version available in the domain. Thanks.

CC: @svisagan83

[piyush94]

@sbtaylor15 Regarding your comment, do i need to change anything in the script to derive application version from latest version? Thanks.

CC: @svisagan83

[sbtaylor15]

@piyush94 no changes are needed.

CC: @svisagan83

[sbtaylor15]

@piyush94 we added the ability to handle multiple component items for non-docker components. See https://github.com/DeployHubProject/Uptime-Jenkins-Pipeline/blob/master/JenkinsfileMulti for an example.

      def component_items = [
       [name: 'file 1', repo: 'GLOBAL.Global_Repo', uri: '/jobs', pattern: 'file11.txt'],
       [name: 'file 2', repo: 'GLOBAL.USTR-NEXUS-1', uri: '/nexus', pattern: 'file12.jar']
      ];
       echo "${component_items}";

      // create component version
      compid = dh.newComponentVersion(url, user, pw, comp, "", version, "file",component_items);
      echo "Creation Done " + compid.toString();

FYI, I need to still add the links between the component items.

CC: @svisagan83

[piyush94]

@sbtaylor15 the uri is not getting updated in the component items.

Thanks.

CC: @svisagan83

[piyush94]

@sbtaylor15 And also one more field for component item is required i.e. "Target Directory". Thanks.

CC: @svisagan83

[sbtaylor15]

Let me check on the URI since it is the only uppercase parameter. Did you update to the latest docker image?

Also I will add target directory.

[piyush94]

I have not updated the docker image yet. Is it required for this?

[sbtaylor15]

Yes, I made a change for the URI.

[piyush94]

@sbtaylor15 Hi Steve, we are getting this error when deploying after updating the image to latest: Runtime error: Select for update failed Unhandled exception: Failed to finish audit recording

It's followed by this error:

We have a modified jssecacerts file mounted into the jre folder with our certificates.

Any idea what can be the issue? Thanks.

CC: @svisagan83

[piyush94]

@sbtaylor15 Also what would be a better way to keep backup of the pgsql from our side? Either pg_dumpall or copy of the pgsql folder mount? Or both.

Thanks,

[sbtaylor15]

@piyush94 Use pg_dumpall if you want to backup while the container is running. Otherwise if the container is stopped you can backup the pgsql host folder.

[sbtaylor15]

@piyush94 can you send me the stack trace for the first error Runtime error: Select for update failed Unhandled exception: Failed to finish audit recording.

It looks like the certs are not installed correctly. How are you getting modified jssecacerts updated into the jre directory? BTW, none of the code around the sync of issues has changed.

[piyush94]

@sbtaylor15 From our side issue was due to two deployhub instances writing to same host pgsql folder. This can be a issue right? It kind of corrupted the database copy. Luckily we had a backup and after restoring it the issue is gone.

There was no stack trace for Runtime error: Select for update failed Unhandled exception: Failed to finish audit recording Thanks.

[sbtaylor15]

@piyush94 Yes, the 2 instances of the postgres server (from the docker containers) accessing the same data files would cause issues/ corruption.

So, is the certificate issue now resolved?

FYI, /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.fc28.x86_64/jre/lib/security is the directory for the jssecacerts base on the JAVA_HOME being used by the jre.

[piyush94]

@sbtaylor15 Yes, issue is resolved now. We are putting jssecacerts at this location only.

Thanks.

[sbtaylor15]

@piyush94 I added the targetdir update for the component item. So pull the latest image. I did not get a chance to test it yet. I also updated the JenkinsfileMulti example

[piyush94]

@sbtaylor15 Is there a recommended way to add certificates to jssecacerts? We are using keytool to add certificates. Thanks.

CC: @svisagan83

[sbtaylor15]

@piyush94 - keytool is the only way that I know of. I believe that you can copy the jssecacerts from another machine if that helps.

CC: @svisagan83

[piyush94]

@sbtaylor15 Hi Steve, any updates on updating component version while creating new application version?

CC: @svisagan83

[sbtaylor15]

@piyush94 we put that in a while ago, but I am not seeing the GitHub comments. Here it is again:

Please pull the latest image.

https://github.com/DeployHubProject/Uptime-Jenkins-Pipeline/blob/master/JenkinsfileMulti

At the bottom is where we add the component version to the application version.

      echo "New Application " + data.toString();
      appid = data[0];

      if (appid > 0 && compid > 0)
      {
       def parent_compid = 0;
       def xpos = 100;
       def ypos = 100;

      data = dh.assignComp2App(url, user, pw, appid, compid, parent_compid, xpos, ypos);
     }    

Note: the ypos needs to be incremented if you are adding multiple component versions to the application version otherwise they will be placed on top of each other.

CC: @svisagan83

[piyush94]

@sbtaylor15 I was asking about, as we had discussed, when creating new application version, it will inherit the component versions from the latest version. And then there should be a way to replace the older component version (if present) with the new version from the Jenkinsfile.

CC: @svisagan83

[sbtaylor15]

@piyush94 yes, the new application version will inherit from its parent version the components. Let me check on how the replace is working at the component level.

CC: @svisagan83

[sbtaylor15]

@piyush94 do you know the old component version name? We will need that to make sure we replace the right component.

CC: @svisagan83

[piyush94]

@sbtaylor15 old component version might not be known as version is auto-generated during the build.

CC: @svisagan83

[sbtaylor15]

@piyush94 do we know part of the name of the component that we can use to do a search? For example, foo;v1 is the component in the application version and we need to update it with foo;v2. We can split on the ; and find the old version based on 'foo'. Let me know what your naming convention is that you are using for the component version names.

CC: @svisagan83