search

zowe / api-layer

The API Mediation Layer provides a single point of access for mainframe service REST APIs.
Eclipse Public License 2.0
54 stars 62 forks source link

localhost.keystore.key not generated with Zowe 0.9.5 #253 nightly build #156

Closed kathysychan closed 5 years ago

kathysychan commented 5 years ago

I installed Zowe 0.9.5 nightly build #253 and found that the Node server is not started and thus the MVD would not come up. Here's the error in Zowesvr stderr:

Error: Unable to access jarfile /opt/zowe/zowe-0.9.5/explorer-server/../scripts/internal/../../api-mediation/scripts/../discovery-service.jar Error: Unable to access jarfile /opt/zowe/zowe-0.9.5/explorer-server/../scripts/internal/../../api-mediation/scripts/../api-catalog-services.jar Error: Unable to access jarfile /opt/zowe/zowe-0.9.5/explorer-server/../scripts/internal/../../api-mediation/scripts/../gateway-service.jar

Also, in the installation log, I see that localhost.keystore.key file is missing in api-mediation/keystore/localhost. Here's the installation log:

Exporting service private key Listing generated files for service: -rw-rw-r-- 1 IBMUSER OMVS 1584 Dec 12 14:35 keystore/localhost/localhost.keystore.cer -rw-rw-r-- 1 IBMUSER OMVS 1584 Dec 12 14:35 keystore/localhost/localhost.keystore.cer-ebcdic -rw-rw-r-- 1 IBMUSER OMVS 1114 Dec 12 14:35 keystore/localhost/localhost.keystore.csr -rw-rw-r-- 1 IBMUSER OMVS 5485 Dec 12 14:35 keystore/localhost/localhost.keystore.p12 -rw-rw-r-- 1 IBMUSER OMVS 1584 Dec 12 14:35 keystore/localhost/localhost.keystore_signed.cer apiml_cm.sh returned: 0

This same machine has no problem installing and running Zowe 0.9.4.

kathysychan commented 5 years ago

@jackjia-ibm Jack is getting this in his install log (localhost.keystore.key is there):

Exporting service private key Listing generated files for service: -rw-rw-r-- 1 OMVSKERN SYS1 1564 Dec 11 23:02 keystore/localhost/localhost.keystore.cer -rw-rw-r-- 1 OMVSKERN SYS1 1564 Dec 11 23:03 keystore/localhost/localhost.keystore.cer-ebcdic -rw-rw-r-- 1 OMVSKERN SYS1 1114 Dec 11 22:57 keystore/localhost/localhost.keystore.csr -rw-rw-r-- 1 OMVSKERN SYS1 1704 Dec 11 23:04 keystore/localhost/localhost.keystore.key -rw-rw-r-- 1 OMVSKERN SYS1 5293 Dec 11 23:01 keystore/localhost/localhost.keystore.p12 -rw-rw-r-- 1 OMVSKERN SYS1 1564 Dec 11 22:58 keystore/localhost/localhost.keystore_signed.cer apiml_cm.sh returned: 0

plavjanik commented 5 years ago

Thank you for reporting the problem.

I would expect to see some error between these two lines:

Exporting service private key
Listing generated files for service:

Can you try following:

cd /opt/zowe/zowe-0.9.5/api-mediation
scripts/apiml_cm.sh --action new-service

and send me the output?

plavjanik commented 5 years ago

I have added more logging into the certificate management script apiml_sm.sh (https://raw.githubusercontent.com/zowe/api-layer/e04050ae00fb8e12f5fecf0e82197ce4af3b4560/scripts/apiml_cm.sh)

Can you upload it to /opt/zowe/zowe-0.9.5/api-mediation/scripts/apiml_cm.sh?

And try:

cd /opt/zowe/zowe-0.9.5/api-mediation
scripts/apiml_cm.sh --action new-service
kathysychan commented 5 years ago

We found that the problem only occurs when we install zowe 0.9.5 using a Jenkins script. If I run zowe-install.sh manually, then the localhost.keystore.key file gets generated and the Node server does come up properly. @kiril-ibm will try the updated shell script and let you know the result.

In the meanwhile, can you let us know which scripts where changed in 0.9.5 from 0.9.4 affecting this area. We want to know if there are extra change directory directives added which could affect the Java process execution as part of the Jenkins script.

plavjanik commented 5 years ago

It can be connected to the JAVA_HOME settings.

kiril-ibm commented 5 years ago

@plavjanik I see following encoding error when we run zowe install script from java. Could it be a problem ? /opt/jenkins/dl/zowe-0.9.5-20181211.203847-253/zowe-0.9.5/install/../temp_2018-12-13/ExportPrivateKey.java:1: error: illegal character: '\u0015' âîóñô£@ëÅ•ÅKâñK∆âìÖ^âîóñô£@ëÅ•ÅKâñK∆âìÖ…ï󧣂£ôÖÅî^âîóñô£@ëÅ•ÅKâñK∆âìÖÊôâ£Öô^âîóñô£@ëÅ•ÅK¢Öɧô⣮K“Ö®^âîóñô£@ëÅ•ÅK¢Öɧô⣮K“Ö®‚£ñôÖ^âîóñô£@ëÅ•ÅK§£âìK¬Å¢ÖˆÙ^ó§ÇìâÉ@ÉìÅ¢¢@≈ßóñô£◊ôâ•Å£Ö“Ö®@¿@@@@óôâ•Å£Ö@∆âìÖ@íÖ®¢£ñôÖ∆âìÖ^@@@@óôâ•Å£Ö@‚£ôâïá@íÖ®‚£ñôÖ„®óÖ^@@@@óôâ•Å£Ö@ÉàÅô≠Ω@íÖ®‚£ñôÖ◊Å¢¢¶ñôÑ^@@@@óôâ•Å£Ö@ÉàÅô≠Ω@íÖ®◊Å¢¢¶ñôÑ^@@@@óôâ•Å£Ö@‚£ôâïá@ÅìâÅ¢^@@@@óôâ•Å£Ö@∆âìÖ@Ößóñô£ÖÑ∆âìÖ^@@@@ó§ÇìâÉ@•ñâÑ@Ößóñô£M]@£àôñ¶¢@≈ßÉÖó£âñï@¿@@@@@@@@“Ö®‚£ñôÖ@íÖ®¢£ñôÖ@~@“Ö®‚£ñôÖKáÖ£…ÅïÉÖMíÖ®‚£ñôÖ„®óÖ]^@@@@@@@@íÖ®¢£ñôÖKìñÅÑMïÖ¶@∆âìÖ…ï󧣂£ôÖÅîMíÖ®¢£ñôÖ∆âìÖ]k@íÖ®‚£ñôÖ◊Å¢¢¶ñôÑ]^@@@@@@@@“Ö®@íÖ®@~@íÖ®¢£ñôÖKáÖ£“Ö®MÅìâÅ¢k@íÖ®◊Å¢¢¶ñôÑ]^@@@@@@@@‚£ôâïá@ÖïÉñÑÖÑ@~@¬Å¢ÖˆÙKáÖ£≈ïÉñÑÖôM]KÖïÉñÑÖ„ñ‚£ôâïáMíÖ®KáÖ£≈ïÉñÑÖÑM]]^@@@@@@@@∆âìÖÊôâ£Öô@ܶ@~@ïÖ¶@∆âìÖÊôâ£ÖôMÖßóñô£ÖÑ∆âìÖ]^@@@@@@@@ܶK¶ôâ£ÖM¬≈«…’@◊Ÿ…¡„≈@“≈Ë]^@@@@@@@@Üñô@Mâï£@â@~@^@â@L@ÖïÉñÑÖÑKìÖïá£àM]^@âNN]@¿@@@@@@@@@@@@âÜ@MMMâ@l@ˆÙ]@~~@]@PP@Mâ@Z~@MÖïÉñÑÖÑKìÖïá£àM]@`@Ò]]]@¿@@@@@@@@@@@@@@@@ܶK¶ôâ£ÖM‡ï]^@@@@@@@@@@@@–@@@@@@@@@@@@ܶK¶ôâ£ÖMÖïÉñÑÖÑKÉàÅô¡£Mâ]]^@@@@@@@@–@@@@@@@@ܶK¶ôâ£ÖM‡ï]^@@@@@@@@ܶK¶ôâ£ÖM≈’ƒ@◊Ÿ…¡„≈@“≈ˇï]^@@@@@@@@ܶKÉìñ¢ÖM]^@@@@–@@@@ó§ÇìâÉ@¢£Å£âÉ@•ñâÑ@îÅâïM‚£ôâïá@Åôá¢≠Ω]@£àôñ¶¢@≈ßÉÖó£âñï@¿@@@@@@@@≈ßóñô£◊ôâ•Å£Ö“Ö®@Ößóñô£@~@ïÖ¶@≈ßóñô£◊ôâ•Å£Ö“Ö®M]^@@@@@@@@Ößóñô£KíÖ®¢£ñôÖ∆âìÖ@~@ïÖ¶@∆âìÖMÅôá¢≠Ω]^@@@@@@@@Ößóñô£KíÖ®‚£ñôÖ„®óÖ@~@Åôá¢≠ÒΩ^@@@@@@@@Ößóñô£KíÖ®‚£ñôÖ◊Å¢¢¶ñôÑ@~@Åôá¢≠ÚΩK£ñ√àÅô¡ôôÅ®M]^@@@@@@@@Ößóñô£KÅìâÅ¢@~@Åôá¢≠ÛΩ^@@@@@@@@Ößóñô£KíÖ®◊Å¢¢¶ñôÑ@~@Åôá¢≠ÙΩK£ñ√àÅô¡ôôÅ®M]^@@@@@@@@Ößóñô£KÖßóñô£ÖÑ∆âìÖ@~@ïÖ¶@∆âìÖMÅôá¢≠ıΩ]^@@@@@@@@Ößóñô£KÖßóñô£M]^@@@@––

plavjanik commented 5 years ago

This can be caused by _BPXK_AUTOCVT environment variable. Can you please send me the output of env command on your z/OS system?

kiril-ibm commented 5 years ago

zoweArtifactoryUrl=https://gizaartifactory.jfrog.io/gizaartifactory SSH_CONNECTION=9.66.240.255 49376 9.30.241.209 22 JENKINS_URL=https://sys-zsystems-zowe-jenkins.swg-devops.com/ IBM_JAVA_ENABLE_ASCII_FILETAG=ON zoweDeployPaxFile=zowe-0.9.5-20181211.203847-253.pax NODE_LABELS=hss tvt4188 tvt4188.svl.ibm.com deployNode=tvt4188.svl.ibm.com zoweDeployVersion=0.9.5-20181211.203847-253 zaasCredentials=zaas-user-credentials JENKINS_HOME=/var/jenkins_home LIBPATH=/usr/lpp/java/J8.0_64/lib/s390x/compressedrefs:/usr/lpp/java/J8.0_64/lib/s390x/j9vm:/usr/lpp/java/J8.0_64/lib/s390x:/usr/lpp/java/J8.0_64/../lib/s390x:/usr/lpp/java/J8.0_64/lib/icc:/usr/lpp/java/J8.0_64/lib/s390x/compressedrefs:/usr/lpp/java/J8.0_64/lib/s390x/j9vm:/usr/lpp/java/J8.0_64/lib/s390x:/usr/lpp/java/J8.0_64/../lib/s390x:/usr/lpp/java/J8.0_64/lib/icc:/usr/lpp/java/J8.0_64/lib/s390x::/usr/lpp/java/J8.0_64/lib/s390x/compressedrefs:/usr/lpp/java/J8.0_64/lib/s390x:/usr/lpp/java/J8.0_64/lib/s390x/compressedrefs:/usr/lpp/java/J8.0_64/lib/s390x zoweUsername=ibmuser hssCredentials=hss-user-credentials STAGE_NAME=Install HUDSON_SERVER_COOKIE=e6d6c11abb45017d TZ=EST5EDT BUILD_TAG=jenkins-zowe-ext-pipeline-multibranch-kiril-dev-499 GRADLE_USER_HOME=/opt/jenkins/.gradle HOME=/u/ibmuser NODE_NAME=tvt4188.svl.ibm.com BUILD_DISPLAY_NAME=#499 GRADLE_HOME=/opt/jenkins/workspace/eline-multibranch_kiril-dev-UNMVYHV6JH6J4XDIAKZ2GTALDFNTF3XRWJHTAH3HMVFYT7WKMVXQ/../../tools/gradle-4.9/bin/gradle zoweCLIBundle=zowe-cli-bundle.zip _BPXK_AUTOCVT=ON mavenTool=Maven 3.5.4 SSH_CLIENT=9.66.240.255 49376 22 WORKSPACE=/opt/jenkins/workspace/eline-multibranch_kiril-dev-UNMVYHV6JH6J4XDIAKZ2GTALDFNTF3XRWJHTAH3HMVFYT7WKMVXQ zoweDeployDir=../../dl/zowe-0.9.5-20181211.203847-253 BUILD_NUMBER=499 toolsDir=../../tools HUDSON_URL=https://sys-zsystems-zowe-jenkins.swg-devops.com/ _EDC_PTHREAD_YIELD=-2 EXECUTOR_NUMBER=0 BRANCH_NAME=kiril-dev Clean=true RUN_CHANGES_DISPLAY_URL=https://sys-zsystems-zowe-jenkins.swg-devops.com/job/zowe-ext-pipeline-multibranch/job/kiril-dev/499/display/redirect?page=changes MAIL=/usr/mail/IBMUSER gradleVersion=4.9 USER=IBMUSER SHELL=/bin/sh JAVA_HOME=/usr/lpp/java/J8.0_64 Init=false atlasAPIURL=https://tvt4188.svl.ibm.com:7443 zowePassword=tivmvs Verify=false BUILD_URL=https://sys-zsystems-zowe-jenkins.swg-devops.com/job/zowe-ext-pipeline-multibranch/job/kiril-dev/499/ GIT_URL=git@github.ibm.com:Zowe-Commercial/zowe-ext-pipeline.git cliNode=kube_helm_cicd JENKINS_NODE_COOKIE=2608fc40-f4b2-4ff3-ad8f-537f66f76fc3 BUILD_ID=499 JOB_BASE_NAME=kiril-dev Install=true JOB_NAME=zowe-ext-pipeline-multibranch/kiril-dev HUDSON_HOME=/var/jenkins_home IBM_JAVA_COMMAND_LINE=/usr/lpp/java/J8.0_64/bin/java -Dorg.gradle.appname=gradle -classpath /opt/jenkins/tools/gradle-4.9/lib/gradle-launcher-4.9.jar org.gradle.launcher.GradleMain --warning-mode all --info --stacktrace --no-daemon --offline -PactiveProfile=hss --init-script /opt/jenkins/workspace/eline-multibranch_kiril-dev-UNMVYHV6JH6J4XDIAKZ2GTALDFNTF3XRWJHTAH3HMVFYT7WKMVXQ@tmp/init-artifactory1027786561469686386gradle install -b gradle-scripts/build.gradle GIT_PREVIOUS_COMMIT=1b6ff8241066969d1f19323c0b34e5762397daa1 GIT_BRANCH=kiril-dev BUILDINFO_PROPFILE=/opt/jenkins/workspace/eline-multibranch_kiril-dev-UNMVYHV6JH6J4XDIAKZ2GTALDFNTF3XRWJHTAH3HMVFYT7WKMVXQ@tmp/buildInfo2941043073758697030.properties PATH=/bin artifactoryDeployRepo=sys-zowe-ext-maven-local artifactoryCredentials=zowe-cicd-artifactory-credentials LOGNAME=IBMUSER Start=false gradleInstallation=../../tools/gradle-4.9 cliDeployVersion=0.9.5-20181126.193239-1 JOB_DISPLAY_URL=https://sys-zsystems-zowe-jenkins.swg-devops.com/job/zowe-ext-pipeline-multibranch/job/kiril-dev/display/redirect IBM_JAVA_OPTIONS=-Djava.io.tmpdir=/opt/tmp -Dfile.encoding=ISO8859-1 -Xnoargsconversion GIT_PREVIOUS_SUCCESSFUL_COMMIT=c6b1ae9a0f70c607054f054c950bfd1f559dc40b HUDSONCOOKIE=543fb7db-c53a-4247-b52a-56f6f4e9eaf7 =/usr/lpp/java/J8.0_64/bin/java RUN_DISPLAY_URL=https://sys-zsystems-zowe-jenkins.swg-devops.com/job/zowe-ext-pipeline-multibranch/job/kiril-dev/499/display/redirect _BPX_SPAWN_SCRIPT=YES activeProfile=hss Stop=false mvdURL=https://tvt4188.svl.ibm.com:8544 hostNode=kube_helm_cicd JENKINS_SERVER_COOKIE=e6d6c11abb45017d zoweArtifactoryCredentials=zowe-artifactory-credentials _CEE_RUNOPTS=POSIX(ON),FILETAG(AUTOCVT,AUTOTAG) artifactoryUrl=https://eu.artifactory.swg-devops.com/artifactory JOB_URL=https://sys-zsystems-zowe-jenkins.swg-devops.com/job/zowe-ext-pipeline-multibranch/job/kiril-dev/ Shutdown=false gradleZipFile=gradle-4.9.zip

kathysychan commented 5 years ago

@plavjanik What Kiril posted above is the env variable we are running zowe-install.sh from within our Jenkins pipeline. @kiril-ibm tried unsetting the _BPXK_AUTOCVT environment variable but that did not help. Please let us know what else to try to possibly fix this problem.

On the other hand, could take a look at the encoding of the ExportPrivateKey.java file? That seems to be the cause of the problem (based on the console output we have running the zowe-install from our Jenkins pipeline).

@nkocsis Nick, this is the problem we found in the 0.9.5 nightly driver. It's blocking us from using our Jenkins pipeline to install Zowe 0.9.5.

plavjanik commented 5 years ago

Thank you for sharing the environment variables. After reading documentation about IBM_JAVA_ENABLE_ASCII_FILETAG, it seems that this one can be causing it. I will do some test tomorrow and make the script work no matter how it is set in your Jenkins

kathysychan commented 5 years ago

@plavjanik Thanks for looking into this. Let us know when there's any nightly driver for us to test with this change.

plavjanik commented 5 years ago

I found the root cause.

It is the setting IBM_JAVA_OPTIONS=-Dfile.encoding=ISO8859-1. I am updating the script to change this setting to the default IBM_JAVA_OPTIONS=-Dfile.encoding=IBM-1047 inside the script.

I think that you can circumvent it by calling: export IBM_JAVA_OPTIONS="-Djava.io.tmpdir=/opt/tmp -Dfile.encoding=IBM-1047" -Xnoargsconversion before calling zowe-install.sh.

Or calling: IBM_JAVA_OPTIONS="-Djava.io.tmpdir=/opt/tmp -Dfile.encoding=IBM-1047" zowe-install.sh as one command instead of zowe-install.sh

plavjanik commented 5 years ago

@kathysychan @kiril-ibm Hello, can you please try the Zowe PAX file that includes the APIML fix? The URL is https://gizaartifactory.jfrog.io/gizaartifactory/libs-snapshot-local/com/project/zowe/0.9.5-PR-196/zowe-0.9.5-20181214.194815-1.pax

kiril-ibm commented 5 years ago

@plavjanik the fix worked for us. Please deliver to the integration branch