Keystore - openssl issue

[Amithshs]

Trying to setup a keystore for ace. While deploying ace container (k8s pod) we are getting below error.

"/usr/local/bin/ace_config_keystore.sh: exit status 127: openssl: symbol lookup error: openssl: undefined symbol: EVP_md2, version OPENSSL_1_1_0"

• .crt and .key files are under /home/aceuser/initial-config/keystore
• Issue occurs in openssl command from the script ace_config_keystore.sh, while pod is getting started.
• if we just run the ace_config_keystore.sh manually in the pod's container, it works fine!

[tdolby-at-uk-ibm-com]

This sounds very much like a mismatch between the openssl binary and libssl/libcrypto in the image, which would certainly cause issues of this sort.

Attempted a recreate using ACE 11.0.0.11 with MQ 9.1.4 but haven't seen the same failure when running docker locally or k8s in IKS:

2021-04-02T16:46:13.496Z Processing configuration in folder keystore
2021-04-02T16:46:13.758Z+00:00 Handling keystore configuration
2021-04-02T16:46:18.116Z+00:00 Keystore configuration complete
2021-04-02T16:46:18.117Z Initial configuration of integration server complete

which seems quite slow in IKS, but works without issue.

This may well be down to how the docker image is built, or something else in the pod spec in k8s (thought it's not clear how k8s config could cause a symbol to go missing!).

Steps so far for the local recreate:

• git clone this repo, and put ace-11.0.0.11.tar.gz in the deps folder
• run docker build -t ace-dev-mq-dev --build-arg ACE_INSTALL=ace-11.0.0.11.tar.gz --file ubi/Dockerfile.acemq .
• docker tag ace-dev-mq-dev:latest tdolby/experimental:ace-mq-dev
• docker push tdolby/experimental:ace-mq-dev
• Create a keystore, and put it plus a .pass file into the initial-config of a docker image called test-image built on top of tdolby/experimental:ace-mq-dev
• Run the container with docker run --rm -ti -p 7600:7600 -p 7800:7800 -p 7843:7843 -p 1414:1414 --env LICENSE=accept --env MQ_QMGR_NAME=QMGR --env ACE_SERVER_NAME=ACESERVER --env ACE_KEYSTORE_PASSWORD=<password> test-image

For the IKS recreate: kubectl apply -f for the following yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-image
spec:
  selector:
    matchLabels:
      app: test-image
  replicas: 1
  template:
    metadata:
      labels:
        app: test-image
    spec:
      containers:
      - name: test-image
        image: tdolby/experimental:test-image
        ports:
        - containerPort: 7800
        env:
        - name: LICENSE
          value: accept
        - name: MQ_QMGR_NAME
          value: QMGR
        - name: ACE_SERVER_NAME
          value: ACESERVER
        - name: ACE_KEYSTORE_PASSWORD
          value: <password>

It would be worth trying test-image in the failing environment, if that environment can pull from dockerhub; even with the wrong password, it should at least show openssl running and failing with "unable to load private key" instead of the linker error.

Other information that may help debugging:

• Which image is being run when the openssl command fails?
• Is it an IBM-published image, or built with docker build?
• If the latter, what was the command used?
• How was the pod configured when running the image?

[tdolby-at-uk-ibm-com]

Re-verified with RedHat code-ready containers 1.21 with the same successful results as above:

2021-04-02T19:09:00.745Z Processing configuration in folder keystore
2021-04-02T19:09:01.461Z+00:00 Handling keystore configuration
2021-04-02T19:09:06.673Z+00:00 Keystore configuration complete

[IBMRob]

/stale