Rest api doesn't show any domains

[creativefre]

Global info: Operating system: Ubuntu 18.04 LTS Kubernetes setup: 1 master and 2 workers Operating version: v2.1

I've succesfully setup the weblogic operator on kubernetes with 1 admin server and 2 managed servers running. My setup script

#!/bin/bash
##
#Create script
##
domainUID=weblogic-domain1
domainNS=weblogic-domain1-ns
operatorName=weblogic-operator
operatorNS=weblogic-operator-ns
serviceAcc=weblogic-operator-sa
username=weblogic
password=welcome1
weblKubOpPath=/scratch/weblogic-kubernetes-operator/
dockerServer=docker.io
##file in your custom values!
dockerUsername=
dockerPassword=
traefikHostname=

#setup the operator
echo "Creating namespace $operatorNS ..."
kubectl create namespace $operatorNS

echo "Creating serviceaccount $serviceAcc for namespace $operatorNS ..."

kubectl create serviceaccount -n $operatorNS $serviceAcc

echo "installing the operator in $operatorNS ..."

helm install ${weblKubOpPath}kubernetes/charts/weblogic-operator \
  --name $operatorName \
  --namespace $operatorNS \
  --set image=oracle/weblogic-kubernetes-operator:2.1 \
  --set serviceAccount=$serviceAcc \
  --set elkIntegrationEnabled=true \
  --set "domainNamespaces={}" \
  --wait

#prepare for domain
echo "Creating namespace $domainNS ..."
kubectl create namespace $domainNS

echo "Set the operator to control the namespace $domainNS ..."
helm upgrade \
  --reuse-values \
  --set "domainNamespaces={"$domainNS"}" \
  --wait \
  weblogic-operator \
  ${weblKubOpPath}kubernetes/charts/weblogic-operator

echo "Set the treafik to control the namespace $domainNS ..."
helm upgrade \
  --reuse-values \
  --set "kubernetes.namespaces={traefik,"$domainNS"}" \
  --wait \
  traefik-operator \
  stable/traefik

echo "Creating a secret for the weblogic-domain in kubernetes ..."
#create a domain Create a Kubernetes secret containing theusernameandpasswordfor the domain using thecreate-weblogic-credentialsscript:
${weblKubOpPath}kubernetes/samples/scripts/create-weblogic-domain-credentials/create-weblogic-credentials.sh \
  -u $username -p $password -n $domainNS -d $domainUID

#create pv and pvc
echo "Creating a pc and pvc for the domain on PV ..."
bash ${weblKubOpPath}kubernetes/samples/scripts/create-weblogic-domain-pv-pvc/create-pv-pvc.sh \
 -i ${weblKubOpPath}kubernetes/samples/scripts/create-weblogic-domain-pv-pvc/my-pv-pvc-inputs.yaml \
 -o /scratch/pv-output

echo "apply the pc and pvc with kubectl..."
kubectl create -f /scratch/pv-output/pv-pvcs/weblogic-domain1-weblogic-domain1-pv.yaml
kubectl create -f /scratch/pv-output/pv-pvcs/weblogic-domain1-weblogic-domain1-pvc.yaml

#create a docker secret
echo "Creating a docker secret ..."
kubectl create secret docker-registry regsecret \
--docker-server=$dockerServer \
--docker-username=$dockerUsername \
--docker-password=$dockerPassword \
-n $operatorNS

echo "Creating a new domain on the pv and auto applying the domain config..."

bash ${weblKubOpPath}kubernetes/samples/scripts/create-weblogic-domain/domain-home-on-pv/create-domain.sh -i ${weblKubOpPath}kubernetes/samples/scripts/create-weblogic-domain/domain-home-on-pv/my-inputs.yaml -o /scratch/my-pvdomainoutput -e

echo "setting the ingress for the weblogic servers..."
helm install ${weblKubOpPath}kubernetes/samples/charts/ingress-per-domain \
  --name ${domainUID}-ingress \
  --namespace ${domainNS} \
  --set wlsDomain.domainUID=${domainUID} \
  --set traefik.hostname=${traefikHostname}

After that I've created a self signed certificate and enabled the rest api in the operator configuration

echo "externalRestEnabled: true" > /scratch/my_cert_values.yaml

bash /scratch/weblogic-kubernetes-operator/kubernetes/samples/scripts/rest/generate-external-rest-identity.sh \
  -a "DNS:lnx-stage01.bryxx.eu,IP:172.17.0.11" \
  -n weblogic-operator-ns -s weblogic-operator-identity >> /scratch/my_cert_values.yaml

helm install /scratch/weblogic-kubernetes-operator/kubernetes/charts/weblogic-operator --name my-operator \
  --namespace weblogic-operator-ns --values /scratch/my_cert_values.yaml --wait

helm upgrade \
  --reuse-values \
  --set "externalRestEnabled=true" \
  --set "externalRestIdentitySecret=weblogic-operator-identity" \
  --wait \
  weblogic-operator \
  /scratch/weblogic-kubernetes-operator/kubernetes/charts/weblogic-operator

When I request to get all the domains of the operator, I don't get any domains in the items list.

Also I noticed that the operator does not respond after a while, for example when I change the domain.yaml file to add a new server to the cluster and apply it with kubectl apply -f domain.yaml. Nothing happens and no extra managed server is spinned up.

Any solutions on why I don't get the domains in the REST api call?

[shean-guangchang]

I will ask the team to look into your issue. As far as changing domain.yaml file do you think it has anything to do with REST certification issue or it is independent?

[shean-guangchang]

Can you set operator debug to FINE or FINER and post the log here when you see the problem?

[shean-guangchang]

One potential workaround to try is to kill the operator pod and see if the re-create pod will solve your problem. Before doing that could you gather the following info:

1. operator config map
2. helm chart

[creativefre]

I'm not sure the domain.yaml has something to do with the rest certification issue. On other occasions after some time (a few hours) I also had instability problems. Pods just failed and did not restart. Apart from that, I've setup the operator with javaLoggingLevel=FINEST and the operator was logging With FINEST. When I enable the REST api. The operator pod recreates it self and goes back to javaLoggingLevel=INFO even when I've set it to FINEST. So for the moment it's not possible to give that logging information.

After enabling the REST API Current helm chart Kubernetes Dashboard operator pod info

Now the REST API is reachable but no domains are given back like before. When I apply a new domain.yaml config with an extra managed server. Nothing happens.

Even when I recreate the pod with helm, the logging level stays on INFO

helm upgrade   --reuse-values --set javaLoggingLevel=FINEST   --wait   weblogic-operator   /scratch/weblogic-kubernetes-operator/kubernetes/charts/weblogic-operator

I'm not sure if this is the operator config map that you mean.

[shean-guangchang]

I cannot see the exact steps you did so maybe you can simply answer the question I was looking for... After you updated the domain namespace did you restart the operator pod and then did you still have issue of getting the domain thru REST?

[creativefre]

Yes, my exact steps are beginning from a kubernetes without the weblogic operator.

First I execute my setup script (First comment). Everything works fine after this.

After that I execute the following code to enable the rest API (Also in the first comment)

echo "externalRestEnabled: true" > /scratch/my_cert_values.yaml

bash /scratch/weblogic-kubernetes-operator/kubernetes/samples/scripts/rest/generate-external-rest-identity.sh \
  -a "DNS:lnx-stage01.bryxx.eu,IP:172.17.0.11" \
  -n weblogic-operator-ns -s weblogic-operator-identity >> /scratch/my_cert_values.yaml
#
#
#After the following step, the operator starts to act weird and restarts
#
#
helm install /scratch/weblogic-kubernetes-operator/kubernetes/charts/weblogic-operator --name my-operator \
  --namespace weblogic-operator-ns --values /scratch/my_cert_values.yaml --wait

helm upgrade \
  --reuse-values \
  --set "externalRestEnabled=true" \
  --set "externalRestIdentitySecret=weblogic-operator-identity" \
  --wait \
  weblogic-operator \
  /scratch/weblogic-kubernetes-operator/kubernetes/charts/weblogic-operator

So I don't say to restart the pod, but it restarts itself for some reason

After that I tried to restart the pod manually with: (The --recreate-pods was ,missing in my previous post but I did execute it)

helm upgrade   --reuse-values --recreate-pods --set javaLoggingLevel=FINEST   --wait   weblogic-operator   /scratch/weblogic-kubernetes-operator/kubernetes/charts/weblogic-operator

And the issue with the REST api remains. Hopefully this is more clear? On which steps I took to enable the REST api.

Else let met know! Thanks for the reply

[shean-guangchang]

I can see from the helm chart you showed above the "targetNamespaces: default" is not the correct value. Could you shutdown everything and start it again and this time first update the helm chart (e.g. the value yaml) with the desired target namespace and then run helm install and then show the configmap and verify REST command?

[creativefre]

Hello thanks for helping me. I found the solution, I executed a wrong command.

I installed the weblogic operator with custom values in my install script. But when I enabled the operator there was also a step in the documentation to reinstall the operator with the my_cert_values.yaml.

Because I already installed the operator I only needed to update and recreate the pods. So this step was not neceserry, It reinstalls the operator with the default values in the yaml file.

helm install /scratch/weblogic-kubernetes-operator/kubernetes/charts/weblogic-operator --name my-operator \
  --namespace weblogic-operator-ns --values /scratch/my_cert_values.yaml --wait

I only needed to execute these steps to enable the external REST api.

bash /scratch/weblogic-kubernetes-operator/kubernetes/samples/scripts/rest/generate-external-rest-identity.sh \
  -a "DNS:lnx-stage01.bryxx.eu,IP:172.17.0.11" \
  -n weblogic-operator-ns -s weblogic-operator-identity 

helm upgrade \
  --reuse-values --recreate-pods \
  --set "externalRestEnabled=true" \
  --set "externalRestIdentitySecret=weblogic-operator-identity" \
  --wait \
  weblogic-operator \
  /scratch/weblogic-kubernetes-operator/kubernetes/charts/weblogic-operator

You can close this issue now.

[shean-guangchang]

so when did you set domainNamespaces now?

[creativefre]

I already set the domainNamespaces in my setup script.

echo "installing the operator in $operatorNS ..."

helm install ${weblKubOpPath}kubernetes/charts/weblogic-operator \
  --name $operatorName \
  --namespace $operatorNS \
  --set image=oracle/weblogic-kubernetes-operator:2.1 \
  --set serviceAccount=$serviceAcc \
  --set elkIntegrationEnabled=true \
  --set "domainNamespaces={}" \
  --wait

#prepare for domain
echo "Creating namespace $domainNS ..."
kubectl create namespace $domainNS

echo "Set the operator to control the namespace $domainNS ..."
helm upgrade \
  --reuse-values \
  --set "domainNamespaces={"$domainNS"}" \
  --wait \
  weblogic-operator \
  ${weblKubOpPath}kubernetes/charts/weblogic-operator

[tbarnes-us]

I'm assuming the issue is resolved by now: from what I can tell, it may have been left open by @rjeberhard as a reminder to enhance the documentation. Feel free to re-open the issue if I'm incorrect.