Kubernetes

[forestuser]

Hello. When creating a pod + service, gearmand stops working.

LOG INFO 2021-09-03 04:46:16.396362 [ main ] Initializing Gear on port tcp://10.101.82.102:4730 with SSL: false INFO 2021-09-03 04:46:16.000000 [ main ] Starting up with pid 1, verbose is set to INFO ERROR 2021-09-03 04:46:16.000000 [ main ] 0.0.0.0:tcp://10.101.82.102:4730 getaddrinfo(Unrecognized service) -> libgearman-server/gearmand.cc:626 INFO 2021-09-03 04:46:16.000000 [ main ] Shutdown complete

containers:

• name: gearmand image: artefactual/gearmand:latest args:
  • --listen=0.0.0.0
  • --port=4730 ports:
  • containerPort: 4730

If you deploy the pod, and deploy the service after germand initialization. Then germand works. If you deploy pod and service together, an error occurs and gearmand goes into shutdown.

[sevein]

I can't help you because I'm not using k8s, but I'd be happy to accept pull requests. Is this something that we can mitigate from our entry point?

[blafasel42]

i have no problem starting gearmand on k8s. Deploying like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gearman
  labels:
    app: gearman
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gearman
  template:
    metadata:
      labels:
        app: gearman
    spec:
      containers:
        - name: gearmand
          image: artefactual/gearmand:latest
---
apiVersion: v1
kind: Service
metadata:
  name: gearman
spec:
  selector:
    app: gearman
  ports:
    - protocol: TCP
      port: 4730
      targetPort: 4730

[blafasel42]

@forestuser But i have a suspicion what your problem is. I ran into it, too: When you call your service "gearmand", kubernetes auto-creates an env var called GEARMAND_PORT which does not contain a port number but something like: "GEARMAND_PORT=tcp://10.43.78.46:4730" And if an env var of this name is set, gearman will override all other config you give it and stop working since this is not a valid port number.

Solution: just rename your service! I called mine "gearman" instead of "gearmand". This makes k8s name the env var for the service port differently and everything works

[esabol]

@sevein asked:

Is this something that we can mitigate from our entry point?

So I think the answer is yes. What do you think about having the entrypoint script check the value of the $GEARMAND_PORT environment variable and, if it starts with tcp://, then parse the port number from the value and set $GEARMAND_PORT correctly? Something like:

if echo "$GEARMAND_PORT" | grep -q '^tcp://[^:][^:]*:[0-9][0-9]*$' then
    export GEARMAND_PORT=$(echo "$GEARMAND_PORT" | cut -d: -f3)
fi

[sevein]

@esabol that would work for now! Thank you.

[blafasel42]

@esabol i would not use the service's port as the port for the gearman process to listen on. This can be different (service can map external port to other port in the pod).

Also i am not sure if k8s-generated vars would always start with tcp.

My Suggestion: If $GEARMAND_PORT is non-numeric, have the entrypoint echo some warning during startup and remove the value altogether. Like "ignoring non-numeric port $GEARMAND_PORT" . That would make it explicit. (i am not good at shell coding, sorry)

[esabol]

OK, so something like this then:

if echo "$GEARMAND_PORT" | grep -q '[^0-9]' then
    echo 'Error: Invalid (non-numeric) value for GEARMAND_PORT. If using Kubernetes, either change the service name or override the GEARMAND_PORT environment variable to be numeric.'
    exit 1
fi

[blafasel42]

This gives a better error message, but problem stays. How about (stackoverflow inspired code):

if ! [[ "$GEARMAND_PORT" =~ ^[0-9]+$ ]] ; then
    echo "WARNING: Ignoring invalid (non-numeric) value for GEARMAND_PORT: $GEARMAND_PORT"
    unset GEARMAND_PORT
fi

[esabol]

Yes, I meant to edit my previous post to change the exit 1 to unset GEARMAND_PORT, but I got sidetracked.

[blafasel42]

Also the regex seemed not quite right.

[esabol]

@blafasel42 wrote:

Also the regex seemed not quite right.

In my version? No, I believe it is correct. It's searching for any character in $GEARMAND_PORT that's not a number. That's what '[^0-9]' matches. If any non-numeric character is found, then my version prints an error. It works correctly. I tested it. But your version is better because it doesn't rely on grep. You could drop the negation from your version, like this:

if [[ "$GEARMAND_PORT" =~ [^0-9] ]] ; then
    echo "WARNING: Ignoring invalid (non-numeric) value for GEARMAND_PORT: $GEARMAND_PORT"
    unset GEARMAND_PORT
fi

which I feel is cleaner, but they will both do the same thing.

[blafasel42]

@esabol you are right. I missed the double meaning of ^: beginning of string or negation. regex keep tricking me

[esabol]

Sadly, I don't think this change to the ENTRYPOINT script was committed to the repository?