search

contribsys / faktory

Language-agnostic persistent background job server
https://contribsys.com/faktory/
Other
5.78k stars 230 forks source link

Wiki documentation ent kubernetes installation #446

Closed aaroalan closed 10 months ago

aaroalan commented 1 year ago

Hi,

I would like to contribute to the wiki (ent installation section), the footer specify that is publicly editable for typos and errors but it does not mention anything about new sections, also because there is not PR feature for github wiki I think is better to validate that instructions are clear/typo free before.

Change is in Ent-Installation.md right after the Linux section. Please feel free to fix/improve/clarify anything and add it yourself to the wiki if you want.

Kubernetes

The next section describe how to deploy faktory ent to a kubernetes cluster using the private contribsys docker image repository.

IMPORTANT: Next configurations are demonstrative of how deploy faktory ent to a kubernetes cluster, but do not represent a one size fits all enviroments and should be used as a guide in order to create the best configurations for your specific requeriments.

1. Generate base64 data.

Kubernetes supports private container image registry/repository but you need to create the secrets with your credentials in order to add the configuration the the faktory k8s manifest. doc

1.1 Generate basic authentication value.

# e.g.
# echo -n 'foo:bar' | base64
# Zm9vOmJhcg==
echo -n '<your username>:<your password>' | base64

1.2 Generate the config.json

Note that the config json requires the result of the previous step 1.1 in the key auth.

# e.g.
# echo -n '{"auths":{"https://docker.contribsys.com": {"username": "foo","password":"bar","email":"foo.bar@quux.com","auth": "Zm9vOmJhcg=="}}}' | base64
# eyJhdXRocyI6eyJodHRwczovL2RvY2tlci5jb250cmlic3lzLmNvbSI6IHsidXNlcm5hbWUiOiAiZm9vIiwicGFzc3dvcmQiOiJiYXIiLCJlbWFpbCI6ImZvby5iYXJAcXV1eC5jb20iLCJhdXRoIjogIlptOXZPbUpoY2c9PSJ9fX0=
echo -n '{"auths":{"https://docker.contribsys.com": {"username": "<your username>","password":"<your password>","email":"<your email>","auth": "<result step 1.1>"}}}' | base64

2. Container image secret

Create secret containing the credentials to pull the image from private container image repository.

apiVersion: v1
kind: Secret
metadata:
  name: contribsys-container-registry-secret
data:
  .dockerconfigjson: <result of step 1.2 e.g. eyJhdXRo...>
type: kubernetes.io/dockerconfigjson

3. Create license secret

Create secret with your license.

apiVersion: v1
kind: Secret
metadata:
  name: license-secret
stringData:
  license: "<your license>"

4. Create faktory ent deployment

Deployment uses contribsys-container-registry-secret to pull the image from the private docker image repository and adds your license in folder /etc/faktory

apiVersion: apps/v1
kind: Deployment
metadata:
  name: faktory-ent
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      name: faktory-ent
  template:
    metadata:
      labels:
        name: faktory-ent
    spec:
      imagePullSecrets:
        - name: contribsys-container-registry-secret
      containers:
        - name: server
          image: docker.contribsys.com/contribsys/faktory-ent:1.8.0
          imagePullPolicy: IfNotPresent
          command:
            - "/faktory"
            - "-b"
            - ":7419"
            - "-w"
            - ":7420"
            - "-e"
            - "production"
          ports:
            - containerPort: 7419
            - containerPort: 7420
          volumeMounts:
            - name: license-volume
              mountPath: /etc/faktory/license
              subPath: license
              readOnly: true
      volumes:
        - name: license-volume
          secret:
            secretName: license-secret
jbielick commented 1 year ago

@aaroalan Your suggested deployment does not allow persistence. Would the faktory server retain data between restarts in this configuration?

aaroalan commented 1 year ago

@jbielick, Sorry I focused in the configuration for the docker image authentication and license in order to have a simple deployment.

This sample is very simple approach and missing "normal" configurations like the mentioned persistent volume for data, env variables, resource configuration (memory, cpu, ..., etc), liveness and readiness but because felt that those are the common configs and may be different per implementation I did not add them here.

Answering your question: If faktory is using the internal Redis then the deployment would need another volume attached to the folder that contains the Redis data which I believe is /var/lib/faktory/db, but if it is using a remote Redis then that volume is not necessary because data won't be in the pod generated for this deployment.

persistent:

...
  volumeMounts:
    - name: faktory-storage-volume
      mountPath: "/var/lib/faktory/db"
...
volumes:
  - name: faktory-storage-volume
    persistentVolumeClaim:
      claimName: faktory-pvc

liveness & readiness:

...
readinessProbe:
  exec:
   command:
   - /bin/sh
   - -c
   - "ps aux -A | grep '/faktory' | grep -v grep"
  initialDelaySeconds: 5
  periodSeconds: 10
livenessProbe:
  exec:
   command:
   - /bin/sh
   - -c
   - "ps aux -A | grep '/faktory' | grep -v grep"
  initialDelaySeconds: 5
  periodSeconds: 10
mperham commented 1 year ago

As you've learned, the instructions need to be production-quality. I don't mind people editing the wiki but you've got to use that power responsibly. We don't want people following the steps, only to lose production data. Persistence is not optional.

mperham commented 1 year ago

For liveness and readyness, should we use a TCP or HTTP probe on port 7420 instead of the ps hack?

aaroalan commented 1 year ago

  1. I did not add a full example because imo will be more confusing specially because there is not a single way to setup some things e.g. Some cloud env like GKE allow you to mount a disk directly without the need to create a separate k8s volume, or you can create a volume attached to the disk, also you can add ENV variable directly, but is common to do it using k8s secrets or have something like vault implemented.

  2. I thought about the liveness and readyness using http but was not sure if it will work when you have authentication on, but if endpoint returns 200 I think is more desirable than the ps command.

I would prefer to remove everything in the deployment.yaml and keep only the parts that are related to contribsys-container-registry-secret and license-volume so it is obvious that you need to complete everything else but also happy to add a more complete example if you think is better to have a full example?

alondahari commented 1 year ago

I agree that the docs can be improved when it comes to kubernetes installation. Here is our yaml that includes the option to have multiple replicas with an elasticache persistence layer. It includes a load balancer in front of the servers:

kind: Service
apiVersion: v1
metadata:
  name: faktory-server
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: true
spec:
  type: LoadBalancer
  selector:
    app: faktory-server
  ports:
    - name: faktory
      protocol: TCP
      port: 7419
      targetPort: 7419
    - name: dashboard
      protocol: TCP
      port: 7420
      targetPort: 7420
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: faktory-server
  labels:
    app: faktory-server
spec:
  replicas: 2
selector:
    matchLabels:
      app: faktory-server
  template:
    metadata:
      labels:
        app: faktory-server
    spec:
      shareProcessNamespace: true
      terminationGracePeriodSeconds: 10
      containers:
        - name: faktory-server
          image: 2
          imagePullPolicy: Always
          ports:
            - containerPort: 7419
              name: faktory
            - containerPort: 7420
              name: dashboard
          env:
            - name: FAKTORY_ENV
              value: production
            - name: FAKTORY_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: faktory-secrets
                  key: FAKTORY_PASSWORD
            - name: REDIS_URL
              valueFrom:
                configMapKeyRef:
                  name: faktory-server-env-vars
                  key: REDIS_URL
          readinessProbe:
            tcpSocket:
              port: 7420
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            tcpSocket:
              port: 7420
            initialDelaySeconds: 15
            periodSeconds: 10
aaroalan commented 1 year ago

@jazzdragon very clean manifest, a couple of questions.

  1. Having two replicas is possible because is using the remote redis in your case elasticache but if faktory uses the local redis this is not possible right?
  2. Do you have password authentication? I did not test the readiness and liveness via http when password is set so would like to know if it works in that situation.
alondahari commented 1 year ago
  1. right, to the best of my knowledge
  2. we do have password auth, the probes seem to work fine with the tcp setup since they are just pinging the server
aaroalan commented 1 year ago

@mperham What about this? it contains only the license and docker registry authentication, but the deployment is incomplete (in addition to the text that clarify that) so is very clear that you need to complete the rest base on your requirements and capabilities ...

Kubernetes

The next section describe how to deploy faktory ent to a kubernetes cluster using the private contribsys docker image repository.

IMPORTANT: Next configurations are incomplete do not represent a one size fits all enviroments and should be used as a guide in order to create the best configurations for your specific requeriments.

1. Generate base64 data.

Kubernetes supports private container image registry/repository but you need to create the secrets with your credentials in order to add the configuration the the faktory k8s manifest. doc

1.1 Generate basic authentication value.

# e.g.
# echo -n 'foo:bar' | base64
# Zm9vOmJhcg==
echo -n '<your username>:<your password>' | base64

1.2 Generate the config.json

Note that the config json requires the result of the previous step 1.1 in the key auth.

# e.g.
# echo -n '{"auths":{"https://docker.contribsys.com": {"username": "foo","password":"bar","email":"foo.bar@quux.com","auth": "Zm9vOmJhcg=="}}}' | base64
# eyJhdXRocyI6eyJodHRwczovL2RvY2tlci5jb250cmlic3lzLmNvbSI6IHsidXNlcm5hbWUiOiAiZm9vIiwicGFzc3dvcmQiOiJiYXIiLCJlbWFpbCI6ImZvby5iYXJAcXV1eC5jb20iLCJhdXRoIjogIlptOXZPbUpoY2c9PSJ9fX0=
echo -n '{"auths":{"https://docker.contribsys.com": {"username": "<your username>","password":"<your password>","email":"<your email>","auth": "<result step 1.1>"}}}' | base64

2. Container image secret

Create secret containing the credentials to pull the image from private container image repository.

apiVersion: v1
kind: Secret
metadata:
  name: contribsys-container-registry-secret
data:
  .dockerconfigjson: <result of step 1.2 e.g. eyJhdXRo...>
type: kubernetes.io/dockerconfigjson

3. Create license secret

Create secret with your license.

apiVersion: v1
kind: Secret
metadata:
  name: license-secret
stringData:
  license: "<your license>"

4. Create faktory ent deployment

Deployment uses contribsys-container-registry-secret to pull the image from the private docker image repository and adds your license in folder /etc/faktory

# This deployment manifest is incomplete and contains only the sections related to the docker image repository
# authentication and license.
apiVersion: apps/v1
kind: Deployment
...
...
    spec:
      ...
      ...
      # Secret used for the docker image repository authentication.
      imagePullSecrets:
        - name: contribsys-container-registry-secret
      containers:
        - name: server
          image: docker.contribsys.com/contribsys/faktory-ent:1.8.0
          imagePullPolicy: IfNotPresent
          ...
          ...
          # Mounting the licese in "/etc/faktory/license"
          volumeMounts:
            - name: license-volume
              mountPath: /etc/faktory/license
              subPath: license
              readOnly: true
      volumes:
        - name: license-volume
          secret:
            secretName: license-secret
mperham commented 1 year ago

Would it be easier to expose the license string as FAKTORY_LICENSE in the environment?

And I assume you are creating all of the credential steps to avoid running docker login docker.contribsys.com manually?

aaroalan commented 1 year ago

Oh I missed that option in the instructions. Yes it would be easier to have it in the env as FAKTORY_LICENSE The secret still necessary so is not hardcoded in the deployment but just setting the env FAKTORY_LICENSE instead of dumping the value to a file.

Yes, the purpose is remove any manual step and have only the Kubernetes manifest, it is not uncommon to not have access to do anything directly in the cluster and all changes are in the k8s files then something like ArgoCD takes that and applies the change to the cluster.