Running Carla On Kubernetes

[WaelKarkoub]

Hello everyone,

CARLA version:

• 0.9.14

Platform/OS:

• Kubernetes/Ubuntu 18.04

Problem you have experienced:

• Running carla on a server without a display I get sh: 1: xdg-user-dir: not found. This seems to be an issue regardless of which platform I run it on (docker on a server)

What you expected to happen:

• I expect it not to crash

Steps to reproduce: I tried many pod definitions, this is a couple of examples:

kind: Pod
metadata:
  name: carla
  labels:
    app: carla
spec:
  containers:
    - name: carla
      image: carlasim/carla:0.9.14
      imagePullPolicy: Always
      args: ["/bin/bash", "./CarlaUE4.sh", "-RenderOffScreen"]
      resources:
        limits:
          nvidia.com/gpu: 1

apiVersion: v1
kind: Pod
metadata:
  name: carla
  labels:
    app: carla
spec:
  containers:
    - name: carla
      image: carlasim/carla:0.9.14
      imagePullPolicy: Always
      args: ["/bin/bash", "./CarlaUE4.sh", "-RenderOffScreen"]
      env:
        - name: SDL_VIDEODRIVER
          value: "offscreen"
        # - name: DISPLAY
        #   value: $DISPLAY
      ports:
        - containerPort: 2000
        - containerPort: 2001
        - containerPort: 2002
      resources:
        limits:
          nvidia.com/gpu: 1
      securityContext:
        privileged: true
      volumeMounts:
        - name: x11-socket
          mountPath: /tmp/.X11-unix
          readOnly: false
        - name: localtime
          mountPath: /etc/localtime
        - name: timezone
          mountPath: /etc/timezone
        - name: vulkan-icd
          mountPath: /usr/share/vulkan/icd.d
  volumes:
    - name: x11-socket
      hostPath:
        path: /tmp/.X11-unix
    - name: localtime
      hostPath:
        path: /etc/localtime
    - name: timezone
      hostPath:
        path: /etc/timezone
    - name: vulkan-icd
      hostPath:
        path: /usr/share/vulkan/icd.d

[seaurching]

Are u pod crash? I have the same error for u.

[Euraxluo]

kind: Deployment
apiVersion: apps/v1
metadata:
  name: carla914
  namespace: carla
  labels:
    app: carla914
  annotations:
    deployment.kubernetes.io/revision: '2'
    kubesphere.io/creator: youl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: carla914
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: carla914
      annotations:
        kubesphere.io/creator: youl
        kubesphere.io/imagepullsecrets: '{}'
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: carlar914-container
          image: 'carlasim/carla:0.9.14'
          command:
            - ./CarlaUE4.sh
          args:
            - '-carla-server -world-port=2000 -RenderOffScreen'
          ports:
            - name: tcp-2000
              containerPort: 2000
              protocol: TCP
          env:
            - name: DISPLAY
              value: ':1'
            - name: XDG_RUNTIME_DIR
              value: /run/user/1000
          resources: {}
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            allowPrivilegeEscalation: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: default
      serviceAccount: default
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

pod log: sh: 1: xdg-user-dir: not found pod: CrashLoopBackOff

[seaurching]

kind: Deployment
apiVersion: apps/v1
metadata:
  name: carla914
  namespace: carla
  labels:
    app: carla914
  annotations:
    deployment.kubernetes.io/revision: '2'
    kubesphere.io/creator: youl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: carla914
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: carla914
      annotations:
        kubesphere.io/creator: youl
        kubesphere.io/imagepullsecrets: '{}'
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: carlar914-container
          image: 'carlasim/carla:0.9.14'
          command:
            - ./CarlaUE4.sh
          args:
            - '-carla-server -world-port=2000 -RenderOffScreen'
          ports:
            - name: tcp-2000
              containerPort: 2000
              protocol: TCP
          env:
            - name: DISPLAY
              value: ':1'
            - name: XDG_RUNTIME_DIR
              value: /run/user/1000
          resources: {}
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            allowPrivilegeEscalation: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: default
      serviceAccount: default
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

pod log: sh: 1: xdg-user-dir: not found pod: CrashLoopBackOff

upgrade or downgrade os kernel version to kernel-3.10.1160-80 below

[joel-mb]

@WaelKarkoub @seaurching @Euraxluo This is an old issue but leaving here some references about running CARLA on kubernetes.

In this repository (https://github.com/carla-simulator/leaderboard-cloud) you can find all the resources, steps and information about the infrastructure used by the CARLA Leaderboard's cloud. There you can find among others:

• EKS cluster configuration using eksctl: https://github.com/carla-simulator/leaderboard-cloud/blob/main/leaderboard_2.0/config/leaderboard-cluster.yaml
• Simulator and agent kubernetes jobs: https://github.com/carla-simulator/leaderboard-cloud/blob/main/leaderboard_2.0/jobs/submission.yaml)
• Generation of docker images for the simulator and others: https://github.com/carla-simulator/leaderboard-cloud/tree/main/leaderboard_2.0/docker

Closing the issue but feel free to open a new one if needed.