search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.41k stars 1.68k forks source link

[Detector Support]: beta2-tensorrt Cuda failure in file 'yolo_layer.cu' #7959

Closed slackr31337 closed 1 year ago

slackr31337 commented 1 year ago

Describe the problem you are having

Starting docker image 0.13.0-beta2-tensorrt for the first time fails to generate model.

Cuda failure in file 'yolo_layer.cu' line 51: 35 /etc/s6-overlay/s6-rc.d/trt-model-prepare/run: line 82: 204 Aborted

Version

0.13.0-0858859

Frigate config file

database:
  path: /config/frigate.db

mqtt:
  client_id: frigate-server07
  host: 10.xxx.xxx.xxx
  port: 1883
  stats_interval: 30

logger:
  default: info

birdseye:
  enabled: True

live:
  quality: 8

ui:
  live_mode: webrtc
  timezone: America/New_York
  use_experimental: false

motion:
  threshold: 24
  # contour_area: 28 
  # improve_contrast: True

detect:
  enabled: true
  fps: 6

  stationary:
    interval: 10
    threshold: 50

detectors:
  tensorrt:
    type: tensorrt

model:
  #path: /trt-models/yolov7-tiny-416.trt
  #path: /config/model_cache/tensorrt/yolov7-tiny-416.trt
  path: /config/model_cache/tensorrt/yolov7x-640.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 640
  height: 640

  labelmap:
    2: vehicle # car
    3: vehicle # motorcycle
    5: vehicle # bus
    7: vehicle # car
    15: animal # bird
    16: animal # cat
    17: animal # dog
    18: animal # horse/deer

objects:
  track:
    - person
    - vehicle
    - animal

  filters:
    vehicle:
      min_score: 0.65
      threshold: 0.74

    animal:
      min_score: 0.65
      threshold: 0.76

snapshots:
  enabled: True
  retain:
    default: 30
    objects:
      animal: 7
      person: 20

record:
  enabled: True
  expire_interval: 300 
  retain:
    days: 10 
    mode: all 

  events:
    pre_capture: 15 
    post_capture: 30

    retain:
      default: 30 
      mode: active_objects
      objects:
        animal: 5

audio:
  #enabled: True
  listen:
    - bark
    - fire_alarm
    - scream
    - speech
    - yell

ffmpeg:  
  global_args: -hide_banner -loglevel warning
  hwaccel_args: preset-nvidia-h264 
  input_args: preset-rtsp-restream
  output_args:
    record: preset-record-generic-audio-aac

go2rtc:
  streams:
    driveway:
      - rtsp://frigate@10.xxx.xxx.xxx:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:driveway#audio=aac#audio=opus"
    driveway_d1:
      - rtsp://frigate@10.xxx.xxx.xxx:554/cam/realmonitor?channel=1&subtype=1
      - "ffmpeg:driveway_d1#audio=aac#audio=opus"

    sideyard:
      - rtsp://frigate@10.xxx.xxx.xxx:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:sideyard#audio=aac#audio=opus"
    sideyard_hd:
      - rtsp://frigate@10.xxx.xxx.xxx:554/cam/realmonitor?channel=1&subtype=2
      - "ffmpeg:sideyard_hd#audio=aac#audio=opus"

    frontdoor:
      - rtsp://frigate@10.xxx.xxx.xxx:554/cam/realmonitor?channel=1&subtype=0
    garage:
      - ffmpeg:rtsp://frigate@10.xxx.xxx.xxx:554/cam/realmonitor?channel=1&subtype=0
    office:
      - rtsp://frigate@10.xxx.xxx.xxx:554/cam/realmonitor?channel=1&subtype=0
    backyard:
      - ffmpeg:rtsp://frigate@10.xxx.xxx.xxx:554/cam/realmonitor?channel=1&subtype=0

  webrtc:
    candidates:
      - 10.xxx.xxx.xxx:9555
      - stun:9555
      - 10.xxx.xxx.xxx:8555
      - stun:8555

##############################
cameras:

  ###############################
  sideyard:
    mqtt:
      enabled: False

    ui:
      order: 0
      dashboard: True

    #audio:
    #  enabled: True

    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy

      inputs:
        - path: rtsp://127.0.0.1:8554/sideyard
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/sideyard_hd
          roles:
            - detect

    live:
      stream_name: sideyard_hd

    record:
      events:
        required_zones:
          - sideyard_yard

    motion:
      mask:
        - 0,720,282,720,287,684,0,679
        - 0,477,143,381,127,203,559,130,730,181,831,137,1131,228,1105,295,1002,283,766,631,581,720,1280,720,1280,0,0,0

    zones:
      sideyard_yard:
        coordinates: 842,162,1119,251,1084,382,923,720,0,720,0,451,126,402,114,239,529,152,728,198

    objects:
      track:
        - person
        - animal

      filters:
        animal:
          mask:
            - 0,0,0,449,126,241,556,151,733,190,909,173,1134,180,1102,303,770,720,1280,720,1280,0
        person:
          mask:
            - 0,0,0,449,126,241,556,151,733,190,909,173,1134,180,1102,303,770,720,1280,720,1280,0

  ###############################
  driveway:
    mqtt:
      enabled: False

    ui:
      order: 1
      dashboard: True

    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy

      inputs:
        - path: rtsp://127.0.0.1:8554/driveway
          roles:
            - record
            - detect

    live:
      stream_name: driveway_d1
      height: 480

    objects:
      track:
        - person
        - vehicle
        - animal

      filters:
        vehicle:
          mask:
            - 0,144,389,79,777,66,1216,462,577,720,1280,720,1280,0,755,0,0,0

        person:
          min_area: 2000
          max_area: 200000
          mask:
            - 0,0,1280,0,1280,208,1161,198,1070,155,847,98,613,87,367,99,0,162

    motion:
      mask:
        - 625,79,485,45,265,73,44,138,40,228,64,302,0,387,0,0,1280,0,1280,197,1196,145,1181,207,1181,277,1112,258,1086,148,949,131,812,78
        - 1280,720,928,720,932,682,1280,676
        - 865,160,1062,300,1083,390,966,446,652,190
        - 533,196,846,509,481,626,371,217

    zones:
      driveway_frontyard:
        coordinates: 1280,500,759,204,1069,176,1280,247

      driveway_sideyard:
        coordinates: 0,339,0,720,441,720,266,260

      parking_spots:
        coordinates: 441,720,1171,720,1280,520,940,315,635,185,281,245,263,399,319,617

      driveway_entrance:
        coordinates: 564,143,711,198,301,267,280,177

      road:
        coordinates: 0,253,291,182,583,132,784,108,763,85,569,83,430,96,287,121,0,179

    record:
      events:
        post_capture: 120
        required_zones:
          - driveway_entrance
          - driveway_sideyard
          - driveway_frontyard

    snapshots:
      required_zones:
          - road
          - driveway_entrance
          - driveway_sideyard
          - driveway_frontyard

  ###############################
  frontdoor:
    mqtt:
      enabled: False

    ui:
      order: 2
      dashboard: True

    ffmpeg:

      inputs:
        - path: rtsp://127.0.0.1:8554/frontdoor
          roles:
            - record
            - detect           

    objects:
      track:
        - person
        - animal

      filters:
        #person:
        #  min_area: 80000
        #  max_area: 200000

        animal:
          mask:
            - 392,720,0,720,0,475,243,458

    motion:
      mask:
        - 1126,191,1154,262,1094,356,1095,400,1000,422,939,369,981,252,1051,191
        - 938,201,917,301,868,326,811,302,802,252,829,199,877,154
        - 1280,94,1208,160,1165,128,1188,94,1202,54,1232,27,1280,53
        - 584,104,564,0,680,0,656,91,654,314,562,340

    zones:
      doorstep:
        coordinates: 534,288,1072,574,776,720,454,720,202,488

      frontdoor_yard:
        coordinates: 1202,48,947,225,795,170,567,287,173,531,0,337,0,0,665,0,1280,0,1280,49

    record:
      events:
        post_capture: 90
        required_zones:
          - doorstep
          - frontdoor_yard

    snapshots:
      retain:
        objects:
          person: 30

      required_zones:
          - doorstep
          - frontdoor_yard

  ###############################
  garage:
    mqtt:
      enabled: False

    ui:
      order: 2
      dashboard: True

    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage
          roles:
            - record
            - detect

    objects:
      track:
        - person

  ###############################
  backyard:
    mqtt:
      enabled: False

    ui:
      order: 3
      dashboard: True

    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/backyard
          roles:
            - record
            - detect

    objects:
      track:
        - person
        - animal

    motion:
      mask:
        - 710,720,825,104,802,41,508,51,450,0,1280,0,1280,720
        - 251,284,487,720,0,720,0,0,117,0,133,212

    zones:
      flowerbed:
        coordinates: 179,720,989,720,756,287,438,307,278,169,73,179
      backyard_path:
        coordinates: 477,78,729,68,734,98,478,126,140,164,91,142
      backyard_shed:
        coordinates: 897,111,884,159,731,134,746,86

  ###############################
  office:
    mqtt:
      enabled: False

    ui:
      order: 4
      dashboard: True

    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/office
          roles:
            - record
            - detect

    objects:
      track:
        - person

    motion:
      mask:
        - 1280,0,1280,315,0,320,0,0

    zones:
      desk:
        coordinates: 1211,348,1218,667,133,683,125,338

docker-compose file or Docker CLI command

[
    {
        "Id": "e5e51a511d0d92fb167fa5bcdae927929199431803ef232648ac356be4ca78ab",
        "Created": "2023-09-26T19:49:39.173268539Z",
        "Path": "/init",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 4008927,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-09-26T19:49:49.693756927Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:da3158e53086ff6734aa52f2dedbdcd385bf5a919ae61642b449fc8bc41b871e",
        "ResolvConfPath": "/mnt/pool00/ix-applications/docker/containers/8ac4f445b10b63dbbfd0bd7e2c2b0517eca1a0b32237d9dcf6188dcbf60b5bef/resolv.conf",
        "HostnamePath": "/mnt/pool00/ix-applications/docker/containers/8ac4f445b10b63dbbfd0bd7e2c2b0517eca1a0b32237d9dcf6188dcbf60b5bef/hostname",
        "HostsPath": "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/etc-hosts",
        "LogPath": "/mnt/pool00/ix-applications/docker/containers/e5e51a511d0d92fb167fa5bcdae927929199431803ef232648ac356be4ca78ab/e5e51a511d0d92fb167fa5bcdae927929199431803ef232648ac356be4ca78ab-json.log",
        "Name": "/k8s_ix-chart_frigate-ix-chart-7bf9b5b4f5-b6z4x_ix-frigate_0426ab0f-0ca1-4841-b7ff-d19932dc4c75_0",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "unconfined",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/mnt/pool02/cameras:/media/frigate",
                "/mnt/pool00/frigate:/config",
                "/mnt/pool00/tensorrt/models:/trt-models",
                "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/volumes/kubernetes.io~empty-dir/ix-emptydir-volume-frigate-0:/tmp/cache",
                "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/volumes/kubernetes.io~empty-dir/ix-emptydir-volume-frigate-1:/dev/shm",
                "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/volumes/kubernetes.io~projected/kube-api-access-wrqpc:/var/run/secrets/kubernetes.io/serviceaccount:ro",
                "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/etc-hosts:/etc/hosts",
                "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/containers/ix-chart/836c1c1e:/dev/termination-log"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "container:8ac4f445b10b63dbbfd0bd7e2c2b0517eca1a0b32237d9dcf6188dcbf60b5bef",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "container:8ac4f445b10b63dbbfd0bd7e2c2b0517eca1a0b32237d9dcf6188dcbf60b5bef",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 1000,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "seccomp=unconfined",
                "label=disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "nvidia",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 2,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "/kubepods/besteffort/pod0426ab0f-0ca1-4841-b7ff-d19932dc4c75",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 100000,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": null,
            "ReadonlyPaths": null
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/mnt/pool00/ix-applications/docker/overlay2/ce842e7420d8e975fb4b0050e607dd8429c3c65745e04892d6dcca5ecc56ca72-init/diff:/mnt/pool00/ix-applications/docker/overlay2/80fc4a62bbe900693d56409a5c7d7d6d65b006302be43adf15ede2bd24c11e85/diff:/mnt/pool00/ix-applications/docker/overlay2/5d1dcd46cae2b15d358ebbb6328e2889896a2b78da4ffb04c196e407254f148a/diff:/mnt/pool00/ix-applications/docker/overlay2/42ce8288105035ba780744e8c549d06724f56170b43a5ad53db0f9745fff9025/diff:/mnt/pool00/ix-applications/docker/overlay2/9353abce122472c09b521ba7ffb380e3c02cd3da6862d0c3440c6c1b9ac640e4/diff:/mnt/pool00/ix-applications/docker/overlay2/5116c014e8465b4eda620598ef334d1d02caa0f72d113849e4539aaab29416dd/diff:/mnt/pool00/ix-applications/docker/overlay2/62df0ef85d2e333a05a265ec0a6131aa74ea88b77ff5cafaa6083e6689b046bd/diff:/mnt/pool00/ix-applications/docker/overlay2/37307e75fb672d9699fe5fe4b2f0877252cf8287736462e651c3394688d6a655/diff:/mnt/pool00/ix-applications/docker/overlay2/bb11ee6e039de61d05020def2af073b545890caa5e1ace08ca2e2dc0e3dd1fd0/diff:/mnt/pool00/ix-applications/docker/overlay2/75a556f9372c404635fcf9c751069d51020ff96d2d1e86e45387074a937900db/diff:/mnt/pool00/ix-applications/docker/overlay2/98353bdaafc1d5f69534c180b194577b5a0276f769fbb7e51ccb30747872d866/diff:/mnt/pool00/ix-applications/docker/overlay2/1dce1a71615a904e2244ce6c39a48b242c3c4ce56180b4df5f01ac5ad52dae57/diff",
                "MergedDir": "/mnt/pool00/ix-applications/docker/overlay2/ce842e7420d8e975fb4b0050e607dd8429c3c65745e04892d6dcca5ecc56ca72/merged",
                "UpperDir": "/mnt/pool00/ix-applications/docker/overlay2/ce842e7420d8e975fb4b0050e607dd8429c3c65745e04892d6dcca5ecc56ca72/diff",
                "WorkDir": "/mnt/pool00/ix-applications/docker/overlay2/ce842e7420d8e975fb4b0050e607dd8429c3c65745e04892d6dcca5ecc56ca72/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/volumes/kubernetes.io~empty-dir/ix-emptydir-volume-frigate-1",
                "Destination": "/dev/shm",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/volumes/kubernetes.io~projected/kube-api-access-wrqpc",
                "Destination": "/var/run/secrets/kubernetes.io/serviceaccount",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/etc-hosts",
                "Destination": "/etc/hosts",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/containers/ix-chart/836c1c1e",
                "Destination": "/dev/termination-log",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/mnt/pool02/cameras",
                "Destination": "/media/frigate",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/mnt/pool00/frigate",
                "Destination": "/config",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/mnt/pool00/tensorrt/models",
                "Destination": "/trt-models",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/0426ab0f-0ca1-4841-b7ff-d19932dc4c75/volumes/kubernetes.io~empty-dir/ix-emptydir-volume-frigate-0",
                "Destination": "/tmp/cache",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "frigate-ix-chart-7bf9b5b4f5-b6z4x",
            "Domainname": "",
            "User": "0",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "1935/tcp": {},
                "5000/tcp": {},
                "8554/tcp": {},
                "8555/tcp": {},
                "8555/udp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "NVIDIA_VISIBLE_DEVICES=GPU-83472e62-cd65-1905-5e4d-e2a8c9efc82a",
                "NVIDIA_DRIVER_CAPABILITIES=compute,utility,video",
                "NVIDIA_VISIBLE_DEVICES=all",
                "USE_FP16=False",
                "CUDA_MODULE_LOADING=LAZY",
                "YOLO_MODELS=yolov7x-640",
                "FRIGATE_IX_CHART_SERVICE_PORT_IX_FRIGATE_15000_0=5000",
                "FRIGATE_IX_CHART_PORT_8554_TCP_ADDR=172.17.84.6",
                "FRIGATE_IX_CHART_PORT_8555_TCP=tcp://172.17.84.6:8555",
                "KUBERNETES_SERVICE_HOST=172.17.0.1",
                "FRIGATE_IX_CHART_PORT_5000_TCP_PORT=5000",
                "FRIGATE_IX_CHART_PORT_8554_UDP_PROTO=udp",
                "FRIGATE_IX_CHART_PORT_8554_UDP_PORT=8554",
                "KUBERNETES_PORT_443_TCP_PORT=443",
                "KUBERNETES_PORT_443_TCP_ADDR=172.17.0.1",
                "FRIGATE_IX_CHART_SERVICE_HOST=172.17.84.6",
                "FRIGATE_IX_CHART_SERVICE_PORT_IX_FRIGATE_9555_2=8555",
                "KUBERNETES_SERVICE_PORT_HTTPS=443",
                "FRIGATE_IX_CHART_PORT_8555_TCP_PROTO=tcp",
                "FRIGATE_IX_CHART_PORT_8555_TCP_ADDR=172.17.84.6",
                "KUBERNETES_PORT_443_TCP_PROTO=tcp",
                "FRIGATE_IX_CHART_SERVICE_PORT=5000",
                "FRIGATE_IX_CHART_PORT=tcp://172.17.84.6:5000",
                "FRIGATE_IX_CHART_PORT_5000_TCP_PROTO=tcp",
                "FRIGATE_IX_CHART_PORT_5000_TCP_ADDR=172.17.84.6",
                "FRIGATE_IX_CHART_PORT_8554_TCP_PORT=8554",
                "FRIGATE_IX_CHART_PORT_8554_UDP=udp://172.17.84.6:8554",
                "KUBERNETES_SERVICE_PORT=443",
                "KUBERNETES_PORT=tcp://172.17.0.1:443",
                "FRIGATE_IX_CHART_PORT_5000_TCP=tcp://172.17.84.6:5000",
                "FRIGATE_IX_CHART_PORT_8554_TCP_PROTO=tcp",
                "FRIGATE_IX_CHART_PORT_8554_UDP_ADDR=172.17.84.6",
                "KUBERNETES_PORT_443_TCP=tcp://172.17.0.1:443",
                "FRIGATE_IX_CHART_SERVICE_PORT_IX_FRIGATE_9554_1=8554",
                "FRIGATE_IX_CHART_SERVICE_PORT_IX_FRIGATE_9554_3=8554",
                "FRIGATE_IX_CHART_PORT_8554_TCP=tcp://172.17.84.6:8554",
                "FRIGATE_IX_CHART_PORT_8555_TCP_PORT=8555",
                "PATH=/usr/lib/btbn-ffmpeg/bin:/usr/local/go2rtc/bin:/usr/local/nginx/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "S6_LOGGING_SCRIPT=T 1 n0 s10000000 T",
                "S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0",
                "TRT_VER=8.5.3"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "NONE"
                ]
            },
            "Image": "sha256:da3158e53086ff6734aa52f2dedbdcd385bf5a919ae61642b449fc8bc41b871e",
            "Volumes": null,
            "WorkingDir": "/opt/frigate",
            "Entrypoint": [
                "/init"
            ],
            "OnBuild": null,
            "Labels": {
                "annotation.io.kubernetes.container.hash": "edbef588",
                "annotation.io.kubernetes.container.ports": "[{\"containerPort\":5000,\"protocol\":\"TCP\"},{\"containerPort\":8554,\"protocol\":\"TCP\"},{\"containerPort\":8554,\"protocol\":\"TCP\"},{\"containerPort\":8555,\"protocol\":\"TCP\"}]",
                "annotation.io.kubernetes.container.restartCount": "0",
                "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
                "annotation.io.kubernetes.container.terminationMessagePolicy": "File",
                "annotation.io.kubernetes.pod.terminationGracePeriod": "30",
                "io.kubernetes.container.logpath": "/var/log/pods/ix-frigate_frigate-ix-chart-7bf9b5b4f5-b6z4x_0426ab0f-0ca1-4841-b7ff-d19932dc4c75/ix-chart/0.log",
                "io.kubernetes.container.name": "ix-chart",
                "io.kubernetes.docker.type": "container",
                "io.kubernetes.pod.name": "frigate-ix-chart-7bf9b5b4f5-b6z4x",
                "io.kubernetes.pod.namespace": "ix-frigate",
                "io.kubernetes.pod.uid": "0426ab0f-0ca1-4841-b7ff-d19932dc4c75",
                "io.kubernetes.sandbox.id": "8ac4f445b10b63dbbfd0bd7e2c2b0517eca1a0b32237d9dcf6188dcbf60b5bef"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {}
        }
    }
]

Relevant log output

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service trt-model-prepare: starting
s6-rc: info: service log-prepare: starting
Generating the following TRT Models: yolov7x-640
Downloading yolo weights
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service go2rtc-healthcheck successfully started
2023-09-26 19:49:50.132666515  [INFO] Preparing go2rtc config...
2023-09-26 19:49:51.059534981  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2023-09-26 19:49:51.138696407  [INFO] Starting go2rtc...
2023-09-26 19:49:51.425243073  19:49:51.424 INF go2rtc version 1.7.1 linux/amd64
2023-09-26 19:49:51.426672302  19:49:51.426 INF [api] listen addr=0.0.0.0:1984
2023-09-26 19:49:51.427606373  19:49:51.427 INF [rtsp] listen addr=0.0.0.0:8554
2023-09-26 19:49:51.428146759  19:49:51.428 INF [webrtc] listen addr=0.0.0.0:8555/tcp
2023-09-26 19:50:00.122605295  [INFO] Starting go2rtc healthcheck service...

Creating yolov7x-640.cfg and yolov7x-640.weights

Done.

Generating yolov7x-640.trt. This may take a few minutes.

Cuda failure in file 'yolo_layer.cu' line 51: 35
/etc/s6-overlay/s6-rc.d/trt-model-prepare/run: line 82:   204 Aborted                 $cmd > /tmp/onnx_to_tensorrt.log
Available tensorrt models:
ls: cannot access '*.trt': No such file or directory
s6-rc: warning: unable to start service trt-model-prepare: command exited 2

Operating system

Debian

Install method

Docker CLI

Coral version

Other

Any other information that may be helpful

HPE DL380 Gen10 TrueNAS-SCALE-22.12.3.2 Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz 16 cores (32 threads) 125.6GiB total available (ECC) Tesla P4 GPU

root@frigate-ix-chart-7bf9b5b4f5-b6z4x:/tmp# nvidia-smi Tue Sep 26 20:04:44 2023
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla P4 Off | 00000000:37:00.0 Off | Off | | N/A 52C P0 23W / 75W | 397MiB / 8192MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| +-----------------------------------------------------------------------------+

NickM-27 commented 1 year ago

CC @NateMeyer

slackr31337 commented 1 year ago

After re-reading the beta-2 release notes I see that the Nvidia driver has to be version 530 or >. I'm currently running 515.65.01. I will upgrade the Nvidia driver and see if that resolves this issue.

NateMeyer commented 1 year ago

Hope that driver helps, I'm not sure what else would be the issue. Not sure if that tensorrt log in the tmp folder is accessible if you still have a problem, that might give another clue.

slackr31337 commented 1 year ago

Hello,

I was able to upgrade TrueNAS to TrueNAS-SCALE-23.10-RC.1 which includes NVIDIA driver 535.54.03. The docker container now generates the trt model successfully.

Also, the tensorrt log in the tmp folder was empty.