search

wandb / server

W&B Server is the self hosted version of Weights & Biases
MIT License
263 stars 21 forks source link

Network error (TransientError), entering retry loop. #129

Closed Stashq closed 7 months ago

Stashq commented 1 year ago

Introduction

I am trying to set up W&B server using an external MySQL database and MinIO storage. This is my docker-compose file:

version: "3"

services:

  mysql-db:
    image: mysql:8.1.0
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: wandb_db
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD}
    stdin_open: true
    tty: true
    volumes:
     - ./wandb_persistence/mysql_data:/var/lib/mysql

  minio-storage:
    image: minio/minio:latest
    restart: always
    ports:
      - 9000:9000
      - 9001:9001
    volumes:
      - ./wandb_persistence/minio_data:/data
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
      MINIO_DEFAULT_BUCKETS: wandb-bucket
      MINIO_HTTP_TRACE: /dev/stdout
      LOGGER_CONSOLE_ENABLED: "true"
    command: ["server", "/data"]

  wandb-service:
    image: wandb/local:0.43.0
    restart: always
    ports:
      - 8081:8080
    depends_on:
      - mysql-db
      - minio-storage
    environment:
      HOST: https://wandb_service
      WANDB_BASE_URL: https://wandb_service
      LICENSE: ${WANDB_LICENSE}
      LOGGING_ENABLED: "true"
      LOCAL_RESTORE: "true"
      BUCKET: s3://${MINIO_ACCESS_KEY}:${MINIO_SECRET_KEY}@minio-storage:9000/wandb-bucket
      BUCKET_QUEUE: internal://
      AWS_REGION: eu-central-1
      MYSQL: mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@mysql-db/wandb_db

After I start MinIO, I manually create access and secret keys.

Problem description

Whenever I try to log to W&B, error with folowing message occures:

wandb: Network error (TransientError), entering retry loop.

Simple metrics are passed to W&B server, but I can not log artifacts like pytorch models. At the end the program stuck in the loop showing this message. When I check the wandb server logs I see an error: "key does not exist".

wandb-service_1 | {"level":"ERROR","time":"2023-10-17T12:21:48.756074416Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/pkg/observability/gerr/reporting.go:193","pid":57},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","projectID":3,"cacheKey":"project:3:approx_run_count"},"message":"key does not exist","dd.trace_id":"12901086739434798565","error":{"kind":"*errors.errorString","message":"key does not exist"}}

What works correctly

W&B logs

All logs are listed below:

wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:41.334681373Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"12597041581390036685","path":"/files/username/mnist-try/mffkn5s9/file_stream"},"message":"Starting request 12597041581390036685 on /files/username/mnist-try/mffkn5s9/file_stream","dd.trace_id":"12597041581390036685"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:41.33812557Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":57},"data":{"entityName":"username","projectName":"mnist-try","runName":"mffkn5s9","projectName":"mnist-try","runName":"mffkn5s9","username":"username","cliVersion":"0.13.11","entityName":"username","runSize":1137,"keysInfoSize":0,"summarySize":0,"configSize":4,"summaryUpdated":false,"keysInfoUpdated":false},"message":"file_stream: logged run of size 1137, keysinfo size 0, summary size 0, config size 4, summaryUpdated false, keysInfoUpdated false","dd.trace_id":"12597041581390036685"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:41.350411195Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"12597041581390036685","duration":"15.73174ms","statusCode":200,"path":"/files/username/mnist-try/mffkn5s9/file_stream"},"message":"Finished request 12597041581390036685 in 15.73174ms with status %!s(int=200) on /files/username/mnist-try/mffkn5s9/file_stream","dd.trace_id":"12597041581390036685"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:44.811070835Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"13216365286806201075","path":"/files/username/mnist-try/gutqi4ed/file_stream"},"message":"Starting request 13216365286806201075 on /files/username/mnist-try/gutqi4ed/file_stream","dd.trace_id":"13216365286806201075"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:44.818343975Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":57},"data":{"runName":"gutqi4ed","entityName":"username","projectName":"mnist-try","entityName":"username","projectName":"mnist-try","runName":"gutqi4ed","username":"username","cliVersion":"0.13.11","runSize":802,"keysInfoSize":0,"summarySize":0,"configSize":4,"summaryUpdated":false,"keysInfoUpdated":false},"message":"file_stream: logged run of size 802, keysinfo size 0, summary size 0, config size 4, summaryUpdated false, keysInfoUpdated false","dd.trace_id":"13216365286806201075"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:44.826303293Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"13216365286806201075","duration":"15.224485ms","statusCode":200,"path":"/files/username/mnist-try/gutqi4ed/file_stream"},"message":"Finished request 13216365286806201075 in 15.224485ms with status %!s(int=200) on /files/username/mnist-try/gutqi4ed/file_stream","dd.trace_id":"13216365286806201075"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.492662838Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"13502922608657722124","path":"/graphql"},"message":"Starting request 13502922608657722124 on /graphql","dd.trace_id":"13502922608657722124"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.495202997Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":57},"data":{"authUser":"username","userID":4,"operationName":"ProbeServerCapabilities","operationName":"ProbeServerCapabilities","authUser":"username","variables":{},"appPath":"N/A"},"message":"Graphql operation ProbeServerCapabilities for user username with variables map[] from app path N/A","dd.trace_id":"13502922608657722124"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.495601322Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":57},"data":{"authUser":"username","userID":4,"operationName":"ProbeServerCapabilities","latencyNs":395166,"statusCode":200,"operationName":"ProbeServerCapabilities","authUser":"username","latencyStr":"395.166µs"},"message":"Graphql operation ProbeServerCapabilities for user username finished in 395.166µs","dd.trace_id":"13502922608657722124"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.495935381Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"13502922608657722124","duration":"3.270998ms","statusCode":200,"path":"/graphql"},"message":"Finished request 13502922608657722124 in 3.270998ms with status %!s(int=200) on /graphql","dd.trace_id":"13502922608657722124"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.499924402Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"11201157216694702900","path":"/graphql"},"message":"Starting request 11201157216694702900 on /graphql","dd.trace_id":"11201157216694702900"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.501787486Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":57},"data":{"authUser":"username","userID":4,"operationName":"Viewer","operationName":"Viewer","authUser":"username","variables":{},"appPath":"N/A"},"message":"Graphql operation Viewer for user username with variables map[] from app path N/A","dd.trace_id":"11201157216694702900"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.522838536Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":57},"data":{"authUser":"username","userID":4,"operationName":"Viewer","latencyNs":21034460,"statusCode":200,"operationName":"Viewer","authUser":"username","latencyStr":"21.03446ms"},"message":"Graphql operation Viewer for user username finished in 21.03446ms","dd.trace_id":"11201157216694702900"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.52330615Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"11201157216694702900","duration":"23.380906ms","statusCode":200,"path":"/graphql"},"message":"Finished request 11201157216694702900 in 23.380906ms with status %!s(int=200) on /graphql","dd.trace_id":"11201157216694702900"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.553850085Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"11514604295870751889","path":"/graphql"},"message":"Starting request 11514604295870751889 on /graphql","dd.trace_id":"11514604295870751889"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.556240424Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":57},"data":{"operationName":"RunResumeStatus","authUser":"username","userID":4,"operationName":"RunResumeStatus","authUser":"username","variables":{"project":"mnist-try","name":"g0eoviev","entity":null},"appPath":"N/A"},"message":"Graphql operation RunResumeStatus for user username with variables map[entity:<nil> name:g0eoviev project:mnist-try] from app path N/A","dd.trace_id":"11514604295870751889"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.681651274Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":57},"data":{"operationName":"RunResumeStatus","authUser":"username","userID":4,"latencyNs":125391007,"statusCode":200,"operationName":"RunResumeStatus","authUser":"username","latencyStr":"125.391007ms"},"message":"Graphql operation RunResumeStatus for user username finished in 125.391007ms","dd.trace_id":"11514604295870751889"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.682296623Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"11514604295870751889","duration":"128.43544ms","statusCode":200,"path":"/graphql"},"message":"Finished request 11514604295870751889 in 128.43544ms with status %!s(int=200) on /graphql","dd.trace_id":"11514604295870751889"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.694038072Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"10097934739550959853","path":"/graphql"},"message":"Starting request 10097934739550959853 on /graphql","dd.trace_id":"10097934739550959853"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.69946085Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":57},"data":{"authUser":"username","userID":4,"operationName":"ProbeServerSettings","operationName":"ProbeServerSettings","authUser":"username","variables":{},"appPath":"N/A"},"message":"Graphql operation ProbeServerSettings for user username with variables map[] from app path N/A","dd.trace_id":"10097934739550959853"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.69977299Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":57},"data":{"authUser":"username","userID":4,"operationName":"ProbeServerSettings","latencyNs":261597,"statusCode":200,"operationName":"ProbeServerSettings","authUser":"username","latencyStr":"261.597µs"},"message":"Graphql operation ProbeServerSettings for user username finished in 261.597µs","dd.trace_id":"10097934739550959853"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.700432851Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"10097934739550959853","duration":"6.379629ms","statusCode":200,"path":"/graphql"},"message":"Finished request 10097934739550959853 in 6.379629ms with status %!s(int=200) on /graphql","dd.trace_id":"10097934739550959853"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.723330961Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"12901086739434798565","path":"/graphql"},"message":"Starting request 12901086739434798565 on /graphql","dd.trace_id":"12901086739434798565"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.727542646Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":57},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","operationName":"UpsertBucket","authUser":"username","variables":{"id":null,"state":null,"displayName":null,"commit":"d29dffbc8dc894f8ab36a426ab2998251c62c1d2","tags":null,"summaryMetrics":null,"sweep":null,"notes":null,"groupName":null,"name":"g0eoviev","description":null,"config":"{\"_wandb\": {\"desc\": null, \"value\": {\"code_path\": \"code/src/runs/... (+ 333 bytes)","host":"stachu-GV62-8RE","program":"/home/stachu/Projects/airplane-damage-detection/src/runs/wandb_testing/lightning.py","repo":"https://github.com/Logistics-Living-Lab/Aircraft-damage-detection.git","jobType":null,"entity":"username","debug":false,"project":"mnist-try"},"appPath":"N/A"},"message":"Graphql operation UpsertBucket for user username with variables map[commit:d29dffbc8dc894f8ab36a426ab2998251c62c1d2 config:{\"_wandb\": {\"desc\": null, \"value\": {\"code_path\": \"code/src/runs/... (+ 333 bytes) debug:false description:<nil> displayName:<nil> entity:username groupName:<nil> host:stachu-GV62-8RE id:<nil> jobType:<nil> name:g0eoviev notes:<nil> program:/home/stachu/Projects/airplane-damage-detection/src/runs/wandb_testing/lightning.py project:mnist-try repo:https://github.com/Logistics-Living-Lab/Aircraft-damage-detection.git state:<nil> summaryMetrics:<nil> sweep:<nil> tags:<nil>] from app path N/A","dd.trace_id":"12901086739434798565"}
wandb-service_1  | {"level":"ERROR","time":"2023-10-17T12:21:48.756074416Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/pkg/observability/gerr/reporting.go:193","pid":57},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","projectID":3,"cacheKey":"project:3:approx_run_count"},"message":"key does not exist","dd.trace_id":"12901086739434798565","error":{"kind":"*errors.errorString","message":"key does not exist"}}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.77665372Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":57},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","statusCode":200,"latencyNs":49133087,"operationName":"UpsertBucket","authUser":"username","latencyStr":"49.133087ms"},"message":"Graphql operation UpsertBucket for user username finished in 49.133087ms","dd.trace_id":"12901086739434798565"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:48.776906388Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"12901086739434798565","duration":"53.593056ms","statusCode":200,"path":"/graphql"},"message":"Finished request 12901086739434798565 in 53.593056ms with status %!s(int=200) on /graphql","dd.trace_id":"12901086739434798565"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.099256289Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"13539481140370043043","path":"/graphql"},"message":"Starting request 13539481140370043043 on /graphql","dd.trace_id":"13539481140370043043"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.101045746Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":57},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","operationName":"RunUploadUrls","authUser":"username","variables":{"name":"mnist-try","run":"g0eoviev","entity":"username","description":null,"files":["wandb-metadata.json"]},"appPath":"N/A"},"message":"Graphql operation RunUploadUrls for user username with variables map[description:<nil> entity:username files:[wandb-metadata.json] name:mnist-try run:g0eoviev] from app path N/A","dd.trace_id":"13539481140370043043"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.10880711Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"10591353008052854308","path":"/graphql"},"message":"Starting request 10591353008052854308 on /graphql","dd.trace_id":"10591353008052854308"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.109092427Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"11768831318845607199","path":"/graphql"},"message":"Starting request 11768831318845607199 on /graphql","dd.trace_id":"11768831318845607199"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.111287698Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":57},"data":{"operationName":"RunStoppedStatus","authUser":"username","userID":4,"operationName":"RunStoppedStatus","authUser":"username","variables":{"projectName":"mnist-try","entityName":"username","runId":"g0eoviev"},"appPath":"N/A"},"message":"Graphql operation RunStoppedStatus for user username with variables map[entityName:username projectName:mnist-try runId:g0eoviev] from app path N/A","dd.trace_id":"11768831318845607199"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.111406649Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":57},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","operationName":"RunUploadUrls","authUser":"username","variables":{"name":"mnist-try","run":"g0eoviev","entity":"username","description":null,"files":["code/src/runs/wandb_testing/lightning.py"]},"appPath":"N/A"},"message":"Graphql operation RunUploadUrls for user username with variables map[description:<nil> entity:username files:[code/src/runs/wandb_testing/lightning.py] name:mnist-try run:g0eoviev] from app path N/A","dd.trace_id":"10591353008052854308"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.21066378Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":57},"data":{"operationName":"RunStoppedStatus","authUser":"username","userID":4,"latencyNs":99384511,"statusCode":200,"operationName":"RunStoppedStatus","authUser":"username","latencyStr":"99.384511ms"},"message":"Graphql operation RunStoppedStatus for user username finished in 99.384511ms","dd.trace_id":"11768831318845607199"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.210934935Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"11768831318845607199","duration":"101.836331ms","statusCode":200,"path":"/graphql"},"message":"Finished request 11768831318845607199 in 101.836331ms with status %!s(int=200) on /graphql","dd.trace_id":"11768831318845607199"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.221718236Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":57},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","latencyNs":120663698,"statusCode":200,"operationName":"RunUploadUrls","authUser":"username","latencyStr":"120.663698ms"},"message":"Graphql operation RunUploadUrls for user username finished in 120.663698ms","dd.trace_id":"13539481140370043043"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.2217616Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":57},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","statusCode":200,"latencyNs":110360790,"operationName":"RunUploadUrls","authUser":"username","latencyStr":"110.36079ms"},"message":"Graphql operation RunUploadUrls for user username finished in 110.36079ms","dd.trace_id":"10591353008052854308"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.22207427Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"13539481140370043043","duration":"122.817242ms","statusCode":200,"path":"/graphql"},"message":"Finished request 13539481140370043043 in 122.817242ms with status %!s(int=200) on /graphql","dd.trace_id":"13539481140370043043"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:49.222261147Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"10591353008052854308","duration":"113.453098ms","statusCode":200,"path":"/graphql"},"message":"Finished request 10591353008052854308 in 113.453098ms with status %!s(int=200) on /graphql","dd.trace_id":"10591353008052854308"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:50.793601563Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"12672516382616715536","path":"/files/username/mnist-try/g0eoviev/file_stream"},"message":"Starting request 12672516382616715536 on /files/username/mnist-try/g0eoviev/file_stream","dd.trace_id":"12672516382616715536"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:50.795968631Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":57},"data":{"entityName":"username","projectName":"mnist-try","runName":"g0eoviev","entityName":"username","projectName":"mnist-try","runName":"g0eoviev","username":"username","cliVersion":"0.13.11","runSize":105,"keysInfoSize":0,"summarySize":4,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":false},"message":"file_stream: logged run of size 105, keysinfo size 0, summary size 4, config size 4, summaryUpdated true, keysInfoUpdated false","dd.trace_id":"12672516382616715536"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:50.884928141Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"12672516382616715536","duration":"91.287767ms","statusCode":200,"path":"/files/username/mnist-try/g0eoviev/file_stream"},"message":"Finished request 12672516382616715536 in 91.287767ms with status %!s(int=200) on /files/username/mnist-try/g0eoviev/file_stream","dd.trace_id":"12672516382616715536"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:53.265804288Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":57},"data":{"requestID":"10410415496601539006","path":"/files/username/mnist-try/g0eoviev/file_stream"},"message":"Starting request 10410415496601539006 on /files/username/mnist-try/g0eoviev/file_stream","dd.trace_id":"10410415496601539006"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:53.921932624Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":57},"data":{"projectName":"mnist-try","runName":"g0eoviev","entityName":"username","username":"username","cliVersion":"0.13.11","entityName":"username","projectName":"mnist-try","runName":"g0eoviev","runSize":105,"keysInfoSize":0,"summarySize":0,"configSize":4,"summaryUpdated":false,"keysInfoUpdated":false},"message":"file_stream: logged run of size 105, keysinfo size 0, summary size 0, config size 4, summaryUpdated false, keysInfoUpdated false","dd.trace_id":"10410415496601539006"}
wandb-service_1  | {"level":"INFO","time":"2023-10-17T12:21:53.957738876Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":57},"data":{"requestID":"10410415496601539006","duration":"691.941693ms","statusCode":200,"path":"/files/username/mnist-try/g0eoviev/file_stream"},"message":"Finished request 10410415496601539006 in 691.941693ms with status %!s(int=200) on /files/username/mnist-try/g0eoviev/file_stream","dd.trace_id":"10410415496601539006"}

MinIO check

I've also checked MinIO http logs using mc admin trace command. After W&B process finishes, I get this error:

2023-10-18T09:39:26.794 [404 Not Found] s3.HeadObject minio-storage:9000/wandb-bucket/wandb_artifacts/2/2/d89e2164c12545e83ae6d568565906c6 172.29.0.4       825µs       ⇣  0s         ↑ 77 B ↓ 0 B
2023-10-18T09:40:00.047 [200 OK] s3.PutObject minio-storage:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz 172.29.0.4       7.12ms       ⇣  0s         ↑ 396 B ↓ 0 B
2023-10-18T09:41:53.642 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       415µs       ⇣  380.444µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:53.643 [200 OK] s3.ListObjectVersions 127.0.0.1:9000/wandb-bucket/?delimiter=%2F&encoding-type=url&max-keys=100&prefix=wandb-audit-logs%2Ffrom_2023_10_18_09_02_56.jsonl.txt.gz&versions=  172.29.0.1       1.152ms      ⇣  1.122664ms  ↑ 114 B ↓ 2.2 KiB
2023-10-18T09:41:53.646 [400 Bad Request] s3.GetObjectLegalHold 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?legal-hold=&versionId=3070c2cf-f6aa-478b-9204-15df13f0174d  172.29.0.1       436µs       ⇣  421.256µs  ↑ 114 B ↓ 487 B
2023-10-18T09:41:53.647 [400 Bad Request] s3.GetObjectRetention 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?retention=&versionId=3070c2cf-f6aa-478b-9204-15df13f0174d  172.29.0.1       557µs       ⇣  520.843µs  ↑ 114 B ↓ 487 B
2023-10-18T09:41:53.648 [200 OK] s3.GetObjectTagging 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?tagging=&versionId=3070c2cf-f6aa-478b-9204-15df13f0174d  172.29.0.1       321µs       ⇣  311.05µs  ↑ 114 B ↓ 75 B
2023-10-18T09:41:53.649 [400 Bad Request] s3.GetObjectLegalHold 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?legal-hold=&versionId=06e2c066-d00c-4774-9d20-e013fb13f739  172.29.0.1       396µs       ⇣  383.039µs  ↑ 114 B ↓ 487 B
2023-10-18T09:41:53.650 [400 Bad Request] s3.GetObjectRetention 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?retention=&versionId=06e2c066-d00c-4774-9d20-e013fb13f739  172.29.0.1       285µs       ⇣  274.806µs  ↑ 114 B ↓ 487 B
2023-10-18T09:41:53.651 [200 OK] s3.GetObjectTagging 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?tagging=&versionId=06e2c066-d00c-4774-9d20-e013fb13f739  172.29.0.1       281µs       ⇣  272.542µs  ↑ 114 B ↓ 75 B
2023-10-18T09:41:53.652 [400 Bad Request] s3.GetObjectLegalHold 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?legal-hold=&versionId=a5112cb2-5a06-4b7d-bd15-87f8447bff5c  172.29.0.1       261µs       ⇣  252.927µs  ↑ 114 B ↓ 487 B
2023-10-18T09:41:53.652 [400 Bad Request] s3.GetObjectRetention 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?retention=&versionId=a5112cb2-5a06-4b7d-bd15-87f8447bff5c  172.29.0.1       254µs       ⇣  245.906µs  ↑ 114 B ↓ 487 B
2023-10-18T09:41:53.653 [200 OK] s3.GetObjectTagging 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?tagging=&versionId=a5112cb2-5a06-4b7d-bd15-87f8447bff5c  172.29.0.1       294µs       ⇣  285.941µs  ↑ 114 B ↓ 75 B
2023-10-18T09:41:53.654 [400 Bad Request] s3.GetObjectLegalHold 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?legal-hold=&versionId=800771ef-e5fc-49f4-8104-d1ebe3fa45bd  172.29.0.1       367µs       ⇣  358.024µs  ↑ 114 B ↓ 487 B
2023-10-18T09:41:53.655 [400 Bad Request] s3.GetObjectRetention 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?retention=&versionId=800771ef-e5fc-49f4-8104-d1ebe3fa45bd  172.29.0.1       431µs       ⇣  419.593µs  ↑ 114 B ↓ 487 B
2023-10-18T09:41:53.656 [200 OK] s3.GetObjectTagging 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz?tagging=&versionId=800771ef-e5fc-49f4-8104-d1ebe3fa45bd  172.29.0.1       566µs       ⇣  546.169µs  ↑ 114 B ↓ 75 B
2023-10-18T09:41:53.677 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       402µs       ⇣  386.766µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:53.678 [200 OK] s3.HeadObject 127.0.0.1:9000/wandb-bucket/wandb-audit-logs/from_2023_10_18_09_02_56.jsonl.txt.gz 172.29.0.1       656µs       ⇣  0s         ↑ 114 B ↓ 0 B
2023-10-18T09:41:54.588 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       408µs       ⇣  394.408µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:54.588 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       263µs       ⇣  248.166µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:54.589 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       293µs       ⇣  273.774µs  ↑ 114 B ↓ 145 B
2023-10-18T09:41:54.589 [200 OK] s3.GetBucketVersioning 127.0.0.1:9000/wandb-bucket/?versioning=  172.29.0.1       363µs       ⇣  347.243µs  ↑ 114 B ↓ 123 B
2023-10-18T09:41:55.345 [200 OK] s3.ListObjectsV2 127.0.0.1:9000/wandb-bucket/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=  172.29.0.1       822µs       ⇣  791.543µs  ↑ 114 B ↓ 698 B
2023-10-18T09:41:55.362 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       487µs       ⇣  439.116µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:55.362 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       622µs       ⇣  493.186µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:55.363 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       323µs       ⇣  308.036µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:55.363 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       364µs       ⇣  342.362µs  ↑ 114 B ↓ 145 B
2023-10-18T09:41:55.363 [200 OK] s3.GetBucketVersioning 127.0.0.1:9000/wandb-bucket/?versioning=  172.29.0.1       276µs       ⇣  262.313µs  ↑ 114 B ↓ 123 B
2023-10-18T09:41:55.363 [200 OK] s3.GetBucketPolicy 127.0.0.1:9000/wandb-bucket/?policy=  172.29.0.1       763µs       ⇣  745.955µs  ↑ 114 B ↓ 409 B
2023-10-18T09:41:55.365 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/wandb-bucket/?tagging=  172.29.0.1       225µs       ⇣  210.438µs  ↑ 114 B ↓ 315 B
2023-10-18T09:41:58.917 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       308µs       ⇣  287.655µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:58.917 [200 OK] s3.ListObjectsV2 127.0.0.1:9000/wandb-bucket/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=  172.29.0.1       1.988ms      ⇣  1.967515ms  ↑ 114 B ↓ 698 B
2023-10-18T09:41:58.920 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       193µs       ⇣  180.879µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:58.921 [200 OK] s3.GetBucketPolicy 127.0.0.1:9000/wandb-bucket/?policy=  172.29.0.1       610µs       ⇣  594.934µs  ↑ 114 B ↓ 409 B
2023-10-18T09:41:58.922 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/wandb-bucket/?tagging=  172.29.0.1       152µs       ⇣  142.277µs  ↑ 114 B ↓ 315 B
2023-10-18T09:41:58.924 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       196µs       ⇣  186.094µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:58.925 [200 OK] s3.GetBucketVersioning 127.0.0.1:9000/wandb-bucket/?versioning=  172.29.0.1       179µs       ⇣  169.888µs  ↑ 114 B ↓ 123 B
2023-10-18T09:41:58.925 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       213µs       ⇣  201.986µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:58.926 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       203µs       ⇣  191.775µs  ↑ 114 B ↓ 145 B
2023-10-18T09:41:58.935 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       265µs       ⇣  249.822µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:58.935 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       319µs       ⇣  309.217µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:58.936 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       275µs       ⇣  255.62µs  ↑ 114 B ↓ 145 B
2023-10-18T09:41:58.936 [200 OK] s3.GetBucketPolicy 127.0.0.1:9000/wandb-bucket/?policy=  172.29.0.1       899µs       ⇣  879.281µs  ↑ 114 B ↓ 409 B
2023-10-18T09:41:58.937 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/wandb-bucket/?tagging=  172.29.0.1       237µs       ⇣  221.294µs  ↑ 114 B ↓ 315 B
2023-10-18T09:41:58.969 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       321µs       ⇣  304.502µs  ↑ 114 B ↓ 128 B
2023-10-18T09:41:58.970 [200 OK] s3.GetBucketPolicy 127.0.0.1:9000/wandb-bucket/?policy=  172.29.0.1       744µs       ⇣  711.575µs  ↑ 114 B ↓ 409 B
2023-10-18T09:41:58.972 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/wandb-bucket/?tagging=  172.29.0.1       261µs       ⇣  240.216µs  ↑ 114 B ↓ 315 B
2023-10-18T09:42:00.418 [200 OK] s3.ListObjectsV2 127.0.0.1:9000/wandb-bucket/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=wandb-audit-logs%2F  172.29.0.1       1.096ms      ⇣  1.082737ms  ↑ 114 B ↓ 683 B
2023-10-18T09:42:00.422 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       244µs       ⇣  232.125µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:00.423 [200 OK] s3.ListObjectVersions 127.0.0.1:9000/wandb-bucket/?delimiter=%2F&encoding-type=url&max-keys=100&prefix=wandb-audit-logs%2F&versions=  172.29.0.1       498µs       ⇣  488.325µs  ↑ 114 B ↓ 2.1 KiB
2023-10-18T09:42:00.431 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       213µs       ⇣  201.086µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:00.431 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       219µs       ⇣  204.823µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:00.432 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       232µs       ⇣  215.435µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:00.432 [200 OK] s3.GetBucketVersioning 127.0.0.1:9000/wandb-bucket/?versioning=  172.29.0.1       245µs       ⇣  231.427µs  ↑ 114 B ↓ 123 B
2023-10-18T09:42:00.432 [200 OK] s3.GetBucketPolicy 127.0.0.1:9000/wandb-bucket/?policy=  172.29.0.1       772µs       ⇣  745.002µs  ↑ 114 B ↓ 409 B
2023-10-18T09:42:00.433 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       342µs       ⇣  327.273µs  ↑ 114 B ↓ 145 B
2023-10-18T09:42:00.433 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/wandb-bucket/?tagging=  172.29.0.1       339µs       ⇣  309.123µs  ↑ 114 B ↓ 315 B
2023-10-18T09:42:07.753 [200 OK] s3.ListObjectsV2 127.0.0.1:9000/wandb-bucket/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=  172.29.0.1       904µs       ⇣  877.889µs  ↑ 114 B ↓ 698 B
2023-10-18T09:42:07.771 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       334µs       ⇣  318.843µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:07.771 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       591µs       ⇣  554.669µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:07.772 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       275µs       ⇣  249.065µs  ↑ 114 B ↓ 145 B
2023-10-18T09:42:07.772 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       375µs       ⇣  350.916µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:07.773 [200 OK] s3.GetBucketVersioning 127.0.0.1:9000/wandb-bucket/?versioning=  172.29.0.1       654µs       ⇣  604.277µs  ↑ 114 B ↓ 123 B
2023-10-18T09:42:07.774 [200 OK] s3.GetBucketPolicy 127.0.0.1:9000/wandb-bucket/?policy=  172.29.0.1       1.287ms      ⇣  1.262464ms  ↑ 114 B ↓ 409 B
2023-10-18T09:42:07.776 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/wandb-bucket/?tagging=  172.29.0.1       275µs       ⇣  256.622µs  ↑ 114 B ↓ 315 B
2023-10-18T09:42:14.016 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       186µs       ⇣  176.169µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:14.017 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       187µs       ⇣  177.21µs  ↑ 114 B ↓ 145 B
2023-10-18T09:42:14.032 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       295µs       ⇣  278.942µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:14.033 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       188µs       ⇣  176.722µs  ↑ 114 B ↓ 145 B
2023-10-18T09:42:14.033 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       223µs       ⇣  214.582µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:14.033 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       242µs       ⇣  232.019µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:14.033 [200 OK] s3.GetBucketVersioning 127.0.0.1:9000/wandb-bucket/?versioning=  172.29.0.1       217µs       ⇣  208.427µs  ↑ 114 B ↓ 123 B
2023-10-18T09:42:14.033 [200 OK] s3.GetBucketPolicy 127.0.0.1:9000/wandb-bucket/?policy=  172.29.0.1       657µs       ⇣  644.447µs  ↑ 114 B ↓ 409 B
2023-10-18T09:42:14.034 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/wandb-bucket/?tagging=  172.29.0.1       170µs       ⇣  160.885µs  ↑ 114 B ↓ 315 B
2023-10-18T09:42:16.529 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       239µs       ⇣  220.943µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:16.529 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       237µs       ⇣  216.02µs  ↑ 114 B ↓ 145 B
2023-10-18T09:42:16.542 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       259µs       ⇣  245.448µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:16.543 [200 OK] s3.GetBucketPolicy 127.0.0.1:9000/wandb-bucket/?policy=  172.29.0.1       576µs       ⇣  556.791µs  ↑ 114 B ↓ 409 B
2023-10-18T09:42:16.544 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/wandb-bucket/?tagging=  172.29.0.1       296µs       ⇣  279.881µs  ↑ 114 B ↓ 315 B
2023-10-18T09:42:16.545 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       342µs       ⇣  319.556µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:16.545 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       391µs       ⇣  372.011µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:16.545 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       333µs       ⇣  302.859µs  ↑ 114 B ↓ 145 B
2023-10-18T09:42:16.545 [200 OK] s3.GetBucketVersioning 127.0.0.1:9000/wandb-bucket/?versioning=  172.29.0.1       432µs       ⇣  415.131µs  ↑ 114 B ↓ 123 B
2023-10-18T09:42:18.548 [200 OK] s3.ListObjectsV2 127.0.0.1:9000/wandb-bucket/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=wandb-audit-logs%2F  172.29.0.1       1.724ms      ⇣  1.70825ms  ↑ 114 B ↓ 683 B
2023-10-18T09:42:18.565 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       527µs       ⇣  484.202µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:18.565 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       526µs       ⇣  485.595µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:18.565 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/wandb-bucket/?location=  172.29.0.1       523µs       ⇣  485.976µs  ↑ 114 B ↓ 128 B
2023-10-18T09:42:18.566 [200 OK] s3.GetBucketObjectLockConfig 127.0.0.1:9000/wandb-bucket/?object-lock=  172.29.0.1       292µs       ⇣  275.794µs  ↑ 114 B ↓ 145 B
2023-10-18T09:42:18.566 [200 OK] s3.GetBucketVersioning 127.0.0.1:9000/wandb-bucket/?versioning=  172.29.0.1       369µs       ⇣  327.649µs  ↑ 114 B ↓ 123 B
2023-10-18T09:42:18.567 [200 OK] s3.GetBucketPolicy 127.0.0.1:9000/wandb-bucket/?policy=  172.29.0.1       514µs       ⇣  498.261µs  ↑ 114 B ↓ 409 B
2023-10-18T09:42:18.570 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/wandb-bucket/?tagging=  172.29.0.1       510µs       ⇣  470.748µs  ↑ 114 B ↓ 315 B

W&B verify

Command wandb verify gives me the same error:

Default host selected: http://localhost:8081
Find detailed logs for this test at: /tmp/tmpw87gjayi/wandb
Checking if logged in...................................................✅
wandb: Network error (TransientError), entering retry loop.

Question

Does anyone have an idea why this error occurs? Is it a problem of the wrong W&B or MinIO setup? Has anybody seen that error before? Thx in advance.

nate-wandb commented 1 year ago

Hi @Stashq, could you possibly let us know the output of running wandb verify from the CLI after you have ran wandb login?

Stashq commented 1 year ago

When I run wandb verify I get the same error:

Default host selected: http://localhost:8081
Find detailed logs for this test at: /tmp/tmpw87gjayi/wandb
Checking if logged in...................................................✅
wandb: Network error (TransientError), entering retry loop.

Ping between W&B server and MinIO works correctly by the way.

anmolmann commented 1 year ago

Hey @Stashq, apologies for the delay here. Network error probably means your machine (or pod/node) isn't able to connect to the server. Do you have any VPN/Proxy configured on your end? Wandb uses service from the minio bucket for storing artifacts/files. Due to this, it requires internet access [maybe via proxy] to log the files/artifacts. Wandb forwards bucket URLs to users and it always requires a proxy to reach. So, you would need to use a proxy to access it.

anmolmann commented 1 year ago

Hi @Stashq, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

Stashq commented 1 year ago

@anmolmann thanks a lot for your reply. I'm not able to check the proxy solution in the next 7 days, but I'll definitely come back to this very soon and inform you about the result.

anmolmann commented 1 year ago

Thanks for the update, @Stashq! Please feel free to write in again and share your results with us.

vanpelt commented 1 year ago

@Stashq the key does not exist is likely indication the minio credentials are either incorrect or not encoded properly. When putting credentials into the URL you must be sure they are url encoded to escape any special characters.

vanpelt commented 1 year ago

Another thing to try that isn't well supported but should work is setting the BUCKET_PROXY=true env variable to the server. That will make the server proxy to the bucket.

Stashq commented 1 year ago

@vanpelt thanks for your answer. My minio access and secret keys were suggested by minio and consist only of numbers and letters. I guess the error message "key does not exist" refers to non-existing objects in minio storage.

I tried setting BUCKET_PROXY: "true" and I got rid of NetworkError. Nevertheless I got new errors, perhaps better describing the source of the problem.

wandb: ERROR Error uploading "wandb-metadata.json": CommError, <Response [405]>
wandb: ERROR Error uploading "code/src/runs/wandb/testing/log_metrics.py": CommError, <Response [405]>
wandb: Waiting for W&B process to finish... (success).
wandb: ERROR Error uploading "/home/stachu/.local/share/wandb/artifacts/staging/tmpojqcwy6h": CommError, <Response [405]>
wandb: ERROR Uploading artifact file failed. Artifact won't be committed.
wandb: ERROR Error uploading "/home/stachu/.local/share/wandb/artifacts/staging/tmpyyeqsabv": CommError, <Response [405]>
wandb: ERROR Uploading artifact file failed. Artifact won't be committed.
wandb: ERROR Error uploading "config.yaml": CommError, <Response [405]>
wandb: ERROR Error uploading "requirements.txt": CommError, <Response [405]>
wandb: ERROR Error uploading "wandb-summary.json": CommError, <Response [405]>

I set bucket read/write access as public, but the error remains.

This are the logs from wandb-server:

wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.453213334Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:567","pid":69},"data":{},"message":"initialized stats","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.453358745Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:580","pid":69},"data":{},"message":"created secret store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.453431168Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:587","pid":69},"data":{},"message":"created tracer","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.456661212Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:608","pid":69},"data":{},"message":"read schema","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.47597925Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:655","pid":69},"data":{},"message":"created history store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.476166213Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:695","pid":69},"data":{},"message":"created file store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.47962581Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:741","pid":69},"data":{},"message":"created metadata store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.479649438Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:742","pid":69},"data":{},"message":"created audit store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.479745132Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:751","pid":69},"data":{},"message":"created dataframe store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.479765567Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:764","pid":69},"data":{},"message":"created repo insights plot store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.4798839Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:782","pid":69},"data":{},"message":"created file store provider","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.480166334Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:808","pid":69},"data":{},"message":"created artifacts bucket","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.480191986Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:829","pid":69},"data":{},"message":"created task executor","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.491796654Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:841","pid":69},"data":{},"message":"created file stream store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.491846642Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:851","pid":69},"data":{},"message":"created email sink","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.499430161Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:865","pid":69},"data":{},"message":"created analytics sink","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.499469954Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:892","pid":69},"data":{},"message":"created notifier","dd.trace_id":""}
wandb-service_1  | sweep provider is: http://127.0.0.1:8082
wandb-service_1  | http://127.0.0.1:8082
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.499493828Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:915","pid":69},"data":{},"message":"created sweep provider","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.499893672Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:920","pid":69},"data":{},"message":"created locker","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.499917361Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:940","pid":69},"data":{},"message":"created auth store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.500154789Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:952","pid":69},"data":{},"message":"sweep run queues enabled: %!t(float64=0)","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.500210633Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:1045","pid":69},"data":{},"message":"created customer secret store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.53802361Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:1193","pid":69},"data":{},"message":"parsed schema","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.540339129Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:1494","pid":69},"data":{},"message":"created session store","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:10.540409778Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/cmd/gorilla.go:1603","pid":69},"data":{"port":"8081"},"message":"Listening on 8081","dd.trace_id":""}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.557379241Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"9942339472638752094","path":"/graphql"},"message":"Starting request 9942339472638752094 on /graphql","dd.trace_id":"9942339472638752094"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.573666069Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"operationName":"ProbeServerCapabilities","authUser":"username","userID":4,"operationName":"ProbeServerCapabilities","authUser":"username","variables":{},"appPath":"N/A"},"message":"Graphql operation ProbeServerCapabilities for user username with variables map[] from app path N/A","dd.trace_id":"9942339472638752094"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.574480116Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"operationName":"ProbeServerCapabilities","authUser":"username","userID":4,"latencyNs":814024,"statusCode":200,"operationName":"ProbeServerCapabilities","authUser":"username","latencyStr":"814.024µs"},"message":"Graphql operation ProbeServerCapabilities for user username finished in 814.024µs","dd.trace_id":"9942339472638752094"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.585374783Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"9942339472638752094","duration":"28.009764ms","statusCode":200,"path":"/graphql"},"message":"Finished request 9942339472638752094 in 28.009764ms with status %!s(int=200) on /graphql","dd.trace_id":"9942339472638752094"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.597647546Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"11342671108643749252","path":"/graphql"},"message":"Starting request 11342671108643749252 on /graphql","dd.trace_id":"11342671108643749252"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.601242175Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"Viewer","operationName":"Viewer","authUser":"username","variables":{},"appPath":"N/A"},"message":"Graphql operation Viewer for user username with variables map[] from app path N/A","dd.trace_id":"11342671108643749252"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.605319717Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"Viewer","latencyNs":4064562,"statusCode":200,"operationName":"Viewer","authUser":"username","latencyStr":"4.064562ms"},"message":"Graphql operation Viewer for user username finished in 4.064562ms","dd.trace_id":"11342671108643749252"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.605859059Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"11342671108643749252","duration":"8.205247ms","statusCode":200,"path":"/graphql"},"message":"Finished request 11342671108643749252 in 8.205247ms with status %!s(int=200) on /graphql","dd.trace_id":"11342671108643749252"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.645228822Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"9594437594475943220","path":"/graphql"},"message":"Starting request 9594437594475943220 on /graphql","dd.trace_id":"9594437594475943220"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.647484472Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"ProbeServerSettings","operationName":"ProbeServerSettings","authUser":"username","variables":{},"appPath":"N/A"},"message":"Graphql operation ProbeServerSettings for user username with variables map[] from app path N/A","dd.trace_id":"9594437594475943220"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.647585685Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"ProbeServerSettings","latencyNs":97605,"statusCode":200,"operationName":"ProbeServerSettings","authUser":"username","latencyStr":"97.605µs"},"message":"Graphql operation ProbeServerSettings for user username finished in 97.605µs","dd.trace_id":"9594437594475943220"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.647794611Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"9594437594475943220","duration":"2.567926ms","statusCode":200,"path":"/graphql"},"message":"Finished request 9594437594475943220 in 2.567926ms with status %!s(int=200) on /graphql","dd.trace_id":"9594437594475943220"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.655412981Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"12929263262223363999","path":"/graphql"},"message":"Starting request 12929263262223363999 on /graphql","dd.trace_id":"12929263262223363999"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.657062252Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","operationName":"UpsertBucket","authUser":"username","variables":{"id":null,"config":"{\"learning_rate\": {\"desc\": null, \"value\": 0.02}, \"architecture\":... (+ 429 bytes)","jobType":null,"name":"a9s3ygtl","summaryMetrics":null,"project":"my-awesome-project","sweep":null,"program":"/home/stachu/Projects/airplane-damage-detection/src/runs/wandb/t... (+ 21 bytes)","state":null,"entity":null,"groupName":null,"host":"stachu-GV62-8RE","displayName":null,"tags":null,"commit":"49cc3c8a262a18c85e17aac73f10170e4bd0f957","notes":null,"debug":false,"repo":"https://github.com/Logistics-Living-Lab/Aircraft-damage-detection.git","description":null},"appPath":"N/A"},"message":"Graphql operation UpsertBucket for user username with variables map[commit:49cc3c8a262a18c85e17aac73f10170e4bd0f957 config:{\"learning_rate\": {\"desc\": null, \"value\": 0.02}, \"architecture\":... (+ 429 bytes) debug:false description:<nil> displayName:<nil> entity:<nil> groupName:<nil> host:stachu-GV62-8RE id:<nil> jobType:<nil> name:a9s3ygtl notes:<nil> program:/home/stachu/Projects/airplane-damage-detection/src/runs/wandb/t... (+ 21 bytes) project:my-awesome-project repo:https://github.com/Logistics-Living-Lab/Aircraft-damage-detection.git state:<nil> summaryMetrics:<nil> sweep:<nil> tags:<nil>] from app path N/A","dd.trace_id":"12929263262223363999"}
wandb-service_1  | {"level":"ERROR","time":"2023-11-20T11:52:58.661330734Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/pkg/observability/gerr/reporting.go:193","pid":69},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","projectID":1,"cacheKey":"project:1:approx_run_count"},"message":"key does not exist","dd.trace_id":"12929263262223363999","error":{"kind":"*errors.errorString","message":"key does not exist"}}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.679262988Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","latencyNs":22209981,"statusCode":200,"operationName":"UpsertBucket","authUser":"username","latencyStr":"22.209981ms"},"message":"Graphql operation UpsertBucket for user username finished in 22.209981ms","dd.trace_id":"12929263262223363999"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.679456569Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"12929263262223363999","duration":"24.047037ms","statusCode":200,"path":"/graphql"},"message":"Finished request 12929263262223363999 in 24.047037ms with status %!s(int=200) on /graphql","dd.trace_id":"12929263262223363999"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.976547517Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"10408580904170177267","path":"/graphql"},"message":"Starting request 10408580904170177267 on /graphql","dd.trace_id":"10408580904170177267"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.977866154Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"userID":4,"operationName":"RunUploadUrls","authUser":"username","operationName":"RunUploadUrls","authUser":"username","variables":{"name":"my-awesome-project","run":"a9s3ygtl","entity":"username","description":null,"files":["wandb-metadata.json"]},"appPath":"N/A"},"message":"Graphql operation RunUploadUrls for user username with variables map[description:<nil> entity:username files:[wandb-metadata.json] name:my-awesome-project run:a9s3ygtl] from app path N/A","dd.trace_id":"10408580904170177267"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.983810443Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"10856395839493353211","path":"/graphql"},"message":"Starting request 10856395839493353211 on /graphql","dd.trace_id":"10856395839493353211"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.983986291Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"11992039313010354236","path":"/graphql"},"message":"Starting request 11992039313010354236 on /graphql","dd.trace_id":"11992039313010354236"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.985060549Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","operationName":"RunUploadUrls","authUser":"username","variables":{"name":"my-awesome-project","run":"a9s3ygtl","entity":"username","description":null,"files":["code/src/runs/wandb/testing/log_metrics.py"]},"appPath":"N/A"},"message":"Graphql operation RunUploadUrls for user username with variables map[description:<nil> entity:username files:[code/src/runs/wandb/testing/log_metrics.py] name:my-awesome-project run:a9s3ygtl] from app path N/A","dd.trace_id":"10856395839493353211"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:58.985100036Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"RunStoppedStatus","operationName":"RunStoppedStatus","authUser":"username","variables":{"runId":"a9s3ygtl","projectName":"my-awesome-project","entityName":"username"},"appPath":"N/A"},"message":"Graphql operation RunStoppedStatus for user username with variables map[entityName:username projectName:my-awesome-project runId:a9s3ygtl] from app path N/A","dd.trace_id":"11992039313010354236"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:59.091221654Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"RunStoppedStatus","latencyNs":106055641,"statusCode":200,"operationName":"RunStoppedStatus","authUser":"username","latencyStr":"106.055641ms"},"message":"Graphql operation RunStoppedStatus for user username finished in 106.055641ms","dd.trace_id":"11992039313010354236"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:59.092417652Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"11992039313010354236","duration":"108.403894ms","statusCode":200,"path":"/graphql"},"message":"Finished request 11992039313010354236 in 108.403894ms with status %!s(int=200) on /graphql","dd.trace_id":"11992039313010354236"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:59.111589879Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","statusCode":200,"latencyNs":126481805,"operationName":"RunUploadUrls","authUser":"username","latencyStr":"126.481805ms"},"message":"Graphql operation RunUploadUrls for user username finished in 126.481805ms","dd.trace_id":"10856395839493353211"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:59.11162617Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"userID":4,"operationName":"RunUploadUrls","authUser":"username","statusCode":200,"latencyNs":133727550,"operationName":"RunUploadUrls","authUser":"username","latencyStr":"133.72755ms"},"message":"Graphql operation RunUploadUrls for user username finished in 133.72755ms","dd.trace_id":"10408580904170177267"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:59.112181541Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"10856395839493353211","duration":"128.35286ms","statusCode":200,"path":"/graphql"},"message":"Finished request 10856395839493353211 in 128.35286ms with status %!s(int=200) on /graphql","dd.trace_id":"10856395839493353211"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:52:59.112802778Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"10408580904170177267","duration":"136.226314ms","statusCode":200,"path":"/graphql"},"message":"Finished request 10408580904170177267 in 136.226314ms with status %!s(int=200) on /graphql","dd.trace_id":"10408580904170177267"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:01.10764669Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"9539402655740707023","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 9539402655740707023 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"9539402655740707023"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:01.135541442Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","username":"username","cliVersion":"0.13.11","entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","runSize":929,"keysInfoSize":686,"summarySize":142,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":true},"message":"file_stream: logged run of size 929, keysinfo size 686, summary size 142, config size 4, summaryUpdated true, keysInfoUpdated true","dd.trace_id":"9539402655740707023"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:01.15030656Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"9539402655740707023","duration":"42.640897ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 9539402655740707023 in 42.640897ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"9539402655740707023"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:08.994536959Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"10326115465093313418","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 10326115465093313418 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"10326115465093313418"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:09.002100582Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","username":"username","cliVersion":"0.13.11","runSize":923,"keysInfoSize":682,"summarySize":140,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":true},"message":"file_stream: logged run of size 923, keysinfo size 682, summary size 140, config size 4, summaryUpdated true, keysInfoUpdated true","dd.trace_id":"10326115465093313418"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:09.013081912Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"10326115465093313418","duration":"18.542687ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 10326115465093313418 in 18.542687ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"10326115465093313418"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:18.995279545Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"10711949161131642249","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 10711949161131642249 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"10711949161131642249"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:19.003712301Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","projectName":"my-awesome-project","runName":"a9s3ygtl","username":"username","cliVersion":"0.13.11","entityName":"username","runSize":906,"keysInfoSize":665,"summarySize":140,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":true},"message":"file_stream: logged run of size 906, keysinfo size 665, summary size 140, config size 4, summaryUpdated true, keysInfoUpdated true","dd.trace_id":"10711949161131642249"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:19.009840814Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"10711949161131642249","duration":"14.563299ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 10711949161131642249 in 14.563299ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"10711949161131642249"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:29.012382027Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"12831515860258384213","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 12831515860258384213 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"12831515860258384213"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:29.029067611Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","username":"username","cliVersion":"0.13.11","runSize":906,"keysInfoSize":665,"summarySize":140,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":true},"message":"file_stream: logged run of size 906, keysinfo size 665, summary size 140, config size 4, summaryUpdated true, keysInfoUpdated true","dd.trace_id":"12831515860258384213"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:29.039062596Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"12831515860258384213","duration":"26.671088ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 12831515860258384213 in 26.671088ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"12831515860258384213"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:31.033004984Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"9267594728739169553","path":"/graphql"},"message":"Starting request 9267594728739169553 on /graphql","dd.trace_id":"9267594728739169553"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:31.039393882Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","operationName":"UpsertBucket","authUser":"username","variables":{"displayName":null,"entity":"username","sweep":null,"state":null,"config":"{\"learning_rate\": {\"desc\": null, \"value\": 0.02}, \"architecture\":... (+ 440 bytes)","id":null,"summaryMetrics":null,"host":null,"debug":false,"jobType":null,"groupName":null,"commit":null,"name":"a9s3ygtl","program":null,"project":"my-awesome-project","notes":null,"description":null,"repo":null,"tags":null},"appPath":"N/A"},"message":"Graphql operation UpsertBucket for user username with variables map[commit:<nil> config:{\"learning_rate\": {\"desc\": null, \"value\": 0.02}, \"architecture\":... (+ 440 bytes) debug:false description:<nil> displayName:<nil> entity:username groupName:<nil> host:<nil> id:<nil> jobType:<nil> name:a9s3ygtl notes:<nil> program:<nil> project:my-awesome-project repo:<nil> state:<nil> summaryMetrics:<nil> sweep:<nil> tags:<nil>] from app path N/A","dd.trace_id":"9267594728739169553"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:31.063959088Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","latencyNs":24672589,"statusCode":200,"operationName":"UpsertBucket","authUser":"username","latencyStr":"24.672589ms"},"message":"Graphql operation UpsertBucket for user username finished in 24.672589ms","dd.trace_id":"9267594728739169553"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:31.064338383Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"9267594728739169553","duration":"31.353697ms","statusCode":200,"path":"/graphql"},"message":"Finished request 9267594728739169553 in 31.353697ms with status %!s(int=200) on /graphql","dd.trace_id":"9267594728739169553"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:39.019549625Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"10261506739865976965","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 10261506739865976965 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"10261506739865976965"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:39.035942561Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"runName":"a9s3ygtl","entityName":"username","projectName":"my-awesome-project","username":"username","cliVersion":"0.13.11","entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","runSize":908,"keysInfoSize":666,"summarySize":141,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":true},"message":"file_stream: logged run of size 908, keysinfo size 666, summary size 141, config size 4, summaryUpdated true, keysInfoUpdated true","dd.trace_id":"10261506739865976965"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:39.045478712Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"10261506739865976965","duration":"25.920295ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 10261506739865976965 in 25.920295ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"10261506739865976965"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:49.029999091Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"11610409074057276811","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 11610409074057276811 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"11610409074057276811"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:49.037918444Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"projectName":"my-awesome-project","runName":"a9s3ygtl","entityName":"username","runName":"a9s3ygtl","username":"username","cliVersion":"0.13.11","entityName":"username","projectName":"my-awesome-project","runSize":902,"keysInfoSize":663,"summarySize":138,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":true},"message":"file_stream: logged run of size 902, keysinfo size 663, summary size 138, config size 4, summaryUpdated true, keysInfoUpdated true","dd.trace_id":"11610409074057276811"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:49.043301423Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"11610409074057276811","duration":"13.297821ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 11610409074057276811 in 13.297821ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"11610409074057276811"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:59.031408034Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"10292773051600704817","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 10292773051600704817 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"10292773051600704817"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:59.0521463Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","username":"username","cliVersion":"0.13.11","runSize":1757,"keysInfoSize":648,"summarySize":140,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":true},"message":"file_stream: logged run of size 1757, keysinfo size 648, summary size 140, config size 4, summaryUpdated true, keysInfoUpdated true","dd.trace_id":"10292773051600704817"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:53:59.058105782Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"10292773051600704817","duration":"26.74811ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 10292773051600704817 in 26.74811ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"10292773051600704817"}
name"wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:09.042252077Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"13300996236940872815","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 13300996236940872815 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"13300996236940872815"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:09.056873215Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"runName":"a9s3ygtl","entityName":"username","projectName":"my-awesome-project","cliVersion":"0.13.11","entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","username":"username","runSize":1807,"keysInfoSize":646,"summarySize":139,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":true},"message":"file_stream: logged run of size 1807, keysinfo size 646, summary size 139, config size 4, summaryUpdated true, keysInfoUpdated true","dd.trace_id":"13300996236940872815"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:09.062332531Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"13300996236940872815","duration":"20.12935ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 13300996236940872815 in 20.12935ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"13300996236940872815"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.04215789Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"11519048823120233687","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 11519048823120233687 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"11519048823120233687"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.045071197Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","runName":"a9s3ygtl","username":"username","cliVersion":"0.13.11","entityName":"username","projectName":"my-awesome-project","runSize":1835,"keysInfoSize":0,"summarySize":167,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":false},"message":"file_stream: logged run of size 1835, keysinfo size 0, summary size 167, config size 4, summaryUpdated true, keysInfoUpdated false","dd.trace_id":"11519048823120233687"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.052761002Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"11519048823120233687","duration":"10.599525ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 11519048823120233687 in 10.599525ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"11519048823120233687"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.136994121Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"12923993797775448338","path":"/graphql"},"message":"Starting request 12923993797775448338 on /graphql","dd.trace_id":"12923993797775448338"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.138690526Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"operationName":"UpsertBucket","authUser":"username","userID":4,"operationName":"UpsertBucket","authUser":"username","variables":{"commit":null,"program":null,"id":null,"groupName":null,"host":null,"jobType":null,"state":null,"displayName":null,"project":"my-awesome-project","config":"{\"learning_rate\": {\"desc\": null, \"value\": 0.02}, \"architecture\":... (+ 443 bytes)","repo":null,"description":null,"debug":false,"sweep":null,"name":"a9s3ygtl","notes":null,"summaryMetrics":null,"entity":"username","tags":null},"appPath":"N/A"},"message":"Graphql operation UpsertBucket for user username with variables map[commit:<nil> config:{\"learning_rate\": {\"desc\": null, \"value\": 0.02}, \"architecture\":... (+ 443 bytes) debug:false description:<nil> displayName:<nil> entity:username groupName:<nil> host:<nil> id:<nil> jobType:<nil> name:a9s3ygtl notes:<nil> program:<nil> project:my-awesome-project repo:<nil> state:<nil> summaryMetrics:<nil> sweep:<nil> tags:<nil>] from app path N/A","dd.trace_id":"12923993797775448338"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.148118685Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"operationName":"UpsertBucket","authUser":"username","userID":4,"latencyNs":9439949,"statusCode":200,"operationName":"UpsertBucket","authUser":"username","latencyStr":"9.439949ms"},"message":"Graphql operation UpsertBucket for user username finished in 9.439949ms","dd.trace_id":"12923993797775448338"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.148330899Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"12923993797775448338","duration":"11.341548ms","statusCode":200,"path":"/graphql"},"message":"Finished request 12923993797775448338 in 11.341548ms with status %!s(int=200) on /graphql","dd.trace_id":"12923993797775448338"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.162610222Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"11980211858038061669","path":"/graphql"},"message":"Starting request 11980211858038061669 on /graphql","dd.trace_id":"11980211858038061669"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.164586442Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"userID":4,"operationName":"ProbeServerCapabilities","authUser":"username","operationName":"ProbeServerCapabilities","authUser":"username","variables":{},"appPath":"N/A"},"message":"Graphql operation ProbeServerCapabilities for user username with variables map[] from app path N/A","dd.trace_id":"11980211858038061669"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.164801601Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"userID":4,"operationName":"ProbeServerCapabilities","authUser":"username","statusCode":200,"latencyNs":217034,"operationName":"ProbeServerCapabilities","authUser":"username","latencyStr":"217.034µs"},"message":"Graphql operation ProbeServerCapabilities for user username finished in 217.034µs","dd.trace_id":"11980211858038061669"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.164976822Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"11980211858038061669","duration":"2.371328ms","statusCode":200,"path":"/graphql"},"message":"Finished request 11980211858038061669 in 2.371328ms with status %!s(int=200) on /graphql","dd.trace_id":"11980211858038061669"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.171625312Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"10008502353855671962","path":"/graphql"},"message":"Starting request 10008502353855671962 on /graphql","dd.trace_id":"10008502353855671962"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.173203331Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"Viewer","operationName":"Viewer","authUser":"username","variables":{},"appPath":"N/A"},"message":"Graphql operation Viewer for user username with variables map[] from app path N/A","dd.trace_id":"10008502353855671962"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.174629531Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"Viewer","latencyNs":1405203,"statusCode":200,"operationName":"Viewer","authUser":"username","latencyStr":"1.405203ms"},"message":"Graphql operation Viewer for user username finished in 1.405203ms","dd.trace_id":"10008502353855671962"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.175142048Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"10008502353855671962","duration":"3.506683ms","statusCode":200,"path":"/graphql"},"message":"Finished request 10008502353855671962 in 3.506683ms with status %!s(int=200) on /graphql","dd.trace_id":"10008502353855671962"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.180905887Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"11508002974860677856","path":"/graphql"},"message":"Starting request 11508002974860677856 on /graphql","dd.trace_id":"11508002974860677856"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.182553998Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"CreateArtifact","operationName":"CreateArtifact","authUser":"username","variables":{"artifactCollectionNames":["job-https___github.com_Logistics-Living-Lab_Aircraft-damage-dete... (+ 47 bytes)"],"description":"","metadata":null,"artifactTypeName":"job","runName":null,"enableDigestDeduplication":true,"digest":"1cb7ae505eb0f90a60e3e92e7cc3b441","projectName":"my-awesome-project","entityName":"username","sequenceClientID":"i6wq4z918tmv3ckpmbh668c3ju3430ec9tg9o0z23aqk8s17lw43xkv0u3veko2r... (+ 64 bytes)","labels":null,"distributedID":"","historyStep":null,"clientID":"8z3eypadi2k2wqez6fjokc7f9anwtesm0sdjq9jusedqritbmsfyg79j3ax0v53b... (+ 64 bytes)","aliases":[{"artifactCollectionName":"job-https___github.com_Logistics-Living-Lab_Aircraft-damage-dete... (+ 47 bytes)","alias":"latest"}]},"appPath":"N/A"},"message":"Graphql operation CreateArtifact for user username with variables map[aliases:[map[alias:latest artifactCollectionName:job-https___github.com_Logistics-Living-Lab_Aircraft-damage-dete... (+ 47 bytes)]] artifactCollectionNames:[job-https___github.com_Logistics-Living-Lab_Aircraft-damage-dete... (+ 47 bytes)] artifactTypeName:job clientID:8z3eypadi2k2wqez6fjokc7f9anwtesm0sdjq9jusedqritbmsfyg79j3ax0v53b... (+ 64 bytes) description: digest:1cb7ae505eb0f90a60e3e92e7cc3b441 distributedID: enableDigestDeduplication:true entityName:username historyStep:<nil> labels:<nil> metadata:<nil> projectName:my-awesome-project runName:<nil> sequenceClientID:i6wq4z918tmv3ckpmbh668c3ju3430ec9tg9o0z23aqk8s17lw43xkv0u3veko2r... (+ 64 bytes)] from app path N/A","dd.trace_id":"11508002974860677856"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.19956057Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"CreateArtifact","latencyNs":17014385,"statusCode":200,"operationName":"CreateArtifact","authUser":"username","latencyStr":"17.014385ms"},"message":"Graphql operation CreateArtifact for user username finished in 17.014385ms","dd.trace_id":"11508002974860677856"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.19973915Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"11508002974860677856","duration":"18.83393ms","statusCode":200,"path":"/graphql"},"message":"Finished request 11508002974860677856 in 18.83393ms with status %!s(int=200) on /graphql","dd.trace_id":"11508002974860677856"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.205582426Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"10968079202994903990","path":"/graphql"},"message":"Starting request 10968079202994903990 on /graphql","dd.trace_id":"10968079202994903990"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.207274673Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"CreateArtifactManifest","operationName":"CreateArtifactManifest","authUser":"username","variables":{"type":"FULL","entityName":"username","includeUpload":false,"name":"wandb_manifest.json","baseArtifactID":null,"runName":"a9s3ygtl","artifactID":"QXJ0aWZhY3Q6Mg==","digest":"","projectName":"my-awesome-project"},"appPath":"N/A"},"message":"Graphql operation CreateArtifactManifest for user username with variables map[artifactID:QXJ0aWZhY3Q6Mg== baseArtifactID:<nil> digest: entityName:username includeUpload:false name:wandb_manifest.json projectName:my-awesome-project runName:a9s3ygtl type:FULL] from app path N/A","dd.trace_id":"10968079202994903990"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.2196769Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"CreateArtifactManifest","latencyNs":12421227,"statusCode":200,"operationName":"CreateArtifactManifest","authUser":"username","latencyStr":"12.421227ms"},"message":"Graphql operation CreateArtifactManifest for user username finished in 12.421227ms","dd.trace_id":"10968079202994903990"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.219874858Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"10968079202994903990","duration":"14.295938ms","statusCode":200,"path":"/graphql"},"message":"Finished request 10968079202994903990 in 14.295938ms with status %!s(int=200) on /graphql","dd.trace_id":"10968079202994903990"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.234245527Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"9409353313781419240","path":"/graphql"},"message":"Starting request 9409353313781419240 on /graphql","dd.trace_id":"9409353313781419240"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.235575177Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"CreateArtifactFiles","operationName":"CreateArtifactFiles","authUser":"username","variables":{"storageLayout":"V2","artifactFiles":[{"artifactManifestID":"QXJ0aWZhY3RNYW5pZmVzdDoy","name":"requirements.frozen.txt","md5":"U59tSJsiNFgilH3dpvAL1w==","artifactID":"QXJ0aWZhY3Q6Mg=="},{"md5":"jcBl6t/lBSlQuVpAg08PJg==","artifactID":"QXJ0aWZhY3Q6Mg==","artifactManifestID":"QXJ0aWZhY3RNYW5pZmVzdDoy","name":"wandb-job.json"}]},"appPath":"N/A"},"message":"Graphql operation CreateArtifactFiles for user username with variables map[artifactFiles:[map[artifactID:QXJ0aWZhY3Q6Mg== artifactManifestID:QXJ0aWZhY3RNYW5pZmVzdDoy md5:U59tSJsiNFgilH3dpvAL1w== name:requirements.frozen.txt] map[artifactID:QXJ0aWZhY3Q6Mg== artifactManifestID:QXJ0aWZhY3RNYW5pZmVzdDoy md5:jcBl6t/lBSlQuVpAg08PJg== name:wandb-job.json]] storageLayout:V2] from app path N/A","dd.trace_id":"9409353313781419240"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.239153322Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"CreateArtifactFiles","latencyNs":3583935,"statusCode":200,"operationName":"CreateArtifactFiles","authUser":"username","latencyStr":"3.583935ms"},"message":"Graphql operation CreateArtifactFiles for user username finished in 3.583935ms","dd.trace_id":"9409353313781419240"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.239365717Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"9409353313781419240","duration":"5.121876ms","statusCode":200,"path":"/graphql"},"message":"Finished request 9409353313781419240 in 5.121876ms with status %!s(int=200) on /graphql","dd.trace_id":"9409353313781419240"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.800947576Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"12898864692170431132","path":"/graphql"},"message":"Starting request 12898864692170431132 on /graphql","dd.trace_id":"12898864692170431132"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.805896439Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","operationName":"RunUploadUrls","authUser":"username","variables":{"name":"my-awesome-project","run":"a9s3ygtl","entity":"username","description":null,"files":["requirements.txt"]},"appPath":"N/A"},"message":"Graphql operation RunUploadUrls for user username with variables map[description:<nil> entity:username files:[requirements.txt] name:my-awesome-project run:a9s3ygtl] from app path N/A","dd.trace_id":"12898864692170431132"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.806133405Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"10826006680377332402","path":"/graphql"},"message":"Starting request 10826006680377332402 on /graphql","dd.trace_id":"10826006680377332402"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.806290685Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"12749723696779111352","path":"/graphql"},"message":"Starting request 12749723696779111352 on /graphql","dd.trace_id":"12749723696779111352"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.811171304Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","operationName":"RunUploadUrls","authUser":"username","variables":{"entity":"username","description":null,"files":["wandb-summary.json"],"name":"my-awesome-project","run":"a9s3ygtl"},"appPath":"N/A"},"message":"Graphql operation RunUploadUrls for user username with variables map[description:<nil> entity:username files:[wandb-summary.json] name:my-awesome-project run:a9s3ygtl] from app path N/A","dd.trace_id":"10826006680377332402"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.811474182Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"operationName":"RunUploadUrls","authUser":"username","userID":4,"operationName":"RunUploadUrls","authUser":"username","variables":{"files":["config.yaml"],"name":"my-awesome-project","run":"a9s3ygtl","entity":"username","description":null},"appPath":"N/A"},"message":"Graphql operation RunUploadUrls for user username with variables map[description:<nil> entity:username files:[config.yaml] name:my-awesome-project run:a9s3ygtl] from app path N/A","dd.trace_id":"12749723696779111352"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.914815764Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"operationName":"RunUploadUrls","authUser":"username","userID":4,"latencyNs":103367957,"statusCode":200,"operationName":"RunUploadUrls","authUser":"username","latencyStr":"103.367957ms"},"message":"Graphql operation RunUploadUrls for user username finished in 103.367957ms","dd.trace_id":"12749723696779111352"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.915078115Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","statusCode":200,"latencyNs":103949178,"operationName":"RunUploadUrls","authUser":"username","latencyStr":"103.949178ms"},"message":"Graphql operation RunUploadUrls for user username finished in 103.949178ms","dd.trace_id":"10826006680377332402"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.915727809Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"10826006680377332402","duration":"109.597984ms","statusCode":200,"path":"/graphql"},"message":"Finished request 10826006680377332402 in 109.597984ms with status %!s(int=200) on /graphql","dd.trace_id":"10826006680377332402"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.916134711Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"12749723696779111352","duration":"109.83764ms","statusCode":200,"path":"/graphql"},"message":"Finished request 12749723696779111352 in 109.83764ms with status %!s(int=200) on /graphql","dd.trace_id":"12749723696779111352"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.916194443Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"RunUploadUrls","latencyNs":110295377,"statusCode":200,"operationName":"RunUploadUrls","authUser":"username","latencyStr":"110.295377ms"},"message":"Graphql operation RunUploadUrls for user username finished in 110.295377ms","dd.trace_id":"12898864692170431132"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:19.917126214Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"12898864692170431132","duration":"116.162183ms","statusCode":200,"path":"/graphql"},"message":"Finished request 12898864692170431132 in 116.162183ms with status %!s(int=200) on /graphql","dd.trace_id":"12898864692170431132"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.194473924Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"12023711550073183706","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 12023711550073183706 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"12023711550073183706"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.202445629Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"projectName":"my-awesome-project","runName":"a9s3ygtl","entityName":"username","runName":"a9s3ygtl","username":"username","cliVersion":"0.13.11","entityName":"username","projectName":"my-awesome-project","runSize":1787,"keysInfoSize":0,"summarySize":167,"configSize":4,"summaryUpdated":true,"keysInfoUpdated":false},"message":"file_stream: logged run of size 1787, keysinfo size 0, summary size 167, config size 4, summaryUpdated true, keysInfoUpdated false","dd.trace_id":"12023711550073183706"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.210033681Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"12023711550073183706","duration":"15.556295ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 12023711550073183706 in 15.556295ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"12023711550073183706"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.212732392Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"13025814516433208298","path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Starting request 13025814516433208298 on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"13025814516433208298"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.215584096Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/runupdates/syncrunupdates/sync_run_updater.go:419","pid":69},"data":{"entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","entityName":"username","projectName":"my-awesome-project","runName":"a9s3ygtl","username":"username","cliVersion":"0.13.11","runSize":2423,"keysInfoSize":0,"summarySize":0,"configSize":579,"summaryUpdated":false,"keysInfoUpdated":false},"message":"file_stream: logged run of size 2423, keysinfo size 0, summary size 0, config size 579, summaryUpdated false, keysInfoUpdated false","dd.trace_id":"13025814516433208298"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.221785861Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"13025814516433208298","duration":"9.052329ms","statusCode":200,"path":"/files/username/my-awesome-project/a9s3ygtl/file_stream"},"message":"Finished request 13025814516433208298 in 9.052329ms with status %!s(int=200) on /files/username/my-awesome-project/a9s3ygtl/file_stream","dd.trace_id":"13025814516433208298"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.231991693Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:57","pid":69},"data":{"requestID":"13440020094646357195","path":"/graphql"},"message":"Starting request 13440020094646357195 on /graphql","dd.trace_id":"13440020094646357195"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.233455554Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:140","pid":69},"data":{"authUser":"username","userID":4,"operationName":"Viewer","operationName":"Viewer","authUser":"username","variables":{},"appPath":"N/A"},"message":"Graphql operation Viewer for user username with variables map[] from app path N/A","dd.trace_id":"13440020094646357195"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.23444807Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/relay.go:235","pid":69},"data":{"authUser":"username","userID":4,"operationName":"Viewer","latencyNs":992307,"statusCode":200,"operationName":"Viewer","authUser":"username","latencyStr":"992.307µs"},"message":"Graphql operation Viewer for user username finished in 992.307µs","dd.trace_id":"13440020094646357195"}
wandb-service_1  | {"level":"INFO","time":"2023-11-20T11:54:20.234744839Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/api/handler/logging.go:62","pid":69},"data":{"requestID":"13440020094646357195","duration":"2.75498ms","statusCode":200,"path":"/graphql"},"message":"Finished request 13440020094646357195 in 2.75498ms with status %!s(int=200) on /graphql","dd.trace_id":"13440020094646357195"}
anmolmann commented 7 months ago

Hey @Stashq , apologies for the delay here. Checking in to see if you were able to resolve the above network errors for your local deployment?

anmolmann commented 7 months ago

Hi @Stashq , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!