GCS storage manifest : Error: "Invalid 'manifest.jsonl file structure': 'name' is required, but not found".

[matt-pathr]

My actions before raising this issue

• [x] Read/searched the docs
• [x] installed cvat
• [x] enabled https
• [x] created / provisioned google cloud storage
• [x] created manifest file from video using my laptop (Windows 10 with WSL Ubuntu 20.04) using prerequisites listed here
• [x] copied manifest file and video file to gs://
• [x] added Google cloud storage to cloud storage -- when I clicked submit, it found the manifest.jsonl and accepts the cloud storage setting, but returned Error: "Invalid 'manifest.jsonl file structure': 'name' is required, but not found"
  -- when browsing the cloud storage page, The storage entry is listed as available and has a Could not fetch preview for cloud storage #1 Error: . "Invalid 'manifest.jsonl file structure': 'name' is required, but not found". error notification on the side
• [x] Searched past issues

First 10 lines of 'manifest.jsonl' file created:

{"version":"1.1"}
{"type":"video"}
{"properties":{"name":"c051.1667156401557.mp4","resolution":[1280,720],"length":108000}}
{"number":0,"pts":0,"checksum":"452dc57b6f792023aece22acecf2b03f"}
{"number":250,"pts":128000,"checksum":"add9fad5ee878f6679537e39d3cbedcd"}
{"number":500,"pts":256000,"checksum":"bf137bcb7e165a1ac7453037ac575b5e"}
{"number":750,"pts":384000,"checksum":"097950e0bb71fd8b6b7b8432d921a238"}
{"number":1000,"pts":512000,"checksum":"34173ccd8456265601c88eee665986d0"}
{"number":1250,"pts":640000,"checksum":"8f340d6b8a007c10c570c6bf562e3c7b"}
{"number":1500,"pts":768000,"checksum":"601279ec9f48e207fe11dd22f56bd9b4"}

last 10 lines of 'manifest.jsonl' file created:

{"number":105500,"pts":54016000,"checksum":"99f055166bb6fea06975e41064cf18fb"}
{"number":105750,"pts":54144000,"checksum":"fe31097afcbdf4ef6e982de4ad1d2eb0"}
{"number":106000,"pts":54272000,"checksum":"58aef41ff357b196a71204ea91d5256d"}
{"number":106250,"pts":54400000,"checksum":"deb824aadc2609d8767bfbe0863d7feb"}
{"number":106500,"pts":54528000,"checksum":"ec5f67812f9de1499f5c6415cd726f5e"}
{"number":106750,"pts":54656000,"checksum":"d5da81da43af3fa4d7c51b393c7145f9"}
{"number":107000,"pts":54784000,"checksum":"72b79309d99e6ed450ac84ee3a525c90"}
{"number":107250,"pts":54912000,"checksum":"5e1f3d6c125c264dc9c03ed95109ca6a"}
{"number":107500,"pts":55040000,"checksum":"7c28d715072ef5f04308e5f68bb4d427"}
{"number":107750,"pts":55168000,"checksum":"9b62713c2e6c52abd26094957306c800"}

Expected Behaviour

Expecting to see the dataset and video file listed as usable objects for task/projects.

Current Behaviour

Unable to specify dataset in tasks, the error shows when trying to access cloud storage listing from the create task windows.

Steps to Reproduce (for bugs)

see "My actions before raising this issue" section

Context

Looking to create secure server to submit jobs for annotation team to complete via backend GSC storage where customer datasets are maintained.

Your Environment

• Git hash commit (git log -1):
• Docker version docker version :

  Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:00:04 2022
  OS/Arch:          linux/amd64
  Experimental:     false
  containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
  runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
  docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• Are you using Docker Swarm or Kubernetes? NO, just docker-compose
• Operating System and version (e.g. Linux, Windows, MacOS):
• • Ubuntu 20.04.5 LTS VM with dedicated vdisk for /var/lib/docker
• Other diagnostic information/logs: Looking at docker-compose logs while showing viewing /cloudstorages?page=1 found warning snd error in cvat_server (below):

  cvat_server             | 2022-11-15 16:27:39,570 DEBG 'runserver' stderr output:
  cvat_server             | [Tue Nov 15 16:27:39.570091 2022] [wsgi:error] [pid 50:tid 140139994048256] [remote 172.19.0.3:41854] WARNING:django.request:Bad Request: /api/cloudstorages/1/preview
  cvat_server             |
  cvat_server             | 2022-11-15 16:52:54,234 DEBG 'runserver' stderr output:
  cvat_server             | [Tue Nov 15 16:52:54.234009 2022] [wsgi:error] [pid 50:tid 140139977262848] [remote 172.19.0.3:58430] [2022-11-15 16:52:54,233] WARNING django.request: Not Found: /api/user-agreements
  cvat_server             |
  cvat_server             | 2022-11-15 16:52:54,234 DEBG 'runserver' stderr output:
  cvat_server             | [Tue Nov 15 16:52:54.234560 2022] [wsgi:error] [pid 50:tid 140139977262848] [remote 172.19.0.3:58430] WARNING:django.request:Not Found: /api/user-agreements
  cvat_server             | 2022-11-15 16:53:03,678 DEBG 'runserver' stderr output:
  cvat_server             | [Tue Nov 15 16:53:03.678581 2022] [wsgi:error] [pid 50:tid 140139994048256] [remote 172.19.0.3:39886] [2022-11-15 16:53:03,678] WARNING django.request: Bad Request: /api/cloudstorages/1/preview
  cvat_server             |
  cvat_server             | 2022-11-15 16:53:03,679 DEBG 'runserver' stderr output:
  cvat_server             | [Tue Nov 15 16:53:03.678689 2022] [wsgi:error] [pid 50:tid 140139994048256] [remote 172.19.0.3:39886] WARNING:django.request:Bad Request: /api/cloudstorages/1/preview

Thank you

[bmetge]

The exemple manifest file for a video https://opencv.github.io/cvat/docs/manual/advanced/dataset_manifest/ seem to also raising this issue

[nmanovic]

@bmetge , @matt-pathr , thanks for the report. Video data isn't supported for now on cloud storage. Need to implement that.

[kir-ark]

I have the same problem. When will you be able to solve the issue?