Basic notebook fails in t.select_frames

[gaurav274]

t.select_frames fails for both tiled and untiled video

python3: /tasm/homomorphic_stitching/include/PictureParameterSet.h:40: stitching::PictureParameterSetMetadata::PictureParameterSetMetadata(stitching::BitStream): Assertion `metadata_.GetValue("end") % 8 == 0' failed.

[maureendaum]

Thanks for reporting, I'll try to debug this tomorrow.

[maureendaum]

I didn't get any errors when I ran through the notebook on a fresh git clone. Were you using the default video in the repository?

If you'd be able to send me the ingested videos, I can try running select_frames on those encoded videos to see if I can reproduce the issue. I think the easiest way to get the ingested videos will be to mount a directory when you start the docker container, and then specify that path for catalog_path in the first cell of the notebook.

[gaurav274]

I'm using the default video in the repository. The notebook works fine when I'm selecting the tile. Can you think of any possible reason for the issue? Or any idea for me to debug the issue?

[gaurav274]

I'm also getting the same error when trying it on untiled_video.

frame_selection = t.select_frames(untiled_video_name, metadata_id, label, first_frame_inclusive, last_frame_exclusive) Edit: I'm seeing the error message in the docker console and the notebook kernel shuts down. Thanks!

[maureendaum]

That part of the code with the assertion is walking through the encoded video file, so it's possible that something went wrong when TASM ingested and tiled the video. If you send me TASM's catalog after you run the first part of the notebook, I can see if anything looks off with the video files.

Since it's happening with the untiled video, it's probably easiest to start debugging with that one. One quick check is if the video's metadata looks ok. After running the first cell in the section "Loading the video and detections into TASM.", can you paste the output of ffprobe -hide_banner -show_streams -show_format basics_resources/resources/birds-untiled/0-360-0/orig-tile-0.mp4 | grep -E 'width|height|nb_frames|r_frame_rate' (run from inside the docker container at /tasm/python/Examples). As a reference, this is what I am getting:

root@cf25be75dde1:/tasm
/python/Examples# ffprobe -hide_banner -show_streams -show_format basics_resources/resources/birds-untiled/0-360-0/orig-tile-0.mp4 | grep -E 'width|height|nb_frames|r_frame_rate'
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'basics_resources/resources/birds-untiled/0-360-0/orig-tile-0.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isom
    creation_time   : 2022-03-17T03:08:34.000000Z
  Duration: 00:00:12.03, start: 0.000000, bitrate: 3133 kb/s
    Stream #0:0(und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv), 1080x1920 [SAR 1:1 DAR 9:16], 3131 kb/s, 30 fps, 30 tbr, 30 tbn, 30 tbc (default)
    Metadata:
      creation_time   : 2022-03-17T03:08:34.000000Z
      handler_name    : GPAC ISO Video Handler
width=1080
height=1920
coded_width=1088
coded_height=1920
r_frame_rate=30/1
nb_frames=361

[gaurav274]

Thanks for looking into the issue. I ran the above command and get the same result. I'm attaching the catalog content. https://drive.google.com/file/d/12vEG1hOhnzZ4OWoHPa98qgHqt9CxFm9H/view?usp=sharing Thanks in advance!

[maureendaum]

Thanks for sharing the videos! I was able to reproduce it with them; somehow the video headers contain flags that we don't expect when parsing them. The assertion you ran into seems to be an actual bug, but then it still fails later on when we get an unexpected value in the header. I need to see if it's possible to configure the encoder to not use those options, or if we have to modify the stitching code to handle them.

[gaurav274]

Sounds good! Btw, you mentioned it is working on your setup. Which package is at culprit here?

[maureendaum]

We're using Nvidia's nvenc to encode the videos, and for some reason it seems to be using different settings on your machine vs. mine. I'm not sure if it could be due to the graphics card or driver version or something else.

[maureendaum]

Sorry for the slow progress; I've been refamiliarizing myself with the HEVC encoding options to see if there's a way to force the videos to be encoded with the flags we expect. I'll push some possible fixes to different branches in the next day or two, and it would be great to have you try encoding videos with them to see if they make any difference. I'll let you know once I have those ready!

[gaurav274]

Sounds good! Thanks for putting in the effort.

[maureendaum]

I just pushed to the tasm-dpb branch. I'm hoping the new encoding parameter will force the encoder to use the same reference pictures as my machine. You shouldn't have to re-build the tasm/environment docker container, just the tasm/tasm one. Can you just ingest the video into tasm then send me the untiled, encoded video I can see if anything changed? Thanks!