Run on every frame in Video

[GantMan]

Much like the discussion over in #38 There's been a request to run on every frame in a video.

It's possible to do in JS like so: https://jsfiddle.net/bmartel/3h98gsvk/11/

Maybe a syntax like model.classifyVideo(video) or model.classifyAny(img|gif|video)

[fvonhoven]

I'll take this one!

[stared]

I would be interested in one.

THough, for video I am curious about performance. (Did you benchmark frames per second? In my case, on a 8-year-old Macbook Pro, it takes ~10s for image classification.)

[GantMan]

I imagine this would take a while client-side. However, I'd like to start at (just do it) and then work on doing it fast. There could be a "random scan" which checks every 10 or 100 frames. Just TOL.

[bongobongo]

Depending on the hardware on the client side, the speed could be increased drastically by creating several instances of the nsfwjs - using Web Workers. Then one could classify multiple frames in parallell. The number of workers that could work efficiently is probably defined by frame size (image size), CPU and memory. I have tested Web Workers against the current version of NSFWJS, and it works in the latest versions of Chrome and Opera. I did it because my web clients need to interact with the page UI immediately, and not wait 5 - 20 seconds (or more, e.g. mobile) for the model to load. Runing NSFWJS in a Web Worker - made it so that both the loading and classifying of images was done in a process separated from what happens in javascript in the main page. More about it here: https://github.com/infinitered/nsfwjs/issues/94#issuecomment-487306805

[GantMan]

I like this idea.

[qgustavor]

How about using keyframe or scene detection to better select what frames to analyze?

[GantMan]

I really like that. Is that doable in JS?

[yzevm]

I'll try to implement it

[uzaysan]

Hi @YegorZaremba Any update on this?

[yzevm]

@uzaysan Yeap, we can use same interface for classifyVideo method as we use for classifyGif https://github.com/infinitered/nsfwjs/pull/401/files#diff-f41e9d04a45c83f3b6f6e630f10117feL194

Add guard that checks is video fully buffered, if not - throw an error. Also for nodejs usage, we should accept Buffers you can see how it works for gif https://github.com/infinitered/nsfwjs/pull/401/files#diff-f41e9d04a45c83f3b6f6e630f10117feR206

  async classifyVideo(
    video: HTMLVideoElement,
    config: classifyConfig = { topk: 5, fps: 25 }
  ): Promise<Array<Array<predictionClassType>>> {
    const totalFrames: number = Math.floor(video.duration * config.fps)
    const interval: number = video.duration / totalFrames

    // @NOTE Messy, but after refactoring please test with 25+ fps
    const acceptedFrames: number[] = []
    for (let i = 0; i < totalFrames; i++) {
      const frame = Math.floor((i * interval) * 100) / 100
      acceptedFrames.push(frame)
    }

    const canvas = createCanvas(video.offsetWidth, video.offsetHeight)
    const context = canvas.getContext('2d')
    const arrayOfClasses: predictionClassType[][]  = []
    for (let i = 0; i < acceptedFrames.length; i++) {
      video.currentTime = acceptedFrames[i]
      context.drawImage(video, 0, 0, canvas.width, canvas.height);

      const image = await loadImage(canvas.toDataURL()) // new Image() onload onerror
      // @ts-ignore
      const classes = await this.classify(image, config.topk);
      arrayOfClasses.push(classes)
    }

    return arrayOfClasses
  }

Problems

const image = await loadImage(canvas.toDataURL())

For most browser using this line of code will throw an error like https://stackoverflow.com/questions/35244215/html5-video-screenshot-via-canvas-using-cors/35245146 I have no idea how to fix it, we can implement this logic as is, and if we have that error, just throw new NSFWError(error), but as led dev of @nsfw-filter is not solved my problems at all

UPD1, I'll return to this issue when I have free time and announce to everyone on this issue

[uzaysan]

@YegorZaremba Thanks for quick reply. I'm using this library in server side(NodeJs). But when I pass video buffer, It throws an unsuported Image error.

Expected image (BMP, JPEG, PNG, or GIF), but got unsupported image type

I'm using it like this:

const model = await nsfwlib.load();
  const video = await tf.node.decodeImage(videoBuffer);
  const predictions = await model.classify(video);

Also I'm using version 2.2.0

What am I doing wrong? Can you please help me?

[yzevm]

But when I pass video buffer, It throws an unsupported Image error.

@uzaysan This feature is not implemented yet, you can slice your video to images by yourself and predict these image, just google some lib mp4 to img node.js, I hope sth exists in the npm

[uzaysan]

So I should classify video frames one by one. Am I right? But Also I can use library bu converting video to gif? Can you confirm?

Also when will video classification be available? Do you have an estimate date? I saw in some issues, you associated this feature to v2.3.0. When will that version come out.

Thanks.

[yzevm]

yeap, great idea So if you want use classifyGif pls install @nsfw-filter/nsfwjs because this feature(classifyGif) has the status "Work in progress" (just some fixes for nsfw.com) but we use this package in @nsfw-filter. It works pretty slow for Buffers in Nodejs (but it works :D)

[uzaysan]

Thank you I will try

[MINORITYmaN]

Hi there! Offscreen canvas could help get it more perfomant with web workers, but https://caniuse.com/offscreencanvas :(

https://developers.google.com/web/updates/2018/08/offscreen-canvas

[RanaOsamaAsif]

Hi all,

It's been almost 2.5 years since the last comment 😅

Are there any updates on this being implemented?

[GantMan]

Currently, we are looking for someone to champion this task/example.

I've been playing with the idea of putting time into updating the library to separate out GIF/VID into separate libraries that depend on this one. But without a break in time, or a client looking to sponsor the project, it's hard to find the time.

@cdanwards - if we ever have non-billable time for you, this might be a good task for the community.