search

remotion-dev / remotion

🎥 Make videos programmatically with React
https://remotion.dev
Other
20.79k stars 1.05k forks source link

`<AnimatedImage>` component #4175

Open JonnyBurger opened 3 months ago

JonnyBurger commented 3 months ago

TIL about ImageDecoder: https://developer.mozilla.org/en-US/docs/Web/API/ImageDecoder

Sample: 

<!DOCTYPE html>
<head>
  <title>WebCodecs Animated GIF Renderer</title>
</head>
<canvas width="320" height="270"></canvas>
<br /><br />
<textarea style="width: 640px; height: 270px"></textarea>
<script>
  let imageDecoder = null;
  let imageIndex = 0;

  function log(str) {
    document.querySelector("textarea").value += str + "\n";
  }

  function renderImage(result) {
    const canvas = document.querySelector("canvas");
    const canvasContext = canvas.getContext("2d");

    canvasContext.drawImage(result.image, 0, 0);

    const track = imageDecoder.tracks.selectedTrack;

    // We check complete here since `frameCount` won't be stable until all data
    // has been received. This may cause us to receive a RangeError during the
    // decode() call below which needs to be handled.
    if (imageDecoder.complete) {
      if (track.frameCount == 1) return;

      if (imageIndex + 1 >= track.frameCount) imageIndex = 0;
    }

    // Decode the next frame ahead of display so it's ready in time.
    imageDecoder
      .decode({ frameIndex: ++imageIndex })
      .then((nextResult) =>
        setTimeout((_) => {
          renderImage(nextResult);
        }, result.image.duration / 1000.0)
      )
      .catch((e) => {
        // We can end up requesting an imageIndex past the end since we're using
        // a ReadableStrem from fetch(), when this happens just wrap around.
        if (e instanceof RangeError) {
          imageIndex = 0;
          imageDecoder.decode({ frameIndex: imageIndex }).then(renderImage);
        } else {
          throw e;
        }
      });
  }

  function logMetadata() {
    log("imageDecoder.type = " + imageDecoder.type);
    log("imageDecoder.tracks.length = " + imageDecoder.tracks.length);
    log("");

    function logTracks() {
      for (let i = 0; i < imageDecoder.tracks.length; ++i) {
        const track = imageDecoder.tracks[i];
        log(`track[${i}].frameCount = ` + track.frameCount);
        log(`track[${i}].repetitionCount = ` + track.repetitionCount);
        log(`track[${i}].animated = ` + track.animated);
        log(`track[${i}].selected = ` + track.selected);
      }
    }

    if (!imageDecoder.complete) {
      log("Partial metadata while still loading:");
      log("imageDecoder.complete = " + imageDecoder.complete);
      logTracks();
      log("");
    }

    imageDecoder.completed.then((_) => {
      log("Final metadata after all data received:");
      log("imageDecoder.complete = " + imageDecoder.complete);
      logTracks();
    });
  }

  function decodeImage(imageByteStream) {
    if (typeof ImageDecoder === "undefined") {
      log("Your browser does not support the WebCodecs ImageDecoder API.");
      return;
    }

    imageDecoder = new ImageDecoder({
      data: imageByteStream,
      type: "image/avif",
    });
    imageDecoder.tracks.ready.then(logMetadata);
    imageDecoder.decode({ frameIndex: imageIndex }).then(renderImage);
  }

  fetch("/2.avif").then((response) => decodeImage(response.body));
</script>

Crazy that it works with animated GIFs, AVIFs and animated WebPs!

We can re-implement the Gif component with it and also support other image formats!

Just-Moh-it commented 3 months ago

hey hey hey… good weekend project and learning experience if you wanna assign!

fitzmode commented 3 months ago

Hey @JonnyBurger would love to take a look at this. Thanks!

JonnyBurger commented 3 months ago

Hey, I give it to @Just-Moh-it because he wrote first. Happy to give you the next one @fitzmode!

💎 This issue has a bounty on it!

Read our contributing guidelines:

/bounty 500

Criteria and hints

There should be a new component added to the core package (remotion) called <AnimatedImage>.

It should work similarly to the @remotion/gif package - fetch the file and parse the frames and display the current frame based on useCurrentFrame().

The @remotion/gif package has a separate implementation for development and rendering - this is not necessary.

When fetching a URL, the component should take a look at the Content-Type header. We will support three headers:

All work the same, just the type in ImageDecoder needs to be adjusted.
Other content types should be rejected.

There should be a good error message if ImageDecoder is not available (e.g. Firefox)

There should be a documentation page that aligns with other docs in the remotion package.

algora-pbc[bot] commented 3 months ago

💎 $500 bounty • Remotion

Steps to solve:

  1. Get assigned: If you'd like to work on this issue, comment /attempt #4175 below to get assigned
  2. Submit work: Create a pull request including /claim #4175 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to remotion-dev/remotion!

Add a bounty • Share on socials

hunxjunedo commented 3 months ago

Hey, would love to have a look at this, I'm in the row.

akhilender-bongirwar commented 3 months ago

Hi, would like to work on this issue. I'm in the queue.

amochuko commented 3 months ago

Am also available to start working on this if chanced.

amochuko commented 3 months ago

/attempt #4175

algora-pbc[bot] commented 3 months ago

@amochuko: Another person is already attempting this issue. Please don't start working on this issue unless you were explicitly asked to do so.

Mubashirshariq commented 2 months ago

@JonnyBurger Can i get this one assigned ,i am really excited to get this component done

Just-Moh-it commented 2 months ago

update: finishing this up, closing in!

JonnyBurger commented 2 months ago

@Just-Moh-it can you open a Draft PR with the progress so far? maybe we can help finishing it

zyronite commented 2 months ago

Lemme give it a Trryyy!

algora-pbc[bot] commented 2 months ago

@j4nlksh: Another person is already attempting this issue. Please don't start working on this issue unless you were explicitly asked to do so.

mrkirthi-24 commented 1 month ago

@JonnyBurger lmk if this issue is up for grabs.

onyedikachi-david commented 1 month ago

@Just-Moh-it Are you still on this?

arshad-muhammad commented 1 month ago

@JonnyBurger since the issue still open I would like to contribute. Can you please assign this for me

aadarsh-nagrath commented 1 month ago

Whats status here ? Is this open for anyone to work ? @JonnyBurger

karelnagel commented 2 weeks ago

I needed this, so I did create it on my own. Works with gif, webp and avif, if browser not supported or wrong content-type then falls back to use <Img /> instead.

Feel free to use it in remotion, or if still needed I could make this meet all the requirements and write the docs to solve this issue.

import { CSSProperties, useEffect, useMemo, useRef, useState } from 'react'
import { Img, continueRender, delayRender, useCurrentFrame, useVideoConfig } from 'remotion'

type ImageDecoderInit = {
  type: string
  data: ReadableStream | ArrayBuffer | ArrayBufferView | Blob | string | null
}

type ImageDecodeResult = {
  image: VideoFrame
  complete: boolean
}

type ImageDecoderVideoTrack = {
  selected: boolean
  frameCount: number
  repetitionCount: number
  frameSize: {
    width: number
    height: number
  }
}

type ImageDecoderTracks = {
  selectedIndex: number
  selectedTrack: ImageDecoderVideoTrack
}

declare class ImageDecoder {
  constructor(init: ImageDecoderInit)
  readonly tracks: ImageDecoderTracks
  readonly completed: Promise<void>
  decode(options?: { frameIndex?: number }): Promise<ImageDecodeResult>
  reset(): void
  close(): void
}

const ANIMATED_IMAGE_CONTENT_TYPES = ['image/gif', 'image/webp', 'image/avif']

type AnimatedImageMetadata = {
  width: number
  height: number
  fps: number | null
  frameCount: number
}

const useAnimatedImage = (src: string) => {
  const [handle] = useState(delayRender)
  const [decoder, setDecoder] = useState<ImageDecoder>()
  const [metadata, setMetadata] = useState<AnimatedImageMetadata>()

  useEffect(() => {
    const effect = async () => {
      if (typeof ImageDecoder === 'undefined') return console.error('ImageDecoder not available in this browser!')

      const response = await fetch(src)
      const contentType = response.headers.get('Content-Type')

      if (!contentType || !ANIMATED_IMAGE_CONTENT_TYPES.includes(contentType))
        return console.error(`Content type '${contentType}' is not supported!`)

      const decoder = new ImageDecoder({ data: response.body, type: contentType })
      await decoder.completed
      const frameCount = decoder.tracks.selectedTrack.frameCount

      // Decoding the first frame to get metadata
      const { image } = await decoder.decode({ frameIndex: 0 })
      const height = image.displayHeight
      const width = image.displayWidth
      // image.duration can be null if it's non-animated image
      const fps = image.duration ? 1000000 / image.duration : null

      setDecoder(decoder)
      setMetadata({ frameCount, height, width, fps })
    }
    effect().then(() => continueRender(handle))
  }, [src])

  return { decoder, metadata }
}

type AnimatedImageProps = {
  src: string
  style?: CSSProperties
  loopBehavior?: 'loop' | 'pause-after-finish'
  playbackRate?: number
}

export const AnimatedImage = ({ src, style, loopBehavior = 'loop', playbackRate = 1 }: AnimatedImageProps) => {
  const canvasRef = useRef<HTMLCanvasElement>(null)

  const { fps } = useVideoConfig()
  const frame = useCurrentFrame()

  const { decoder, metadata } = useAnimatedImage(src)

  const frameIndex = useMemo(() => {
    if (!metadata || !metadata.fps) return 0
    const currentFrame =
      loopBehavior === 'loop'
        ? (frame * playbackRate) % metadata.frameCount
        : Math.min(frame * playbackRate, metadata.frameCount)

    return Math.floor((currentFrame * metadata.fps) / fps)
  }, [frame, playbackRate, metadata, loopBehavior])

  useEffect(() => {
    const renderFrame = async () => {
      const canvas = canvasRef.current
      if (!decoder || !canvas) return

      const ctx = canvas.getContext('2d')
      if (!ctx) return

      const { image } = await decoder.decode({ frameIndex })
      ctx.drawImage(image, 0, 0)
    }

    const handle = delayRender()
    renderFrame().then(() => continueRender(handle))
  }, [decoder, frameIndex])

  if (!metadata) return <Img src={src} style={style} />
  return <canvas ref={canvasRef} width={metadata.width} height={metadata.height} style={style} />
}