bkaradzic / bgfx

Cross-platform, graphics API agnostic, "Bring Your Own Engine/Framework" style rendering library.
https://bkaradzic.github.io/bgfx/overview.html
BSD 2-Clause "Simplified" License
15.15k stars 1.95k forks source link

Capturing Video (on mobile platforms) #1029

Open pplux opened 7 years ago

pplux commented 7 years ago

There are several problems with the current capture implementation for video recording:

  1. The lack of buffering, glReadPixels will block anyone that reads the memory, that probably should be done on a different thread (not the call itself, but the actual read of the memory). I'm aware this is in the pipeline of bgfx development.

  2. Not capturing in the best format available forcing the driver to do format conversion, for example, on Android the best format to capture is RGB565, although, RGBA is supported by the specification that doesn't imply is the best format to capture. The best format can be queried with

      glGetIntegerv(GL_IMPLEMENTATION_COLOR_READ_FORMAT_OES, &p_crf);
      glGetIntegerv(GL_IMPLEMENTATION_COLOR_READ_TYPE_OES, &p_crt);

    On every android I have tested, GL_UNSIGNED_SHORT_5_6_5 and GL_RGB was the result, showing a huge performance boost.

  3. Not allow for platform specific, or user specific methods to bypass default capture method. On android, it is possible to use hardware accelerated video encoder with a given surface to record video. I've tested this with bgfx, and works flawlessly, but the lack of multiple window support for android doesn't allow to render and capture at the same time (We were able to capture bgfx, but not showing render result at the same time)

I'm inclined to solve issue 3, and with that give the user the opportunity to work at his/here own risk. For example the capture interface could be something like:

 class Capture {
 protected:

  virtual void captureBegin() {}
  virtual void captureEnd() {}
  virtual void captureFrame(const void *data, uint32_t size) {}

  void performCapture(void *raw_mem, TextureFormat::Enum f); // not virtual, implemented inside bgfx
 // example of default but overrideable implementation 
  virtual void capture(uint16_t width, uint16_t height) {
    void *mem = BX_ALLOC(width*height*4);
     performCapture(mem, TextureFormat::RGBA);
    captureFrame(mem, width*height*4);
    BX_FREE(mem);
  }
};

with an overridable capture method the user have all the control, it can solve problems (1), (2), and (3) and preserves the default implementation.

bkaradzic commented 7 years ago
  1. Will be added.

  2. TextureFormat::Enum will be added to capture callback.

  3. The problem with virtual void capture(uint16_t width, uint16_t height) { and void performCapture(void *raw_mem, TextureFormat::Enum f); is that they don't give you any internals from renderer. This might seem like it would work with GL, but it would not since all gl* API calls are inside bgfx::gl namespace. This definitely wouldn't work with D3Dx, VK, Metal where you have device concept.