How to use "add_frame_arrived" ?

[tfluan0606]

I'm trying to implement window capture on specific window and send it to opencv. I get a frame by keep calling try_get_next_frame(), send it to opencv and successfully show it. But it can only got 1 frame, what i want to do is like obs window capture, keep listen window ,so that opencv can do some object detection. I have read "screen-capture" and "screen-capture-video" by Microsoft build, I notice that both of these use FrameArrived to handle framepool. I think the equivalent one is add_frame_arrived, but how do I use it ? https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture#acquire-capture-frames https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture-video#start-capturing

The reson I want to use WGC is that the window I want to capture is hardware acceleration rendering.

Thank you for reading.

[dlech]

You can currently find some info on how to use event handlers at https://github.com/pywinrt/pywinrt/blob/main/projection/readme.md#event-handlers.

[tfluan0606]

I have modified the Event Handlers example, but I think I made something wrong, my hadler never work.

#these function write in a class

#handler
def handle_received(self,sender, event_args):
  self.frame = sender.try_get_next_frame()
  self.process_frame()

async def get_frame(self):
  event_loop = asyncio.get_running_loop()
  self.frame_arrived = self.framepool.add_frame_arrived(lambda s, e: event_loop.call_soon_threadsafe(self.handle_received, s, e))
  try:
      self.session.start_capture()
      #I think the wrong place is here?
      async def check():
          while self.frame == None:
              print('wait for frame')
              await asyncio.sleep(1)
      printer_task = asyncio.create_task(check())
      await printer_task
  finally:
      self.framepool.remove_frame_arrived(self.frame_arrived)
      self.close_all()

#wincap is the class name
asyncio.run(wincap.get_frame()) 

program stuck on the check(), keep printing "wait for frame", that means my handler never invoke. I've been trying all day, but still can't solve it.

[tfluan0606]

After I changed from 'create' to 'create_free_threaded', my handler seems to have executed successfully. But it can only be executed once.

self.framepool = wgc.Direct3D11CaptureFramePool.create_free_threaded(
                    self.d3d,
                    DirectXPixelFormat.B8_G8_R8_A8_UINT_NORMALIZED,
                    1,
                    self.item.size)
self.session = self.framepool.create_capture_session(self.item)

def handle_received(self,sender, event_args):
        print("hadle occur")
        self.received_queue.put_nowait(event_args)
        self.frame = sender.try_get_next_frame()
        self.process_frame()

async def get_frame(self):
    event_loop = asyncio.get_running_loop()
    self.frame_arrived = self.framepool.add_frame_arrived(lambda s, e: event_loop.call_soon_threadsafe(self.handle_received, s, e))
    self.session.start_capture()
    i = 0

    while i !=5:
        await self.received_queue.get()  #loop run only one time,and stuck on here, that means handler only put one event in Queue, waiting for event to get
        i+=1
        print('check '+str(i))

#wincap is the class name
asyncio.run(wincap.get_frame())

what I want is to keep getting new frames It seems like the framepool buffer never clear? When I changed the buffer size of the frame pool to 2, it could only be executed twice.

[sebmerry]

@tfluan0606 did you figure it out? I was hitting the same issue where it seemed like the framepool's frames weren't being released so it would get "stuck" at received_queue.get() once the framepool was filled (number_of buffers specified during frame pool creation).

It turned out to be how I was getting the D3D device! I was using LearningModelDevice to get the D3D device. The workaround was to use MediaCapture. See this comment: https://github.com/pywinrt/python-winsdk/issues/11#issuecomment-1315345318

[tfluan0606]

@tfluan0606 did you figure it out? I was hitting the same issue where it seemed like the framepool's frames weren't being released so it would get "stuck" at received_queue.get() once the framepool was filled (number_of buffers specified during frame pool creation).

It turned out to be how I was getting the D3D device! I was using LearningModelDevice to get the D3D device. The workaround was to use MediaCapture. See this comment: #11 (comment)

I'm sorry, I haven't found a solution yet. I'm also using the LearningModelDevice approach, and I don't know how to use the MediaCapture.

[Avasam]

Both approaches are still workarounds/hacks and have their limitations. I'd prefer if python-winsdk exposed (or documented using ctypes/windll maybe?) an official way to get the Direct3D Device.