Displaying on top of the klibs window.

[nathanvdh]

Hi, I'm assisting with an experiment in which there is a video played halfway through the experiment. Currently we're just doing this with a subprocess call to run ffplay (from ffmpeg) and play the video. However, the klibs window seems to have exclusive fullscreen and the ffplay window won't show on top. I've noticed that the ffplay window WILL show on top (as desired) on the couple of laptops I've tried, but on the two PCs I've tried nothing will display on top of the klibs window. I can get the desired behaviour on the PCs by removing the SDL_WINDOW_FULLSCREEN_DESKTOP flag from the window created in KLGraphics/core.py. So that's a bit of a hack at the moment.

Could you make this a configurable option? Or would you consider removing that flag? Do you have any other thoughts?

Thank you for the library, it's quite good :)

[a-hurst]

Hi @nathanvdh, thanks for the bug report! Unfortunately SDL_WINDOW_FULLSCREEN_DESKTOP is necessary for fullscreen to work properly on macOS, and is the only option that properly detects and uses the current desktop resolution on all platforms (though this latter bit can be worked around). I can definitely make it a _params.py-configurable option for your sake, though.

Another option would be to play the video directly in KLibs: I recently had to figure this out for an eye tracking experiment (the PI was interested in whether people with higher anxiety would spend less time looking at clips with negative emotional valence), and it seems to work pretty well! Basically, you can read in the video with the scikit-video package, save the audio as a separate MP3/Ogg file, and then render the frames of the video (which are numpy arrays) with blit(). Here's the class I came up with:

class VideoClip(object):

    def __init__(self, path, fps=30, opacity=255, audio=None):

        self._path = path
        self.audio = None
        self.fps = fps

        self._clip = skvideo.io.vread(path)
        self.frames, self.h, self.w, _ = self._clip.shape
        self.duration = self.frames * (1 / self.fps)

        self._frame = 0
        self._start_time = None
        self._tmp_rgba = np.full((self.h, self.w, 4), opacity, dtype=np.uint8)

        if audio:
            self.audio = AudioClip(audio)

    def play(self):
        self._start_time = precise_time()
        if self.audio:
            self.audio.play()

    def reset(self):
        self._frame = 0
        self._start_time = None
        if self.audio:
            self.audio.stop()

    def next_frame(self):

        if self.playing:
            self._tmp_rgba[:, :, :3] = self._clip[self._frame][:, :, :]
            self._frame += 1
            frame_by_time = int((precise_time() - self._start_time) / (1 / self.fps))
            if frame_by_time > self._frame:
                self._frame = frame_by_time

        return self._tmp_rgba

    @property
    def playing(self):
        return (self._frame < self.frames and self._start_time)

    @property
    def size(self):
        return (self.w, self.h)

Then, in the actual experiment code, I play the clips like this:

        self.clip.play()
        while self.clip.playing:

            ui_request()
            elapsed = last_frame_time - start_time

            # Check for gaze position and fixation end events on the video clip
            el_q = self.el.get_event_queue()
            clip_bounds.update_gaze(el_q)
            clip_bounds.update_fixations(el_q)

            # Fetch the next frame for the video and draw it to the screen
            frame = self.clip.next_frame()
            for refresh in range(2):
                fill()
                blit(frame, 5, P.screen_c)
                flip()
            last_frame_time = precise_time()

            # If in devmode, skip to the next trial when any key is pressed
            if P.development_mode and key_pressed():
                self.clip.reset()

The main annoyance is that skvideo needs an ffmpeg binary and doesn't supply one, so you either have to install one yourself with a package manager (if using Linux or macOS) or alternatively add the static_ffmpeg package as a dependency and try to point skvideo towards that (a bit of a pain, but that got it working on the Windows 10 PC in the lab I programmed it for).

Hope that helps!

[nathanvdh]

Thanks very much, I wasn't very familiar with the KLGraphics display stuff at all. I ended up going with opencv-python instead of scikit-video, I wanted to use its resizing and it also comes bundled with ffmpeg. Also, I don't need audio, just to loop video for a specified amount of time. Here's the code in case it's of interest to anyone else:

class VideoClip(object):

    def __init__(self, path):
        self.path = path
        self.cap = cv.VideoCapture(path)
        self.fps = self.cap.get(cv.CAP_PROP_FPS)
        self.src_h = self.cap.get(cv.CAP_PROP_FRAME_HEIGHT)
        self.src_w = self.cap.get(cv.CAP_PROP_FRAME_WIDTH)
        self._last_frame_time = None
        self.scale = 1

    def start(self, fullscreen=False, screen_size=None, opacity=255):
        if fullscreen:
            if not screen_size:
                raise ValueError('Screen size must be specified for fullscreen playback')
            scr_h = screen_size[1]
            # currently only resizing based on height
            self.scale = scr_h/self.src_h
        ret, frame = self.cap.read()
        if not ret:
            raise RuntimeError('Error reading video')
        # KLGraphics requires an alpha channel
        self._current_frame = cv.cvtColor(cv.resize(frame, None, fx=self.scale, fy=self.scale, interpolation=cv.INTER_CUBIC), cv.COLOR_RGB2RGBA)
        self._last_frame_time = precise_time()

    def reset(self):
        self._last_frame_time = None
        self.scale = 1

    def next_frame(self):
        if self._last_frame_time is None:
            raise RuntimeError('Must call start() before geting video frames')

        if (precise_time() - self._last_frame_time >= 1/self.fps):
            ret, frame = self.cap.read()
            if not ret:
                # loop video after end
                self.cap.set(cv.CAP_PROP_POS_FRAMES, 0)
                _, frame = self.cap.read()
            self._current_frame[:, :, :3] = cv.resize(frame, None, fx=self.scale, fy=self.scale, interpolation=cv.INTER_CUBIC)

        return self._current_frame

    @property
    def size(self):
        return (self.src_h*self.scale, self.src_w*self.scale)

Used by:

    def play_video(self, length=300):
            self.video.start(fullscreen=True, screen_size=P.screen_x_y)
            start_time = precise_time()
            while (precise_time() - start_time < length):
                ui_request()
                frame = self.video.next_frame()
                for refresh in range(2):
                    fill()
                    blit(frame, 5, P.screen_c)
                    flip()