Working on sending frames to p5 for BodyPix processing

[shiffman]

Quick question before I keep going? Would I be better served by loading the image files from the client itself or is my methodology of loading using fs and sending as base64 via a message sound?

      const results = await unpackVideoToFrames(arg);
      // Hard coding one frame for now
      const imgb64 = fs.readFileSync('frames/out001.jpg', {
        encoding: 'base64'
      });
      event.reply('FRAMES_READY', imgb64);

[joeyklee]

@shiffman - I'm def not an expert in this, but as a simple implementation this makes sense.

One thing worth looking into is maybe the https://www.electronjs.org/docs/api/native-image electron Native Image object? Maybe this helps simplify things and also reduces the need for loading images on both server and client?

[shiffman]

Ah this is a great idea, I will definitely take a look, thank you for sharing!!

[joeyklee]

I think what you can do then is send the jpg or png image to the renderer with the .toPNG() function https://www.electronjs.org/docs/api/native-image#imagetopngoptions

[shiffman]

native-image worked great!

[joeyklee]

@shiffman YESSS DANNNNNN

[shiffman]

This is not complete but we could merge it in for continued work. To do's off the top of my head are:

• [ ] auto-detect canvas size (currently hard-coded)
• [ ] get total frames from FFMPEG and pass in (currently hard-coded)
• [ ] send each frame back to write to video file

I won't be able to return to this until tomorrow so we can merge if anyone else wants to pick up where I left off.

[joeyklee]

@shiffman - let's merge in and we can work from where you've left off. I think I can resolve at least auto-detect canvas size (currently hard-coded)

[shiffman]

Great, just fixed a merge conflict let me make sure it still works and I will merge.

[shiffman]

Ok, merging!

[joeyklee]

🙏