[Feature Request] Recording gyro data when recording video

[igorkulka]

Is your feature request related to a problem? Please describe. I'd like M5StickC Plus to start recording gyro data when video recording is triggered (and stop recording gyro data when video recording is stopped) so it can be used for post-process video stabilization using Gyroflow.

Describe the solution you'd like Many newer Sony cameras such as Sony ZV-1, ZV-E10, Sony FX30, Sony α7 IV etc. can record gyro data which are later on used for post-process video stabilization using Sony's proprietary software Catalyst Browse. However there's similar open source software called Gyroflow that can load data from external source and use it for video stabilization in a similar way Sony's Catalyst Browse does it. It'd be awesome to have such a small device as M5StickC Plus is, attached to older Sony cameras (such as Sony a6400 in my case) to achieve similar results.

Describe alternatives you've considered Older Sony a6xxx series cameras (a6000, a6300, a6500) run on Android system and are tweakable (f.e. using this ) so maybe it could be possible to implement android app that would capture camera's internal gyro sensor data. However as far as I know, this is not possible for newer but not entirely new models such as Sony a6400 or a6600.

[igorkulka]

here's some gyro logger repo that should be compatible with Gyroflow in case it'd be useful

[frank26080115]

Maybe I will work on this, it is in the spirit of what I typically do. But I don't make enough videos to have expertise on this topic.

I did some calculations, 60 FPS means roughly 16 milliseconds per frame. if I did one sample every 5 milliseconds, then the M5StickC-Plus can record about 800 seconds of IMU data. And that's in a binary format, not CSV. Another tool will be required to convert the binary into the CSV that Gyroflow can use.

(I am assuming a 2MB memory, the M5StickC-Plus has 4MB total but I have other files on there)

Getting files off of the internal memory will be a pain in the ass. It would require the usage of a serial terminal, the user would need to type in some Wi-Fi credentials, then if the connection succeeds, the user can download a file through a URL displayed in the terminal. Maybe the HTTP server can do the file conversion though.

The screen will not be able to show a timer. The current code prioritizes communication with the camera, but is not well suited for super timing accurate tasks like logging with millisecond precision. Updating the screen all the time will make it hard to achieve the timing. So if I implement this, the screen will simply show "logging started", and it will end when the user presses the button. I can make the LED flash as it logs. There will be no connection to the camera during this time.

What do you think?

[igorkulka]

Maybe I will work on this, it is in the spirit of what I typically do. But I don't make enough videos to have expertise on this topic.

First of all, thanks for your willingness to work on such thing. It's highly appreciated.

The current code prioritizes communication with the camera, but is not well suited for super timing accurate tasks like logging with millisecond precision.

I think the precision doesn't necessarily need to be spot on as there's auto sync function in Gyroflow that actually can match gyro data graph with optical flow algorithm approximation of the movement. So as soon as it's roughly in sync with recording, it should be good enough.

The screen will not be able to show a timer. The current code prioritizes communication with the camera, but is not well suited for super timing accurate tasks like logging with millisecond precision. Updating the screen all the time will make it hard to achieve the timing. So if I implement this, the screen will simply show "logging started", and it will end when the user presses the button. I can make the LED flash as it logs. There will be no connection to the camera during this time.

Sure, this is no problem, I understand it's challenging. There's no need to see timer on the screen.

I did some calculations, 60 FPS means roughly 16 milliseconds per frame. if I did one sample every 5 milliseconds, then the M5StickC-Plus can record about 800 seconds of IMU data. And that's in a binary format, not CSV. Another tool will be required to convert the binary into the CSV that Gyroflow can use.

(I am assuming a 2MB memory, the M5StickC-Plus has 4MB total but I have other files on there)

Getting files off of the internal memory will be a pain in the ass. It would require the usage of a serial terminal, the user would need to type in some Wi-Fi credentials, then if the connection succeeds, the user can download a file through a URL displayed in the terminal. Maybe the HTTP server can do the file conversion though.

I see, the memory limitation is tricky. Usually those videos capturing motion aren't too long, however it'd mean it'd be necessary to download gyro data after every ~13mins of the footage which can be suboptimal. I am wondering if anything from the following would be possible:

1. it'd be possible to add some memory module to M5StickC Plus to extend the storage
2. pair-up with f.e. smartphone app and offload the gyro data once recording is finished
3. run the same code on some other M5 device, f.e. M5Stack Core 2 that has memory card slot

This is just quick brainstorming, I don't expect you to implement f.e. mobile app for the second idea. I am just trying to figure out some convenient solution.

[frank26080115]

There's no way to extend the memory. The spare pins available are not enough.

I don't ever make apps for IoT projects. I always simply implement a web portal, and code everything using HTML and JavaScript. So the app is always a web app, no install, no security risk, no need to buy a mac... I think I already implemented Wi-Fi AP mode HTTP server for the Alpha-Fairy so that part is not hard.

The M5Stack Core would help. But getting the rest of Alpha-Fairy working on it would take a lot of work.

Uh, I have a robotics competition at the end of August I'm focusing on, don't expect me to work on this fast

[igorkulka]

Ok I understand. Just to double check.

I think I already implemented Wi-Fi AP mode HTTP server for the Alpha-Fairy so that part is not hard

does this mean I'd be able to connect and download data via smartphone browser without need to connect M5Stick using cable?

The M5Stack Core would help. But getting the rest of Alpha-Fairy working on it would take a lot of work.

This would be unnecessarily difficult it seems like. If there's convenient way to download data from it, 13mins of gyro data on M5StickC Plus will be enough.

Uh, I have a robotics competition at the end of August I'm focusing on, don't expect me to work on this fast

OFC, no pressure on this. I'll fork your repo and try it myself in the meantime once I'll get my M5StickC Plus I ordered even though this would be my first experience with writing code for such device.