Custom string with the api

iFieldSmart-Tech commented 3 months ago

Our use case is we have a plugin that acts as a bridge for communication between our mobile app and the camera. For images, we call the plugin's api which has a parameter that accepts a string, the plugin calls the camera's api to take picture and once done, update the resulting image's exif data with the string received with the api. We tried the same with the video, but after the camera's stop recording api is called, it exits, we do not receive any response from the camera. The recording stops, but we do not get to update the video's exif data with the string received in the plugin's api. We also tried adding a 15-20 seconds wait, but even after waiting it does nothing, the code exits, no breakpoints are hit. Can you please help us with understanding what we are doing wrong? or any other way to pass a custom string with the start or stop recording api calls?

codetricity commented 3 months ago

From reading the forum, it looks like you're using the WebAPI from inside the plugin.

This is what is in the forum

So the app calls the plugin's api which is a custom api,

192.168.1.1:8888/video_start

This triggers the plugin to call the camera api to start recording. The api called by the plugin is,
192.168.1.1/osc/commands/execute

"name", "camera.startCapture"

DaisukeHohjoh commented 2 months ago

@iFieldSmart-Tech @codetricity

I am sorry for our late checking for your issue.

I read the post too, and we need to say that Theta X plugin does not support WebAPI to start/stop video recording. I am sorry that this spec difference between Theta Z1 and X is came from the difference of plugin architecture.

If you want to do video recording in plugin, using the Camera API is strongly recommended. This repository is showing how to use Camera API to execute take picture and start/stop video recording.

Best regards, Daisuke Hohjoh

codetricity commented 2 months ago

@DaisukeHohjoh oh, this is interesting information. I tried for many days to use Flutter with the webAPI to start/stop the video inside the plugin. I had assumed it was a limitation of the Flutter architecture.

Do you know of other restrictions with the webAPI used in the THETA X plugin?

https://github.com/ricohapi/theta-api-specs/blob/main/ricoh-theta-plugin/web-api.md

Is there an example of how @iFieldSmart-Tech can get the file location of the video file using the camera api and then feed that into another method to process the file (such as append metadata to the notes field).

Maybe this needs to be set?

https://github.com/ricohapi/theta-plugin-camera-api-sample-x/blob/5115c7ff53449a716d8d0c300911c27472e7b51d/app/src/main/java/com/theta360/sample/camera/MainActivity.kt#L52

There's an example here that may be related to getting the file path?

https://stackoverflow.com/a/8554433

DaisukeHohjoh commented 2 months ago

@codetricity

Do you know of other restrictions with the webAPI used in the THETA X plugin?

Mainly video recording feature / video related Options are unavailable for Theta X. As you said, RIcoh need to add description about this limitation to https://github.com/ricohapi/theta-api-specs/blob/main/ricoh-theta-plugin/web-api.md .

If plugin use Camera API / MediaRecorder API to do video recording, you need to specify the file name yourself.

https://github.com/ricohapi/theta-plugin-camera-api-sample-x/blob/5115c7ff53449a716d8d0c300911c27472e7b51d/app/src/main/java/com/theta360/sample/camera/MainActivity.kt#L469

I am sorry for NOT my sharing this information in public.

Best regards, Daisuke Hohjoh

iFieldSmart-Tech commented 2 months ago

So using camera api is suggested, but the main reason for using the web api is to send a string with the plugin's webapi which needs to be inserted in the output video's exif data. We cannot do it with the camera api, right? In order to send a string along with the api, we have no choice but to use a web api that triggers the camera api, right? Or is there any other way to achieve our use case? The end goal is just to pass a string along with the video recording start api. This api will be called from our mobile app and the plugin will in turn start the recording and update the video exif data once the file is ready.

codetricity commented 2 months ago

@iFieldSmart-Tech when you setOutputFile(mFilepath), you know where the video file is going to be.

If you search on google, there seems to be many solutions to read/write metadata of the mp4 container. i have not tried any. However, this one looks like it can do it. https://github.com/sannies/mp4parser

iFieldSmart-Tech commented 2 months ago

@DaisukeHohjoh If i try another way; is it possible to fetch the video metadata without having to use a plugin in between? Is it possible using only the camera and its api? Can i get the exif data from a video directly from the camera using the camera api and without having to download the video on the phone first? If that is possible that provides a workaround for my use case

DaisukeHohjoh commented 2 months ago

@iFieldSmart-Tech

Can i get the exif data from a video directly from the camera using the camera api and without having to download the video on the phone first?

Theta Plugin Library maybe able to do it but I think I need to understand correctly your requirement. I read this post and my understanding and questions are following.

You are developing mobile phone app.
Your mobile phone app can send WebAPI (or some kind of commands) to do take picture and start/stop video .
Your mobile phone wants to get only Exif information without downloading photo / video to smartphone.

My question #1 : What kind of Exif metadata do you need ? Just to get Exif metadata information from video, WebAPI camera._getMetadata (link) may help it. Have you already checked it ?

My question #2 : Based on this your comment, you hope to insert? add? overwrite? some kind of information to Exif in captured photo / video, correct ? Can you tell more detail about this requirement ? For example,

What metadata do you insert to Exif ?
Your mobile photo does it after downloading photo and video to smartphone ? Or else, do you want to do it inside Theta, so that you need to develop Theta plugin ?

Best regards, Daisuke Hohjoh

iFieldSmart-Tech commented 2 months ago

@DaisukeHohjoh So our first goal was to insert exif data into the image/video while the file is still on the camera, no need to download it to the phone to insert the data. We noticed an exif parameter "ImageDescription" which is always empty so decided to insert a random string in it so later when the file is uploaded to our portal, we can read that parameter and determine which metadata that file belongs to. We were able to do it successfully with images, but were struggling with videos. The ImageDescription parameter is in the videos as well and just like the images, it is always empty and can be used for our custom purpose.

As we were failing at inserting the data for video files we thought as a workaround maybe we try reading all the exif data and save the whole data in our db which can later be used for mapping. But regardless, yesterday we were able to get it to work without the plugin. We can get list of files directly using the camera api, so we decided to ditch the plugin for videos. We are now using the camera api to start and stop recording and then get the filename once the recording stops and the file is created. We will rely on the filename for the mapping.

DaisukeHohjoh commented 2 months ago

@iFieldSmart-Tech Thanks for sharing it with us.

So.. does following my understanding match your direction ?

You will NOT develop your own Theta plugin.
- This means also, you decided NOT to insert some strings to exif ImageDescription tag of photo and video.
Your mobile phone app send WebAPI to execute command camera.takePicture (photo) and camera.startCapture (video).
Your mobile phone app get metadata information of photo and video, by WebAPI camera.getMetadata and camera.listFiles.
- Your mobile phone app will use these information for some kind of mapping between filename and its contents.

The ImageDescription parameter is in the videos as well and just like the images, it is always empty and can be used for our custom purpose.

This is just for your information, if you develop plugin to capture photo and/or video with using Camera API, you can insert ImageDescription tag by following Camera.Parameter setting.

        camera?.apply {
            val p = parameters
            p.set(
                Camera.Parameters.PLUGIN_INFO,
                "You can put any String"
            )
            parameters = p
}

With this setting, photo and/or video will have ImageDescription as your setting String value.

Best regards, Daisuke Hohjoh

iFieldSmart-Tech commented 2 months ago

Hi @DaisukeHohjoh

Thank you for the detailed explanation. However, we went ahead and did it without the plugin and using only the camera api. We use to camera api to start and stop capture, then stop capture returns the filename, we save that filename and the timestamp to our database. When user uploads the file later, we use the filename and the exif data timestamp to map the two. This still leaves an issue where in case if the user changed the filename before uploading the video, but for now, we need to live with that issue and request the user to not change the file names.

iFieldSmart-Tech commented 2 months ago

@DaisukeHohjoh I needed help with something else too. The ricoh desktop app has the option to perform stitching as well as top bottom correction on videos. We figured out the stitching part but not sure how to do the topbottom correction at our end. Is there any api that can do that? It does not have to be done on the camera, we can do it on our server after the user uploads the video. But is there any way to programmatically figure out which video needs correction and how to do it?

DaisukeHohjoh commented 2 months ago

@iFieldSmart-Tech

we went ahead and did it without the plugin and using only the camera api.

I understood your direction. Thank you.

how to do the topbottom correction at our end. Is there any api that can do that? ....

Theta X Theta X can do topbottom correction on the camera. Default value is ON. So I expect that you need not consider about it for Theta X.

Theta Z1 On the other hand, Theta Z1 cannot topbottom correction on the camera for video recording mode. If you want to do it at your end (such as on your server), without Ricoh app, followings are option.

Calculate camera atitude with using IMU sensor data which is stored in MP4 file, and do post-processing at your end. Please refer to here for meta data information.

is there any way to programmatically figure out which video needs correction and how to do it?

Yes MP4 file has such Ricoh specified metadata but it is not disclosed now. If you need such information, I think Ricoh can discuss about it.

Best regards, Daisuke Hohjoh

ricohapi / theta-plugin-camera-api-sample-x

Custom string with the api #20