Gimbal and Camera control screen

[rmackay9]

ArduPilot has recently implemented (at least partially) the MAVLink gimbal v2 messages. There are also a growing number of highly functional 3-axis gimbal+cameras available to users at reasonable prices (e.g. the Siyi A8, Gremsy ZIO).

The missing piece is the ground station controls provided to users of smart tablets running QGC (like the Herelink or Siyi MK32). It is quite cumbersome for users to control the gimbal and camera because of the limited number of buttons on these controllers compared to the large number of controls AP provides the user including pitch and yaw control, take a picture, start/stop recording video, zoom in/out, focus in/out, auto focus.

I think we should somehow enhance the live video feed screen to allow the user to control the gimbal angle and the camera features.

Related MP issue including a picture with my thoughts on what the screen might look like https://github.com/ArduPilot/MissionPlanner/issues/2906

[zdanek]

This is great idea. Thank you @rmackay9 I totally support it and might help with an implementation. Lately I had to implement controlling on advanced Topotek camera but we could not choose Gimbal messages. Instead I made a "translator" from RC channel commands into Topotek's native protocol. It was made on Drone, in companion computer. This could be made as a translation of dedicated Gimbal messages if we can finish what you propose.

[Davidsastresas]

This would be an interesting feature, and I am willing to work on it. I think there are several ways to approach it, I will share my thoughts here to start a discussion:

• QGC currently supports extended parameter protocol to populate advanced camera menus. For instance, Workswell cameras use this. These cameras talk mavlink on its own, and they provide QGC with a link to download their parameter definition. Having that parameter definition file, QGC is able to populate an advanced menu with extended controls of the camera, like zoom, thermal palette, etc etc.

I wonder if we could take advantage of the above somehow to work on this. Because from your description, it seems it is not yet "standarized" so each gimbal has its own set of features working. If we could do something like this it would be easy for QGC to present to the users only the features supported.

• Another approach could be to read the MNT_TYPE of Ardupilot, and populate our gimbal menus based on this. This is not as ideal, as it requires extra manteinance in case Ardupilot changes the features supported by those cameras, but could be done as well.

• We need to know how PX4 manages all of this. My understanding is that Ardupilot is using here Mavlink standards, but I am not sure if PX4 has these capabilities yet. Maybe we can work over APMFirmwarePlugin for the moment.

• Regarding the graphical interface itself, where should we place it? I was thinking on making these controls only visible when video is set to full screen ( map minimized ). Otherwise in small Android screen it will be easy for the screen to get crowded with information and control from flight control and camera control.

I think most of the time a user will want to use these control, they will most likely be on full screen video. Does this assumption make sense? Placing the controls in the full video screen would also allow in desktop to have all of this in a separate screen, as the video window can be detached. I think this could be interesting.

• If I am not mistaken, we support in Ardupilot both velocity and position control for gimbals. Your sketch of the buttons makes sense for position control, but it would be weird for velocity control. In my experience, camera control is way more natural in speed versus in position. But I do think position control might have benefits under some kind of operations.

Having said the above, I think for speed control nothing beats a virtual stick, being compact and intuitive, something similar to what you see on QGC when you enable "virtual sticks" under QGC settings. I think we could do something like this for pan/tilt camera control.

for position control, I think we could do a combination of Sliders and a couple of buttons for fine control. It would be more compact and hopefully more intuitive than only button based controls.

I would also place separately movement controls ( pan/tilt ), from modes control. So I would place in a group pan/tilt movement, maybe having closely center buttons as well, and I would place the lock and control modes option elsewhere in the screen. This approach should be more intuitive. This way we can place the pan/tilt control close to any of the edges of the screen, closer to the thumbs, as they will be used more often, and the other controls elsewhere, even in a submenu, like having a small icon that after pushing it, opens a submenu with the rest of the options.

@rmackay9 what do you think about the advanced parameter protocol? Does something come to your mind about using this for this matter?

I am willing to work on this, but the scope is quite broad at the moment, so I would love to narrow down the specifications of this request. Any feedback is welcomed! Thanks!

[rmackay9]

@Davidsastresas,

I think the virtual sticks idea appearing on the video when it is in full screen mode would be great. My ideas on the screen pasted above were really designed for MP which doesn't have the virtual sticks that QGC has. So for QGC I agree these would be great and then we could lose all the position and rate buttons. I think users would still appreciate center and "lock" buttons ("lock"/"follow" only applies to yaw by the way).

I don't think that QGC should talk directly to the gimbals.. I think there's too much variety, too many differences in capabilities. If we need QGC to know what the camera and gimbal are capable of we should perhaps make a gimbal-capabilities and/or camera-capabilities MAVLink message.

[Davidsastresas]

Great, then we can do a first approach with the virtual joystick, see how it feels and keep iterating from there in case we need buttons for finer control.

We will need something a configuration tab somewhere to configure this joystick. My understanding is that we would be using a cmd_do_mnt_control, correct? If so, the "rate or position mode" we have in Ardupilot ( rc controls actual angle, or rc controlling angular speed ) if i am not mistaken is handled inside Ardupilot itself, but there is no way of sending rate control by mavlink message right? Please let me know if you have more information about this one.

If we don't have possibility of rate control by mavlink, we would need to do a workaround in QGC to have this mode using position messages. We could also use mavlink rc channel override capability for this matter, detecting the ardupilot parameters used for mnt_rc_tilt/pan_in in case of rate control.

Although I would avoid this one as it could lead to users missunderstanding this and doing crazy stuff that could lead to dangerous situations, like setting mnt_rc_tilt/pan_in to a channel used for any other mayor function like roll control, QGC detecting roll channel as mount tilt channel for example, and imagine the rest... When changing to full screen and controlling gimbal it would actually command roll, and in that moment it could be in the air already, so I really don't like that approach. Obviously the users should not do that configuration in parameters, but it could happen.

In short, I think we would need a way of controlling the mount by mavlink with rate control. What are your thoughts on this? i just saw there is some work in progress in mavlink, it is exactly what we need for this, but not sure of the state of it:

https://mavlink.io/en/messages/common.html#GIMBAL_MANAGER_SET_ATTITUDE https://mavlink.io/en/messages/common.html#GIMBAL_MANAGER_SET_PITCHYAW

Regarding QGC talking to gimbals or not, I agree, at some situations/setups it will make sense, but not always. I imagine we could send from Ardupilot https://mavlink.io/en/messages/common.html#GIMBAL_DEVICE_INFORMATION, so QGC understands the capabilities of the gimbal. But we would need to manage somehow from Ardupilot the gimbals that will actually send this themselves. Maybe just a parameter or mount type with this consideration? let me know your thoughts please.

We need this because some advanced gimbals that support video tracking, have thermal cameras, etc could need to report its own capabilities to QGC directly, and being controlled by QGC directly. For examples, in these kinds of gimbals it doesn't make sense that a zoom command is processed by the autopilot, or a tracking command. The gimbal will receive the tracking command, and it will talk to the autopilot if needed to change yaw, possibly follow target using guided mode in Ardupilot, etc.

Let me know your thoughts!

[rmackay9]

@Davidsastresas,

For the virtual sticks it probably makes sense to use the GIMBAL_MANAGER_SET_MANUAL_CONTROL and then set the "pitch" and "yaw" fields to NaN and "pitch_rate" and "yaw_rate" according to the stick position. AP doesn't support this message yet but we could add it easily. We would convert the -1 to +1 to an actual rate using the MNT1_RC_RATE parameter perhaps so the ground station wouldn't need to know the maximum rotate rate of the gimbal.

The alternative is to use GIMBAL_MANAGER_SET_PITCHYAW which AP already supports.

For now I'm going to stick with my position that QGC should never directly talk to the gimbal. For the zoom example, AP can control the camera's zoom and this allows this to be done either using RC switches, Lua scripts or autonomous missions. It's possible you're right for the tracking situation (e.g. user clicks on the screen to indicate what they want to follow) but I think many other situations should go through the autopilot.

[highfreq]

We implemented virtual josticks in the past and are rather a nightmare to use on the rather smalm monitors of current radios. We find much more usefull to use double tap on the point we want the gimbal to point to.

[rmackay9]

@highfreq,

That's an interesting idea. It would mean knowing the FOV of the camera though which would likely need to be a bit of configuration held in the GCS.

[highfreq]

Yes, we read fov based on zoom level and we move accordingly, a lot easier to use than touch commands that never work good on small panels, expecially on cold days.

[rmackay9]

@highfreq,

OK, thanks. I'm not sure that we have a way for the autopilot to send back the current zoom level of the camera to the GCS.. maybe we do in the new MAVLink camera messages but I haven't investigated those much yet.

[Davidsastresas]

Understood. I think next week I will spare the time to do a first approach to this.

I will review how Ardupilot manages the GIMBAL_MANAGER_SET_PITCHYAW message, and I think I will start from there, as it is already supported. We can always keep evolving this and adapt when ardupilot includes the new messages.

@highfreq what you say makes sense. As Randy says we would need the FOV of the camera, but it would be cool to try. It won't be super precise because it isn't only a matter of FOV, but also of lens disortion, not all of them are linear, this is more true the wider the FOV is. Ideally we would need to put in there a whole lens calibration file, similar to how we calibrate cameras when working with image processing for depth perception, precision landing, etc. I guess we can try with a linear approach and see if it is nice enough to use!

I think I will add a new menu under vehicle settings, available if mnt_type is detected. In that menu we can add all the configuration necessary for this.

[highfreq]

We use a linear approach on Sony blockcm 7520A and 8530. They work surprisingly well. When fully zoomed in on a 30X camera moving it with a stick is quite difficult but tapping on the screen is as easy as when not zoomed.

[Lucky1st]

About "standardization", in the world of ptz cameras, the VISCA and PELCO protocols is quite common. But they also don't standardize changing optical and thermal channel e.g.

It seems to me that at this stage it is impossible to do without an adapter board or companion computer on board that will convert MAVLink camera messages to VISCA or any other protocol understandable for the gimbal

What can be implemented using the simplest Arduino and the MAVLink pass-through function over UART

[Davidsastresas]

Yes, that is true about VISCA, but as it is a serial protocol there isn't really a way of implementing that directly in QGC. The approach would be to include it in the autopilot firmware, and from there control from the GCS using mavlink.

[Lucky1st]

Oh sure. I just feel the need to mention it here.

Personally, when working with the gimbal and the camera, I encountered two problems:

Lack of interface for gimbal and camera control Lack of documentation on how to send commands to a VISCA-controlled camera via (GCS/RC)/MAVLink/Autopilot

And the guesswork begins. Perhaps I need custom MAVLink commands for FOV and focus and then listen to listen to UART and convert signals and so on

With a gimbal, of course, it’s easier because BaseCam has done a lot of work in this direction.

Perhaps I'm missing something

I'm new to this and I'm giving the user's opinion and don't have much expertise to debate the best way to implement this feature. For now :)

Returning to the discussion, in the file src/Joystick/Joystick.cc there are assignable commands "Continuous Zoom In/Out", "Gimbal Down/Up/Left/Center", based on this, there are some standards for passing these commands within projects QGC/MAVLink/Ardupilot or is it purely internal functions that are mapped to RC channels?

[Davidsastresas]

I understand, thanks for the feedback!

Indeed Basecam is a joy to work with, it is nicely integrated in Ardupilot, but I agree everything related to gimbals could be better on current GCS. We are trying to improve that now!

Regarding Visca, I did some development to support basic visca commands ( I think only zoom ) over Ardupilot. You could hook one serial port to the Visca port and zoom could be controlled by RC in ardupilot. It is probably very outdated now, and it would need a rebase to be up to date with today's ardupilot. If you are interested let me know and I will point you to the branch.

Regarding what you saw in Joystick.cc, indeed there are some nice interesting options. I tracked some of them and it seemed they were meant to be sent to a "mavlink camera". Nowadays some cameras talk mavlink, and you can command zoom to them directly from the GCS, using mavlink ( like workswell cameras ). Some of those functions for joystick also send gimbal controls, although I think the one it is using is deprecated for Ardupilot. After the matter on this issue is finished, I will probably go over all of it to get it updated.

Thanks for the feedback!

[rmackay9]

I agree with @Davidsastresas that the low level controls (e.g. visca commands) should be in the autopilot driver and then the ground station can hopefully control those features via mavlink. Coincidentally I'm working on an ArduPilot ViewPro driver written in Lua which implements a few visca commands. It doesn't work yet but in the coming weeks it probably will.

[OpenSource-Study]

Hey @Davidsastresas ,

I think you are the right guy and probably the only guy who has done some development for the VISCA protocol-based camera control. I would really like to know and am interested in controlling my VISCA protocol-based camera ZOOM. I don't want any other control, just the ability to zoom in and out. If you could guide me on how to make this happen, I'd be super grateful. Thanks a bunch in advance, and I'm really excited to hear what you have to say about it!

Cheers!

[Davidsastresas]

Hey @Davidsastresas ,

I think you are the right guy and probably the only guy who has done some development for the VISCA protocol-based camera control. I would really like to know and am interested in controlling my VISCA protocol-based camera ZOOM. I don't want any other control, just the ability to zoom in and out. If you could guide me on how to make this happen, I'd be super grateful. Thanks a bunch in advance, and I'm really excited to hear what you have to say about it!

Cheers!

Hello!

Some years ago I did some work over Ardupilot to allow this, via an aux channel, it only supports zoom in/out. It needs a huge rebase to work on modern Ardupilot, and the way it is done is no longer nice, if we were to do that nowadays it should go on AP_camera, not on AP_Mount, but anyway, I think you can pull that string to make it work on your own probably.

https://github.com/Davidsastresas/ardupilot/tree/Visca_Protocol

Best regards.

[rmackay9]

@Davidsastresas @OpenSource-Study,

I've create issue https://github.com/ArduPilot/ardupilot/issues/24815 to capture this enhancement request. I'm tempted to try and revive this. Let's discuss on the issue or here in the forums.

[OpenSource-Study]

Hey @rmackay9 and @Davidsastresas,

A big shoutout to both of you! Your work has been amazing. I'm thrilled about the idea of bringing back David's work on the VISCA Protocol. I've been a big fan of your efforts. If there's anything I can do to help, like creating documents or testing code or any other thing, count me in. Here's a thought: Maybe we could take small steps, like making a little change, testing it, and then building up from there. Starting with zoom control sounds like a smart plan, and we can add more camera controls later on. What do you guys think? I'm totally on board. Can't wait to dive into this!

Thanks a ton! Excitedly waiting for what's to come.

[rmackay9]

This item is included in 4.4 so I'm going to close this. Of course, it's important that we get that same code into master so that it's not lost but I think @Davidsastresas has this on this project plan.

[Davidsastresas]

This item is included in 4.4 so I'm going to close this. Of course, it's important that we get that same code into master so that it's not lost but I think @Davidsastresas has this on this project plan.

Indeed, I expect that to get done within the next 2-3 weeks!