PTZOptics PT20X-SE-GY-G3 looses connection at first command

[jschaeberle81]

We have a new PTZOptics PT20X-SE-GY-G3 camera that loses connection the first time it receives any command from the Stream Deck. It executes the command but the status on the Companion Connections screen immediately goes to "Connection Failure". If we disable and re-enable the connection, it immediately gets a green status and then again only executes one command before going off line.

[jschaeberle81]

Just looked at the logs. The messages are: error: Received return message not starting with 0x90 (VISCA bytes [A0 41 FF]) info: Return message processing completed error: Unhandled command rejection was suppressed: Error: Received return message not starting with 0x90 (VISCA bytes [A0 41 FF])

[soonerdude]

I'm getting the same thing. We have two PTZOptics PT12X-SE-WH-G3 cameras. One has been updated to the new firmware after free NDI was added (SOC v9.1.35), the other still has the latest firmware prior to the NDI firmware upgrade (SOC v9.1.27). After I updated BitFocus Companion to 3.3.1, any time we send any sort of command to the camera that has the newer firmware, I get the same error mention above about the return message not starting with 0x90.

With debugging enabled on both camera connections in Companion, below are the log results when a command is sent. For each camera, I'm sending a command to switch to Preset 1

Older firmware cam: RECV: [90 42 FF] Newer firmware cam: RECV: [A0 42 FF]

[soonerdude]

It looks like PTZOptics changed the ACK responses from "90 4y FF" to "A0 4y FF" on the Move SE after the April firmware update that added native NDI support to the cameras. I can't find any documentation on their G3 VISCA over IP Commands PDF files to back it up, but the log files for my two identical model cameras, one having the older firmware, plus the initial post here having the same "A0 4y FF" response seems to back up that assumption.

[jswalden]

Past VISCA docs specified VISCA replies as being z0 ... where z would be 0x8 (or maybe 0x9?) or higher. But the latest docs linked directly on the developer portal say specifically 90, so that's what the module assumes (not sure if I added this assumption or it predated my interest in the module). It looks like they changed this without saying they changed it. (Not too cool, IMO.)

It shouldn't be hard to allow messages that start with z0 for z from 0x8..=0xF: change the doc-comment and maybe a little bit of code inside VISCAPort.#readReturnMessages, a small amount of code in VISCAPort.#processReturnMessages, and I think that might be it. (And randomly perturbing tests to use not just 90 but a variety of values in the range of permitted z values, to ensure decent testing.)

I haven't upgraded my cameras to the new firmware (I don't use NDI now) so hadn't noticed this. It's probably a small patch, but I'm not upgrading firmware on my cameras to test it immediately before I travel for a bit starting end of week. :-) But maybe someone here can lend a hand on that, if I actually do write a patch before I'm ready to upgrade any camera firmware.

[soonerdude]

I opened a ticket with PTZOptics and this was the initial response about 25 minutes ago:

Hello John,

The ACK/Completion responses should not have changed with the firmware update.

And just to double check, the camera that was updated, was it tested and working prior to the update? Or had you never used the updated camera with Bitfocus prior to the update?

Are you getting the non-standard response for specific commands or everything you have tried so far? If it seems consistent across all commands, what sort of commands are you typically sending?

Can you let me know the serial numbers of these cameras? (please note which has been upgraded and is experiencing the issue)

Then about 15 minutes later, I received this response

Hi John,

I have a quick follow-up to my last message.

I was able to replicate the issue with an updated Move SE here. However, I was able to fix the issue by doing a Factory Reset on the camera. After the reset, I started getting the standard responses again.

Try triggering a Factory Reset on your camera. Unfortunately, this is a full reset, so the IP and any saved presets/menu options will be lost.

I just got to work so I haven't had a chance to reply, but for us, doing a factory reset isn't an option. We spent a long time getting the presets set up and then getting the color / white balance settings set and we couldn't find a way to apply all those settings to all the presets at once, it required changing each preset individually, which took half a day.

So it appears that the "90 4y FF" and "90 5y FF" are still the correct ACK/Completion responses, but there is a bug after updating the firmware to the latest, but the bug goes away if you factory reset.

For now, I've rolled back BitFocus Companion to the previous version I had downloaded back in March and it seems to be working.

[jswalden]

Interesting. Sending A0 instead of 90 doesn't have to be a bug, it's structurally compliant VISCA (to the extent VISCA has a consistent structural definition), so we could still flex things to allow it anyway here. But perhaps they can adjust things upstream to make the current module code work again.

[jschaeberle81]

I should be able to do a factory reset on Sunday. We only have 9 presets so recreating them is not an issue. Thanks for the help!

[soonerdude]

I had a recent update from them today. I won't be able to try these steps for about a week since we have VBS going this week so I'm not able to go in after work and mess around with anything. Hopefully it works so that I can avoid factory resetting. I hate for jswalden to have to make coding changes to accommodate an issue that may not exist after their next round of firmware updates. Even though the A0 is structurally compliant, it was not intended to be the ACK response by PTZOptics and somehow reverts back to 90 if you factory reset it.

Hello John,

This seems to be an odd issue; it does not affect all of the SEs.

Can you try flashing the latest firmware onto this camera again the next time you are on site with it? I was able to fix a camera that had the same issue by reflashing the latest firmware. This would not affect any of your current settings/presets. If you don't have the firmware file anymore, you can download another copy from the firmware files page.

If this doesn't work for you, please let me know.

[soonerdude]

I just applied the same firmware file to our Move SE 12x camera that I used to upgrade it a few weeks ago and am now retaining connection to that camera in Companion after sending a command to it. The log shows the ACK is indeed 90 instead of A0. This didn't require any sort of factory reset, so I was able to retain all the settings and presets.

[tonsofpcs]

I utilize the PTZOptics VISCA module for non-PTZOptics cameras using "Custom command" and they have other replies that shouldn't be handled and are instead being handled as an error, disabling the connection.

If any replies unhandled could be "not processed" like the replies in #51 (or at least not cause an error to disable the connection) I believe it would resolve this issue.

Example:

info: SEND: [81 01 06 05 FF]...
info: RECV: [90 50 FF]
info: Completion in socket 0
error: Received Completion for socket 0, but no command is executing in it (VISCA bytes [90 50 FF])```

[jswalden]

If any replies unhandled could be "not processed" like the replies in #51 (or at least not cause an error to disable the connection) I believe it would resolve this issue.

Example:

info: SEND: [81 01 06 05 FF]...
info: RECV: [90 50 FF]
info: Completion in socket 0
error: Received Completion for socket 0, but no command is executing in it (VISCA bytes [90 50 FF])```

That only works in #51 because that particular reply sequence is not used in any other PTZOptics camera replies. It thus can safely be filtered out without affecting PTZOptics cameras.

Unfortunately, if your example fully represents what your camera is doing, you're sending a command to your camera, and then your camera is responding not with a standard ACK (90 4y FF) followed by a Completion (90 5y FF), but instead only with a Completion. But Completion messages are in fact used/expected for every PTZOptics command, and to simply discard a Completion risks leaving some already-sent command in permanently-acknowledged, never-completed state. There isn't really a way to filter out unexpected Completion messages, without the module's understanding of the camera protocol risking falling into decoherence, with the module thinking the camera is in some known state when really it's in a different one (potentially risking subsequent commands' responses being completely misinterpreted).

That said, if that specific byte sequence really is an oddball in PTZOptics cameras as well -- docs say it's a Pan Tilt: Pan Tilt Drive: Reset command -- then it would be reasonable to add that handling. I don't have access to my cameras to test that right now (and am really only slowly digging out of a few weeks' email backlog that accrued during a recent vacation mostly without usable Internet).