uibc event not reaching

[lati-urr]

Hello I use miraclecast for mirroring Android to raspberry pi 3 running Ubuntu. It works great! Raspberry pi mirrors Android screen.

I run miracle-sinkctl --uibc, but Android isn't responsed touch or keyborad event. Here is log files. log.zip According to journal.log, it seems that raspberry pi sends UIBC event. journal.log

Jun 03 10:27:23 ubuntu miracle-sinkctl-gst[3896]: INFO: getUIBCGenericKeyPacket (3,0xFF08,0x0000
Jun 03 10:27:23 ubuntu miracle-sinkctl-gst[3896]: )
Jun 03 10:27:23 ubuntu miracle-sinkctl-gst[3896]: INFO: getUIBCGenericKeyPacket splitedStr tokens=[3]
Jun 03 10:27:23 ubuntu miracle-sinkctl-gst[3896]: INFO: getUIBCGenericKeyPacket typeId=[3]
Jun 03 10:27:23 ubuntu miracle-sinkctl-gst[3896]: INFO: getUIBCGenericKeyPacket splitedStr tokens=[0xFF08]
Jun 03 10:27:23 ubuntu miracle-sinkctl-gst[3896]: INFO: getUIBCGenericKeyPacket key code 1=[65288]
Jun 03 10:27:23 ubuntu miracle-sinkctl-gst[3896]: INFO: getUIBCGenericKeyPacket splitedStr tokens=[0x0000
Jun 03 10:27:23 ubuntu miracle-sinkctl-gst[3896]: ]
Jun 03 10:27:23 ubuntu miracle-sinkctl-gst[3896]: INFO: getUIBCGenericKeyPacket key code 2=[0]

But my Android device is not responsed. It supports UIBC, I confirm that with windows projection function.

Any ideas?

Devices source Huawai Mate 9(Android 9) or Xiaomi Pad 5(Android 11)

sink Ubuntu 22.04 LTS

[albfan]

Looks a problem in viewer emiiting messages:

Jun 03 10:27:49 ubuntu miracle-sinkctl-gst[3895]: Traceback (most recent call last):
Jun 03 10:27:49 ubuntu miracle-sinkctl-gst[3895]:   File "/usr/bin/gstplayer", line 161, in on_mouse_pressed
Jun 03 10:27:49 ubuntu miracle-sinkctl-gst[3895]:     print('{0},1,0,{1},{2}'.format(type, uibc_x , uibc_y))

Jun 03 10:28:15 ubuntu miracle-sinkctl-gst[3895]: Traceback (most recent call last):
Jun 03 10:28:15 ubuntu miracle-sinkctl-gst[3895]:   File "/usr/bin/gstplayer", line 164, in on_key_pressed
Jun 03 10:28:15 ubuntu miracle-sinkctl-gst[3895]:     print("3,0x%04X,0x0000" % event.keyval)

but that is really device specific.

I suggest to check if miracle-uibcctl is running

maybe modifying uibc-viewer with

gstplayer $@ | tee uibc.log | miracle-uibcctl $IP $UIBC_PORT --daemon &

to see why this messages do not reach the source

in any case you can rerun the miracle-uibcctl with that IP and port to send test mouse events

[lati-urr]

I modified uibc-viewer with gstplayer $@ 2>&1 | tee uibc.log | miracle-uibcctl $IP $UIBC_PORT --daemon & and rebuild. Here is uibc log files. uibclogs.zip

Huawai Mate 9 isn't respond. (But according to uibc-huawaimate9.log, it seems that raspberry pi sends first two key event) Xiaomi Pad 5 is respond to key board event, but isn't respond to touch event.

I set up $XDG_RUNTIME_DIR=/tmp/runtime already, but error: XDG_RUNTIME_DIR not set in the environment. happened. I don't know how to fix GStreamer-Video-CRITICAL **: 11:17:29.026: gst_video_center_rect: assertion 'src->h != 0' failed

I check connection with ss command. Here is a result.

# Huawai Mate 9
> ss -t
State Recv-Q Send-Q Local Address:Port  Peer Address:Port Process
ESTAB 0      0      192.168.49.129:57832 192.168.49.1:7236
# Xiaomi Pad 5
> ss -t
State Recv-Q Send-Q Local Address:Port  Peer Address:Port Process
ESTAB 0      0      192.168.49.90:47060 192.168.49.1:7236
ESTAB 0      0      192.168.49.90:48148 192.168.49.1:4321

When I use Huawai Mate 9, there is only 1 connection. miracle-sinkctl received screen data from port 7236, sends touch/keyboard event to port 4321, right? Perhaps, this is caused by device specific. So, I want Xiaomi Pad 5 to respond with touch event.

Thank you for supporting!

[albfan]

(gstplayer:6490): GStreamer-Video-CRITICAL **: 11:17:29.026: gst_video_center_rect: assertion 'src->h != 0' failed

https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/video/gstvideosink.c#L137

just at some point video do not have height, probably nothing to worry if you final see video.

error: XDG_RUNTIME_DIR not set in the environment.

This is due to run all code under root, something to fix for miraclecast, but not for this issue

/usr/bin/gstplayer:16: PyGIDeprecationWarning: Since version 3.11, calling threads_init is no longer needed. See: https://wiki.gnome.org/PyGObject/Threading GObject.threads_init()

Something to fix if you use recent version of gobject. Nothing to worry, just keep can avoid problems on old distros.

So as uibc-viewer pipe gstplayer output to miracle-uibcctl, doesn't make much sense to me it only sends two events.

Attaching logs for support of Huawei mate 9 we can check if there's no support for uibc or any extra parameter it needs.

There're some TODOs on miracle-uibcctl we can fix for this issue to test this correctly.

See we provide a default setup for uibc but there're more options:

res/sinkctl.protocol-extension.example
extends.wfd_uibc_capability=input_category_list=GENERIC, HIDC;generic_cap_list=Keyboard;hidc_cap_list=Keyboard/USB, Mouse/USB, MultiTouch/USB, Gesture/USB, RemoteControl/USB;port=none

src/ctl/ctl-sink.c

        /* wfd_uibc_capability */
        if (uibc_option) {
            gchar* wfd_uibc_capability = "input_category_list=GENERIC;"
                                         "generic_cap_list=Mouse,SingleTouch;"
                                         "hidc_cap_list=none;"
                                         "port=none";

generic-cap-list = inp-type *(“,” SP generic-cap-list)
inp-type = “Keyboard” / “Mouse” / “SingleTouch” / “MultiTouch” / “Joystick”/ “Camera” / “Gesture” / “RemoteControl”

hidc-cap-list = detailed-cap *(“,” SP hidc-cap-list)
detailed-cap = inp-type “/” inp-path
inp-path = “Infrared” / “USB” / “BT” / “Zigbee” / “Wi-Fi” / “No-SP”; “NoSP” means vendor specific

so you can try with extends configuration and setup more generic(like Multitouch) or hidc (I expect "/USB" suffix to just work)

See https://github.com/albfan/miraclecast/wiki/miracle-sinkctl#extend-request-protocol

At some point we can implement fine grained solution to support config related with source device.

Again logs for support on xiaomi pd 5 should explain any missconfiguration or extra parameter needed.

[lati-urr]

I edited ~/.miracastrc with

[sinkctl]
external-player=gstplayer
rstp-port=7236
extends.intel_friendly_name=miraclecast
extends.intel_sink_device_URL=http://github.com/albfan/miraclecast
extends.intel_sink_manufacturer_name=GNU Linux
extends.intel_sink_model_name=Ubuntu
extends.microsoft_diagnostics_capability=none
extends.microsoft_format_change_capability=none
extends.microsoft_latency_management_capability=none
extends.wfd_idr_request_capability=1
extends.microsoft_rtcp_capability=none
extends.microsoft_color_space_conversion=none
extends.wfd_uibc_capability=input_category_list=GENERIC, HIDC;generic_cap_list=Keyboard,SingleTouch;hidc_cap_list=Keyboard/USB,Mouse/USB,MultiTouch/USB;port=none

but uibc event do not works.

Attaching logs for support of Huawei mate 9 we can check if there's no support for uibc or any extra parameter it needs.

Which logs should I see to know extra parameter it needs? sink.log or wifid.log ? And one more question. When does ~/.miracastrc referenced? Should I rerun miracle-wifid after editng ~/.miracastrc

Thank for your help!

[albfan]

Section "4.11.2 UIBC Establishment and Maintenance" from spec (you can read on https://github.com/albfan/miraclecast/wiki/Miracast) describes how devices negotiate UIBC capability.

You're only supposed to define extends.wfd_uibc_capability to tweak this message.

In miracle-sinkctl.log we should see the response about uibc capability, maybe that device do not accept uibc at all or it only accepts HIDC

miraclecast ini file (~/.miraclecast or ~/.config/miraclecastrc, I check wiki https://github.com/albfan/miraclecast/wiki/Configuration and ~/.miraclecastrc is wrong) is readed at miracle-sinkctl start, and for this config that's the step relevant as this is under section [sinkctl]

[lati-urr]

I understand how devices negotiate UIBC, thanks! I moved config file to ~/.config/miraclecastrc and edited it as follows.

[sinkctl]
external-player=gstplayer
rstp-port=7236
# extends.intel_friendly_name=miraclecast
# extends.intel_sink_device_URL=http://github.com/albfan/miraclecast
# extends.intel_sink_manufacturer_name=GNU Linux
# extends.intel_sink_model_name=Ubuntu
# extends.microsoft_diagnostics_capability=none
# extends.microsoft_format_change_capability=none
# extends.microsoft_latency_management_capability=none
# extends.wfd_idr_request_capability=1
# extends.microsoft_rtcp_capability=none
# extends.microsoft_color_space_conversion=none
extends.wfd_uibc_capability=input_category_list=GENERIC;generic_cap_list=Keyboard,Mouse,SingleTouch,MultiTouch;hidc_cap_list=none;port=none

But according to sinkctl.log, it seems that wfd_uibc_capability parameter was not changed.

wfd_content_protection: none
wfd_video_formats: 00 00 03 10 0000001e 00000003 000000ff 00 0000 0000 10 none none
wfd_audio_codecs: AAC 00000007 00
wfd_client_rtp_ports: RTP/AVP/UDP;unicast 1991 0 mode=play
wfd_uibc_capability: input_category_list=GENERIC;generic_cap_list=Mouse,SingleTouch;hidc_cap_list=none;port=none

and source responsed

wfd_video_formats: 00 00 02 10 00000000 00000002 00000000 00 0000 0000 00 none none
wfd_audio_codecs: AAC 00000001 00
wfd_presentation_URL: rtsp://192.168.49.1/wfd1.0/streamid=0 none
wfd_client_rtp_ports: RTP/AVP/UDP;unicast 1991 0 mode=play
wfd_uibc_capability: input_category_list=GENERIC;generic_cap_list=Keyboard;hidc_cap_list=none;port=38057
wfd_uibc_setting: enable

generic_cap_list parameter is Keyboard. Does it mean only keyboard support? Here is a log file. miracle-sinkctl.log

Did I mistake miracast ini path or settings?

Thank you!

[albfan]

Well from that we see that this default config is accepted. 192.168.49.1 and port 38057 should be where uibc is waiting.

We can improve the miracle-uibcctl to disable daemon mode and allow to send some custom commands and see if that break for some reason.

cat a template (like the one you provide should allow us to see if there's any problem with that data)

cat input-data | miracle-uibcctl 192.168.49.1 38057

where file input-data is:

3,0x0061,0x0000
3,0x0073,0x0000
3,0x0073,0x0000
3,0x0073,0x0000
3,0xFFE1,0x0000
3,0x0071,0x0000
3,0x0067,0x0000
3,0x0063,0x0000
3,0x0068,0x0000
3,0x0078,0x0000
3,0x0073,0x0000
3,0x0064,0x0000
3,0x007A,0x0000
3,0x0078,0x0000
3,0x006E,0x0000
3,0x0061,0x0000
3,0x0061,0x0000
3,0x0066,0x0000
3,0x0067,0x0000
3,0x0078,0x0000
3,0x006E,0x0000
3,0x0064,0x0000
3,0x006D,0x0000
3,0x0068,0x0000
3,0x0074,0x0000
3,0x0065,0x0000
3,0x006F,0x0000
3,0x006B,0x0000
3,0x0075,0x0000
3,0x0072,0x0000
3,0x0065,0x0000
3,0x0067,0x0000
3,0x0068,0x0000
3,0x0066,0x0000
3,0x0067,0x0000
3,0x0068,0x0000
3,0x0068,0x0000
3,0x0068,0x0000
3,0x0061,0x0000
3,0x0061,0x0000
3,0x0061,0x0000
3,0x0061,0x0000
3,0x0020,0x0000

or similar.

but if that port is not open anymore we need to check how to send a common to enable again (reopen and reconnect is not managed)

[albfan]

see gtsplayer do not start miracle-uibcctl

https://github.com/albfan/miraclecast/blob/43505ab3dd3911084331f7e9bdb562d1f5047190/src/ctl/sinkctl.c#L429

comment or remove external-player too.

[lati-urr]

Sorry for late response. I would like to check uibc data with miracle-uibcctl command, but suddenly miraclecast cannot connect. I try rerun miracle-wifid, reboot, and re-install Ubuntu many times, but never works. On [miraclectl], [DISCONNECT] as soon as [CONNECT]. Here is logfiles. logfiles.zip

Recenly, is there any updates? Thank you!

[albfan]

I cannot see exactly how this happen. I will add some timestamps so we can correlate when this disconnect happens.

[albfan]

See now there's --log-date-time so all traces come with timestamp so it's easy to correlate.

We can reopen if problem still exists and you provide new logs