search

rustdesk / rustdesk

An open-source remote desktop application designed for self-hosting, as an alternative to TeamViewer.
https://rustdesk.com
GNU Affero General Public License v3.0
76.82k stars 10.35k forks source link

Windows 10 client/server hwcodec does not work #5192

Closed woodholly closed 1 year ago

woodholly commented 1 year ago

Bug Description

Client: Windows 10, nvidia gpu Server: Windows 10, intel gpu + nvidia gpu Whatever I do, hardware acceleration fails. I tried both portable and install mode.

Client check-hwcodec-config/rustdesk_rCURRENT.log:

[2023-07-29 14:31:03.231537 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\encode.rs:231] prepare yuv 680.2µs
[2023-07-29 14:31:03.242169 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\encode.rs:253] hevc_qsv new failed 8.6589ms
[2023-07-29 14:31:03.393466 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\encode.rs:244] hevc_nvenc new 159.9744ms
[2023-07-29 14:31:03.395566 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\encode.rs:253] h264_qsv new failed 162.0817ms
[2023-07-29 14:31:03.395633 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\encode.rs:253] hevc_amf new failed 162.1505ms
[2023-07-29 14:31:03.397204 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\encode.rs:247] hevc_nvenc encode 3.7021ms
[2023-07-29 14:31:03.475536 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\encode.rs:244] h264_nvenc new 242.0505ms
[2023-07-29 14:31:03.475597 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\encode.rs:253] h264_amf new failed 242.1161ms
[2023-07-29 14:31:03.477612 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\encode.rs:247] h264_nvenc encode 2.0137ms
[2023-07-29 14:31:03.497012 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:hevc_qsv device:AV_HWDEVICE_TYPE_NONE new:376.3µs
[2023-07-29 14:31:03.497341 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:h264_qsv device:AV_HWDEVICE_TYPE_NONE new:519.8µs
[2023-07-29 14:31:03.498382 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:hevc device:AV_HWDEVICE_TYPE_NONE new:965.6µs
[2023-07-29 14:31:03.498541 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:h264 device:AV_HWDEVICE_TYPE_NONE new:1.9444ms
[2023-07-29 14:31:03.501489 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:345] name:hevc_qsv device:AV_HWDEVICE_TYPE_NONE decode failed:4.4ms
[2023-07-29 14:31:03.503630 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:345] name:h264_qsv device:AV_HWDEVICE_TYPE_NONE decode failed:6.2127ms
[2023-07-29 14:31:03.507975 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:337] name:h264 device:AV_HWDEVICE_TYPE_NONE decode:9.3655ms
[2023-07-29 14:31:03.514714 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:337] name:hevc device:AV_HWDEVICE_TYPE_NONE decode:16.2499ms
[2023-07-29 14:31:03.582830 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:hevc device:AV_HWDEVICE_TYPE_CUDA new:86.0512ms
[2023-07-29 14:31:03.582937 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:h264 device:AV_HWDEVICE_TYPE_CUDA new:86.4092ms
[2023-07-29 14:31:03.645500 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:h264 device:AV_HWDEVICE_TYPE_D3D11VA new:148.7573ms
[2023-07-29 14:31:03.671792 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:h264 device:AV_HWDEVICE_TYPE_DXVA2 new:175.1199ms
[2023-07-29 14:31:03.675608 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:337] name:hevc device:AV_HWDEVICE_TYPE_CUDA decode:92.6931ms
[2023-07-29 14:31:03.677904 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:337] name:h264 device:AV_HWDEVICE_TYPE_CUDA decode:94.8986ms
[2023-07-29 14:31:03.678650 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:hevc device:AV_HWDEVICE_TYPE_D3D11VA new:181.7266ms
[2023-07-29 14:31:03.741300 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:325] name:hevc device:AV_HWDEVICE_TYPE_DXVA2 new:244.4238ms
[2023-07-29 14:31:03.759116 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:353] name:hevc_qsv device:AV_HWDEVICE_TYPE_QSV new failed:262.1617ms
[2023-07-29 14:31:03.782065 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:337] name:h264 device:AV_HWDEVICE_TYPE_D3D11VA decode:136.5385ms
[2023-07-29 14:31:03.832128 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:337] name:h264 device:AV_HWDEVICE_TYPE_DXVA2 decode:160.2553ms
[2023-07-29 14:31:03.833428 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:337] name:hevc device:AV_HWDEVICE_TYPE_D3D11VA decode:154.7603ms
[2023-07-29 14:31:03.855065 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:337] name:hevc device:AV_HWDEVICE_TYPE_DXVA2 decode:113.673ms
[2023-07-29 14:31:03.870114 +03:00] DEBUG [C:\Users\runneradmin\.cargo\git\checkouts\hwcodec-8b4454f7f5e13466\3ea7986\src\decode.rs:353] name:h264_qsv device:AV_HWDEVICE_TYPE_QSV new failed:373.5532ms
[2023-07-29 14:31:03.872298 +03:00] DEBUG [libs\hbb_common\src\config.rs:422] Configuration path: C:\Users\user\AppData\Roaming\RustDesk\config\RustDesk_hwcodec.toml
[2023-07-29 14:31:03.872354 +03:00] ERROR [libs\hbb_common\src\config.rs:405] Failed to load config: Failed to load configuration file
[2023-07-29 14:31:03.872540 +03:00] DEBUG [libs\hbb_common\src\config.rs:422] Configuration path: C:\Users\user\AppData\Roaming\RustDesk\config\RustDesk_hwcodec.toml
[2023-07-29 14:31:03.872576 +03:00] ERROR [libs\hbb_common\src\config.rs:405] Failed to load config: Failed to load configuration file

Last lines report that it cannot load config file, but C:\Users\user\AppData\Roaming\RustDesk\config\RustDesk_hwcodec.toml exists and can be opened, how to debug this error?

RustDesk_hwcodec.toml:

[options]
bestHwEncoders = '''
{
  "h264": {
    "name": "h264_nvenc",
    "format": "H264",
    "vendor": "NVIDIA",
    "score": 92,
    "hwdevice": "AV_HWDEVICE_TYPE_NONE"
  },
  "h265": {
    "name": "hevc_nvenc",
    "format": "H265",
    "vendor": "NVIDIA",
    "score": 94,
    "hwdevice": "AV_HWDEVICE_TYPE_NONE"
  }
}'''
bestHwDecoders = '''
{
  "h264": {
    "name": "h264",
    "format": "H264",
    "vendor": "OTHER",
    "score": 94,
    "hwdevice": "AV_HWDEVICE_TYPE_CUDA"
  },
  "h265": {
    "name": "hevc",
    "format": "H265",
    "vendor": "OTHER",
    "score": 95,
    "hwdevice": "AV_HWDEVICE_TYPE_CUDA"
  }
}'''

Client general log has "PermissionDenied" message, but no reason for that:

[2023-07-29 14:31:04.110880 +03:00] DEBUG [libs\hbb_common\src\config.rs:422] Configuration path: C:\Users\user\AppData\Roaming\RustDesk\config\RustDesk_hwcodec.toml
[2023-07-29 14:31:04.110987 +03:00] DEBUG [libs\hbb_common\src\config.rs:422] Configuration path: C:\Users\user\AppData\Roaming\RustDesk\config\RustDesk_hwcodec.toml
[2023-07-29 14:31:04.111047 +03:00] DEBUG [libs\hbb_common\src\config.rs:1360] HW_CODEC_CONFIG refreshed successfully
[2023-07-29 14:31:04.111063 +03:00] DEBUG [libs\scrap\src\common\hwcodec.rs:359] Os { code: 5, kind: PermissionDenied, message: "Отказано в доступе." }, scrap::common::hwcodec:libs\scrap\src\common\hwcodec.rs:359
:21
[2023-07-29 14:31:04.142280 +03:00] INFO [libs\scrap\src\common\hwcodec.rs:362] Check hwcodec config, exit with: exit code: 0
[2023-07-29 14:31:04.142307 +03:00] DEBUG [libs\hbb_common\src\config.rs:422] Configuration path: C:\Users\user\AppData\Roaming\RustDesk\config\RustDesk_hwcodec.toml
[2023-07-29 14:31:04.142393 +03:00] DEBUG [libs\hbb_common\src\config.rs:1360] HW_CODEC_CONFIG refreshed successfully

I have tested hardware acceleration coder/decoder on this machine using other software and it works without issues. How to debug this ?

How to Reproduce

n/a

Expected Behavior

n/a

Operating system(s) on local side and remote side

Windows 10 -> Windows 10

RustDesk Version(s) on local side and remote side

1.2.1 -> 1.2.1

Screenshots

n/a

Additional Context

No response

21pages commented 1 year ago

If you can debug, do some log in these two functions. https://github.com/rustdesk/rustdesk/blob/3c4852a254d6cad25cc72b5b113e4c53ab0f301a/libs/scrap/src/common/hwcodec.rs#L328 https://github.com/rustdesk/rustdesk/blob/3c4852a254d6cad25cc72b5b113e4c53ab0f301a/libs/scrap/src/common/hwcodec.rs#L365

rustdesk commented 1 year ago

Failed to load config: Failed to load configuration file

@21pages remove this message which causes a lot of confusion. If needed, modify the crate.

woodholly commented 1 year ago

@rustdesk lol, this is not "absent config" problem

21pages commented 1 year ago

Does the H264/H265 option appear on the remote toolbar codec menu? 1690709750531

woodholly commented 1 year ago

There are no H264/H265 options at remote (server), same at client. No idea why H264/H265 absent, Nvidia GPU works fine in all other applications both on the client and server. (I tested coding/decoding with https://github.com/HandBrake)

Yesterday I tried Ubuntu as a client on this PC and H264/H265 options were available on the client, but connection to windows 10 remote was not hardware accelerated, probably because server does not support H264/H265.

21pages commented 1 year ago

PermissionDenied

This may be the reason. The content of RustDesk_hwcodec.toml also shows support. Could you try this? https://github.com/21pages/rustdesk/releases/download/nightly/rustdesk-1.2.2-x86_64_no_global_var.exe

woodholly commented 1 year ago

PermissionDenied

This may be the reason. The content of RustDesk_hwcodec.toml also shows support. Could you try this? https://github.com/21pages/rustdesk/releases/download/nightly/rustdesk-1.2.2-x86_64_no_global_var.exe

With this file H264/H265 options appeared and hardware acceleration finally works!

rustdesk commented 1 year ago

https://github.com/rustdesk/rustdesk/pull/5197 fixed this? @21pages

21pages commented 1 year ago

I don't know the actual reason, maybe removing global var helps, the exe is compiled by that pr.