search

grishka / libtgvoip

VoIP library for Telegram clients
The Unlicense
389 stars 158 forks source link

audioClient->Initialize failed: HRESULT=0x80070057 #106

Closed casper closed 3 years ago

casper commented 3 years ago

I get the below message in the debug log whenever I try to make a call with Telegram on Windows:

libtgvoip v2.4.4 on Windows 6.0.6002 Service Pack 2 x86
Log started on 24/10/2021 at 2:57:07
....
10-24 02:57:07 E: audioClient->Initialize failed: HRESULT=0x80070057
10-24 02:57:07 E: audioClient->Initialize failed: HRESULT=0x80070057
10-24 02:57:07 I: AEC: 1 NS: 1 AGC: 1
10-24 02:57:07 E: Error initializing audio playback

The Telegram GUI then refuses the call and displays: "There seems to be a problem with your sound card."

In the code of libtgvoip I have found the spot which generates the error:

    const GUID guid = { 0x2c693079, 0x3f59, 0x49fd, { 0x96, 0x4f, 0x61, 0xc0, 0x5, 0xea, 0xa5, 0xd3 } };
    res = audioClient->Initialize(AUDCLNT_SHAREMODE_SHARED, AUDCLNT_STREAMFLAGS_EVENTCALLBACK | AUDCLNT_STREAMFLAGS_NOPERSIST | AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM | AUDCLNT_STREAMFLAGS_SRC_DEFAULT_QUALITY, 60 * 10000, 0, &format, &guid);
    CHECK_RES(res, "audioClient->Initialize");

I also checked that 0x80070057 means E_INVALIDARG, which is described on this page: https://docs.microsoft.com/en-us/windows/win32/api/audioclient/nf-audioclient-iaudioclient-initialize

Parameter pFormat points to an invalid format description; or the AUDCLNT_STREAMFLAGS_LOOPBACK flag is set but ShareMode is not equal to AUDCLNT_SHAREMODE_SHARED; or the AUDCLNT_STREAMFLAGS_CROSSPROCESS flag is set but ShareMode is equal to AUDCLNT_SHAREMODE_EXCLUSIVE. A prior call to SetClientProperties was made with an invalid category for audio/render streams.

But that's as far as I got. I tried reinstalling and fiddling with my audio device settings in many different ways, but nothing seems to help.

Recording audio messages in Telegram works fine, as well as audio playback on videos. No problems in any other apps either. It's only when trying to make VOIP calls that it fails.

Any clue what could be the problem?

grishka commented 3 years ago

This library is no longer maintained since Pavel fired me in April 2019.

grishka commented 3 years ago

As for the audio from the Telegram app itself, it relies on a cross-platform library for that, and iirc on Windows it always uses DirectSound, which I've never implemented here.

casper commented 3 years ago

Well that's a bummer :(

Is there any way I could debug this issue further for myself? Do you have any idea what the issue might be?

Maybe I could just pluck the init code from AudioOutputWASAPI.cpp into a dummy project and see if I can debug it in Visual Studio? Do you think that would work, or are there too many dependencies for such a simple test? I don't want to start building the whole Telegram app from scratch...

grishka commented 3 years ago

Wait.

Windows 6.0.6002

Is this Vista?..

It looks like it is, according to https://docs.microsoft.com/en-us/windows/win32/sysinfo/operating-system-version

casper commented 3 years ago

Yes. Vista!

grishka commented 3 years ago

Lol. Yeah well that's your problem right here :D

I only ever tested on XP (with waveOut API), 7, and 10. I just assumed no one used vista any more because 7 is better in about every way. Might be an issue with how it implements WASAPI as it was the first Windows version to implement it.

grishka commented 3 years ago

But — try setting the sound card to 48000 hz 16 bit, maybe it doesn't support sample rate/format conversion. And remove AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM.

casper commented 3 years ago

I know but I love my Vista :-/ According to the MS docs there should not be any Win7 specific options in your Initialize code.

Is Telegram always compiled with the TGVOIP_WINXP_COMPAT flag? Maybe I could binary patch the exe to run the WINXP audio path, as you seem to check for it in only one location. I just need to patch one number in the code to switch the path I think.

Yeah I tried the 48kHz trick. I thought that was the issue also. No go. I also tried the exclusive non-exclusive settings. Very strange as Vista audio should be almost identical to Win 7 as far as I know.

grishka commented 3 years ago

Is Telegram always compiled with the TGVOIP_WINXP_COMPAT flag?

I have no idea, and the dev does like to change the build script fairly often. It may be, because he didn't touch the parts that I wrote. It may be not, because it dropped XP support 2-3 years ago.

Maybe I could binary patch the exe to run the WINXP audio path, as you seem to check for it in only one location.

You can do that. You can also hook GetVersion() and pretend that you're running XP. Oh, and actually, maybe that "compatibility mode" thingy in the properties window might do the trick. So do try that first.

casper commented 3 years ago

Oh yeah. Great idea with the compatibility! Let's see....

casper commented 3 years ago

Holy crap. It works! Lol. But the audio output is very choppy. Have to troubleshoot that a little bit.

grishka commented 3 years ago

Nice. I didn't really expect it to work.

casper commented 3 years ago

Yeah the microphone input is smooth, but the speaker output is choppy. Tried sample rates and exclusive settings. Dooh...sooo close.

Maybe I have to make a test harness for your code to figure out which of the Init flags fails on Vista in the WASAPI code. Because if it's just a flag maybe I could patch the exe that way too. But getting late. Have to continue tomorrow...

casper commented 3 years ago

Ok I didn't go to bed but spent the whole night building a test harness for the WASAPI code. Finally got it to compile and run.

Note I have no audio to test with. I only check if the Initialize call goes through or not.

Conclusion: it's the last two flags to Initialize that are incompatible with Vista.

AUTOCONVERTPCM + SRC_DEFAULT_QUALITY -> fail.
AUTOCONVERTPCM alone seems to initialize SRC_DEFAULT_QUALITY only -> fail.

It's strange that MSDN says they should work, but apparently they don't. On Visual Studio 2010 that I use they were not even defined, had to enter them manually.

I guess I could patch the binary to remove those two, but I wonder how it will affect audio quality.

Now going to bed for real.

casper commented 3 years ago

Good news. Patching the binary works! :) No problem with sound quality as far as I can tell.

It took a while to figure out how to do it, but in the end I searched the binary for the combined audio property flags which is 0x880c0000. There are only two spots in the binary with that sequence.

I then patched the last byte 88 to 00, which makes the total flag equal to 0x000c0000, which means we removed the last two Windows 7 flags that were causing the problem.