OpenSSL Issue When Running Azure Speech (TTS) in GKE Container

[AlanLiu96]

(X posted from https://learn.microsoft.com/en-us/answers/questions/1599303/openssl-issue-when-running-azure-speech-(tts)-in-g?comment=question#newest-question-comment)

Hey folks, I'm attempting to run an API call to Azure Speech (TTS), however, I seem to be having connection issues with OpenSSL cert verification when I attempt to make the connection. I've followed all of the instructions in the "Installing Speech SDK" page and the "Configuring Linux for Speech" page with more details below.

The relevant snippet of logs included here:

[508749]: 5933081ms SPX_TRACE_SCOPE_EXIT:  uws_web_socket.cpp:149 Open
[508749]: 5933081ms SPX_TRACE_INFO:  usp_connection.cpp:762 Create requestId  for messageType 0
[508749]: 5933155ms SPX_TRACE_ERROR: AZ_LOG_ERROR:  tlsio_openssl.c:691 error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
[508749]: 5933155ms SPX_TRACE_ERROR: AZ_LOG_ERROR:  tlsio_openssl.c:2441 FORCE-Closing tlsio instance.
[508749]: 5933155ms SPX_TRACE_SCOPE_ENTER:  uws_web_socket.cpp:247 OnWebSocketOpened
[508749]: 5933155ms SPX_TRACE_ERROR:  web_socket.cpp:907 WS open operation failed with result=1(WS_OPEN_ERROR_UNDERLYING_IO_OPEN_FAILED), code=2550[0x000009f6], time=2024-02-27T02:48:42.3471745Z

Setup script:

# OpenSSL 1.1.1 since 3.0 isn't supported
wget -O - https://www.openssl.org/source/openssl-1.1.1u.tar.gz | tar zxf -
cd openssl-1.1.1u
./config --prefix=/usr/local
make -j $(nproc)
make install_sw install_ssldirs
ldconfig -v 
openssl version -d # Confirmed that /etc/ssl/certs contains a whole list of certs (~600 pem files), there's a ca_certs.crt in that file as well
export SSL_CERT_DIR=/etc/ssl/certs

# Additional package upgrades
apt-get update
apt-get install build-essential libssl-dev ca-certificates libasound2 wget

Since this may be OS Specific:

# cat /etc/issue
Debian GNU/Linux 12 

This is running in a container in GKE in an autopilot cluster (with their Debian based custom OS)

What I've tried: Setting SSL_CERT_FILE to ca_certs.crt in the respective directory Verifying that the cert dir actually contains the list of certificates Disabling CRL (which doesn't seem to affect this step) Confirming that I can access microsoft speech endpoints from the container Confirming the same commit works as expected outside the container on an Ubuntu VM Setting OPENSSL_DISABLE_CRL_CHECK, OPENSSL_SINGLE_TRUSTED_CERT

Would appreciate any help to debug, thanks!

[pankopon]

@AlanLiu96 Hi, try and run the following:

On the host OS

docker pull debian:12
docker run -it debian:12 bash

In the container (as root by default)

cat /etc/debian_version # should be 12.5
apt-get update
apt-get install --yes build-essential ca-certificates git libasound2 wget
cd
wget -O - https://www.openssl.org/source/old/1.1.1/openssl-1.1.1w.tar.gz | tar zxf -
cd openssl-1.1.1w
./config --prefix=/usr/local
make -j $(nproc)
make install_sw install_ssldirs
ldconfig -v
export SSL_CERT_DIR=/etc/ssl/certs
cd ..
git clone https://github.com/Azure-Samples/cognitive-services-speech-sdk
cd cognitive-services-speech-sdk/quickstart/cpp/linux/text-to-speech/
export SPEECHSDK_ROOT="$PWD/SpeechSDK"
mkdir -p "$SPEECHSDK_ROOT"
wget -O SpeechSDK-Linux.tar.gz https://aka.ms/csspeech/linuxbinary
tar --strip 1 -xzf SpeechSDK-Linux.tar.gz -C "$SPEECHSDK_ROOT"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SPEECHSDK_ROOT/lib/x64"
sed -i 's#/change/to/point/to/extracted/##' Makefile
sed -i 's/YourSubscriptionKey/PUT-YOUR-KEY-HERE/' helloworld.cpp
sed -i 's/YourServiceRegion/PUT-YOUR-REGION-HERE/' helloworld.cpp
make
./helloworld

Example input and output:

> testing
Speech synthesized to speaker for text [testing]
Press enter to exit...

I just did this and it works as shown. So if your container is indeed Debian 12 based then similar steps should apply. Note that if you build a custom container, environment variables won't stick unless you set them with ENV in the dockerfile.

[pankopon]

To add, the only strings you need to replace in the example above are PUT-YOUR-KEY-HERE and PUT-YOUR-REGION-HERE - do not change anything else. Also be sure to verify that each command runs successfully without errors before you continue with the next.

[AlanLiu96]

Thanks for the example @pankopon, I can confirm that the example works on the debian container. I see the corresponding usage in the Azure Speech Metrics when I run the hello-world. I also ran the hello-world on my custom container and it had the same effect.

My understanding is that this should mean there's nothing "incorrect" with the cluster setup.

That said, I'm still at a loss on how to further debug the cert issues in the application. Per your recommendation, I attempted shifting the export SSL_CERT_DIR=/etc/ssl/certs to ENV SSL_CERT_DIR=/etc/ssl/certs in the Dockerfile. The same error however still occurs.

[MikeAlhayek]

I am having similar issue. My app is running on Azure Web App. The Dockerfile looks like this

FROM mcr.microsoft.com/dotnet/sdk:8.0-jammy-amd64 AS build-env
LABEL stage=build-env
WORKDIR /app

# Copy and build
COPY ./src /app
COPY NuGet.Config /app/

RUN dotnet publish /app/ProjectName.Web -c Release -o ./build/release --framework net8.0

# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:8.0-jammy-amd64 AS aspnet

# START - Install prerequisites for Azure Speech Services
# See https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/quickstarts/setup-platform
RUN apt-get update \ 
    && apt-get -y install build-essential libssl-dev libasound2 wget

# Next two lines are needed in order to be able to run on .NET 8.0
# until this issue is fixed by Microsoft: https://github.com/Azure-Samples/cognitive-services-speech-sdk/issues/2204
# These lines will install OpenSSL 1.1.1 which is needed by the Speech SDK.
RUN wget -O - https://www.openssl.org/source/openssl-1.1.1u.tar.gz | tar zxf -
RUN cd openssl-1.1.1u \
    && ./config --prefix=/usr/local \
    && make -j $(nproc) \
    && make install_sw install_ssldirs \
    && export SSL_CERT_DIR=/etc/ssl/certs \
    && export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

RUN apt-get update \ 
    && apt-get -y install ca-certificates && update-ca-certificates

# END - Install prerequisites for Azure Speech Services

RUN apt update \
    && apt install -y --no-install-recommends openssh-server \
    && mkdir -p /run/sshd \
    && echo "root:Docker!" | chpasswd

COPY sshd_config /etc/ssh/sshd_config

RUN apt-get clean

EXPOSE 80 2222
ENV ASPNETCORE_URLS http://+:80
ENV LD_LIBRARY_PATH "/usr/local/lib:${LD_LIBRARY_PATH}"
ENV SSL_CERT_DIR "/etc/ssl/certs"

WORKDIR /app
COPY --from=build-env /app/build/release .

ENTRYPOINT ["/bin/bash", "-c", "/usr/sbin/sshd && dotnet ProjectName.Web.dll"]

The code works fine locally on a windows PC using Visual Studio. It's like the wss request from the asp.net 8 app to the Speech services is failing.

The only error that I get is this

Failed to speak an incoming text. Reason: Error. Error code: ConnectionFailure. Error details: Connection failed (no connection to the remote host). Internal error: 1. Error details: Failed with error: WS_OPEN_ERROR_UNDERLYING_IO_OPEN_FAILED

[pankopon]

@AlanLiu96 @MikeAlhayek Try the following.

Dockerfile (minimal, not optimized):

FROM debian:12

RUN apt-get update --quiet || true && \
    apt-get install --quiet --yes build-essential ca-certificates libasound2 libssl-dev wget

RUN export DEBIAN_FRONTEND=noninteractive && \
    OPENSSL1=openssl-1.1.1w && \
    wget -q -O - https://www.openssl.org/source/old/1.1.1/$OPENSSL1.tar.gz | tar zxf - && \
    cd $OPENSSL1 && \
    ./config --prefix=/usr/local && \
    make -j $(nproc) && \
    make install_sw install_ssldirs && \
    ldconfig -v && \
    cd .. && \
    rm -rf $OPENSSL1

ENV SSL_CERT_DIR /etc/ssl/certs

Build:

docker build -t test:1 - < Dockerfile

Now you have a Debian 12 based container ready for using the Speech SDK with.

Test:

docker run -it test:1 bash

cd
apt-get install --yes git
git clone https://github.com/Azure-Samples/cognitive-services-speech-sdk
cd cognitive-services-speech-sdk/quickstart/cpp/linux/text-to-speech/
export SPEECHSDK_ROOT="$PWD/SpeechSDK"
mkdir -p "$SPEECHSDK_ROOT"
wget -O SpeechSDK-Linux.tar.gz https://aka.ms/csspeech/linuxbinary
tar --strip 1 -xzf SpeechSDK-Linux.tar.gz -C "$SPEECHSDK_ROOT"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SPEECHSDK_ROOT/lib/x64"
sed -i 's#/change/to/point/to/extracted/##' Makefile
sed -i 's/YourSubscriptionKey/PUT-YOUR-KEY-HERE/' helloworld.cpp
sed -i 's/YourServiceRegion/PUT-YOUR-REGION-HERE/' helloworld.cpp
make
./helloworld

Example input/output:

Type some text that you want to speak...
> testing
Speech synthesized to speaker for text [testing]
Press enter to exit...

If you, too, can run this successfully on some host then I think we can agree that the container itself can be configured correctly. Then any issues with connectivity on a specific host would have to be due to factors outside the container, e.g. firewall, proxy...

[MikeAlhayek]

@pankopon I am using Azure Web App host. Not sure how it is possible to execute the instructions you provided from there.

Thoughts?

[pankopon]

@MikeAlhayek First test the container locally.

[MikeAlhayek]

@pankopon

[AlanLiu96]

On my end, I'm still not clear where the issue lies. Given that I was able to run the provided quickstart test code in python (which is what my app is written in), this might be out of the immediate scope of getting the quickstart running.

I was able to exec into the same container on the same cluster as the one my application was running and was able to get the quickstart running after downloading the repo. Also verified that ldconfig, SSL_CERT_DIR and the azure credentials were successfully set as env vars in the application code. This seems to imply that there is no connectivity issue between this particular container and the speech endpoints (no firewall or egress issues).

Since the application code is the same as what I'm running on a VM where it runs successfully, this also implies that there isn't anything apparent in the application code that is preventing the connection.

This overall just leaves me quite puzzled as to where the certificate verification issue is being caused from:

SSL routines:tls_process_server_certificate:certificate verify failed

Though I recognize this may be beyond the scope of quickstart, I would still appreciate any help or additional context you may have @pankopon on the OpenSSL part of this issue.

[MikeAlhayek]

@pankopon when I run the container locally everything working as expected.

However, when I deploy my docker image on Azure Web App, then try to execute the following script

cd
apt-get install --yes git
git clone https://github.com/Azure-Samples/cognitive-services-speech-sdk
cd cognitive-services-speech-sdk/quickstart/cpp/linux/text-to-speech/
export SPEECHSDK_ROOT="$PWD/SpeechSDK"
mkdir -p "$SPEECHSDK_ROOT"
wget -O SpeechSDK-Linux.tar.gz https://aka.ms/csspeech/linuxbinary
tar --strip 1 -xzf SpeechSDK-Linux.tar.gz -C "$SPEECHSDK_ROOT"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SPEECHSDK_ROOT/lib/x64"
sed -i 's#/change/to/point/to/extracted/##' Makefile
sed -i 's/YourSubscriptionKey/PUT-YOUR-KEY-HERE/' helloworld.cpp
sed -i 's/YourServiceRegion/PUT-YOUR-REGION-HERE/' helloworld.cpp
make
./helloworld

I get the error

Maybe something is blocking the wss connection on Azure Web app that I can't figure out.

Here is the latest Dockerfile I used which copies similar to what you have done. The main difference is that I use mcr.microsoft.com/dotnet/sdk:8.0-jammy-amd64 image which pulls Ubuntu 20.04 instead of Debian 12.

FROM mcr.microsoft.com/dotnet/sdk:8.0-jammy-amd64 AS aspnet-sdk
LABEL stage=aspnet-sdk
WORKDIR /app

# Copy and build
COPY ./src /app
COPY NuGet.Config /app/

RUN dotnet publish /app/ProjectName.Web -c Release -o ./build/release --framework net8.0

# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:8.0-jammy-amd64 AS aspnet

# START - Install prerequisites for Azure Speech Services
# See https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/quickstarts/setup-platform

RUN apt-get update --quiet || true && \
    apt-get install --quiet --yes build-essential ca-certificates libasound2 libssl-dev wget

# The following line is needed in order to be able to run on .NET 8.0
# until this issue is fixed by Microsoft: https://github.com/Azure-Samples/cognitive-services-speech-sdk/issues/2204
# These lines will install OpenSSL 1.1.1 which is needed by the Speech SDK.
RUN OPENSSL1=openssl-1.1.1w && \
    wget -q -O - https://www.openssl.org/source/old/1.1.1/$OPENSSL1.tar.gz | tar zxf - && \
    cd $OPENSSL1 && \
    ./config --prefix=/usr/local && \
    make -j $(nproc) && \
    make install_sw install_ssldirs && \
    ldconfig -v && \
    cd .. && \
    rm -rf $OPENSSL1

# END - Install prerequisites for Azure Speech Services

RUN apt update \
    && apt install -y --no-install-recommends openssh-server \
    && mkdir -p /run/sshd \
    && echo "root:Docker!" | chpasswd

COPY sshd_config /etc/ssh/sshd_config

EXPOSE 80 2222
ENV ASPNETCORE_URLS http://+:80
ENV SSL_CERT_DIR /etc/ssl/certs

WORKDIR /app
COPY --from=aspnet-sdk /app/build/release .

ENTRYPOINT ["/bin/bash", "-c", "/usr/sbin/sshd && dotnet ProjectName.Web.dll"]

RUN apt-get clean

[MikeAlhayek]

After hours of trial and error, I managed to fix the problem by removing Microsoft.CognitiveServices from the service endpoints. So let the outbound connection to the Cognitive Services to flow using public traffic instead of the virtual network.

[pankopon]

@AlanLiu96 This seems a bit confusing. You wrote earlier

I can confirm that the example works on the debian container. (...) I also ran the hello-world on my custom container and it had the same effect.

and

I was able to exec into the same container on the same cluster as the one my application was running and was able to get the quickstart running after downloading the repo.

Did I understand correctly that you could successfully run the TTS quickstart in both the minimal example Debian 12 container and your custom container, even on the cluster where it's deployed? And your custom container is Debian 12 based, too?

[AlanLiu96]

That’s correct.

[pankopon]

@AlanLiu96 Then is the issue that a custom application of yours that uses the Speech SDK, fails to connect to speech services while running in a custom container - even though a simple TTS quickstart works, in the same custom container? If yes, it sounds like the problem is not with the container or outside, but the application setup.

Your application is written in Python, correct? Can you describe how it's been installed in the container, and how is it launched?

[pankopon]

Closing the item since no further information provided; if quickstarts/samples run successfully in the container but a custom application doesn't then it sounds like an application specific setup issue.

[AlanLiu96]

Following up, I resolved the issue by further downgrading openssl from

wget -O - https://www.openssl.org/source/openssl-1.1.1u.tar.gz | tar zxf - cd openssl-1.1.1u

to

wget -O - https://www.openssl.org/source/openssl-1.1.1k.tar.gz | tar zxf - cd openssl-1.1.1k

No clue why this worked.

[Baklap4]

Linking issue: https://github.com/Azure-Samples/cognitive-services-speech-sdk/issues/2048 since all other issues are closed.