Action stuck indefinitely on deploy

[tinuoommen]

While deploying through the azure function deploy action - https://github.com/marketplace/actions/azure-functions-action - the action is stuck indefinitely. It is able to establish connectivity to the Function App as you can see from the below screenshot, but then the action is stuck until I finally have to cancel it.

Below is what little I could find on the internet for this issue: o https://github.com/Azure/actions-workflow-samples/issues/15#issuecomment-612410143 – this talks about a previously known issue in the action that prevents the function app from spinning up after the deployment. I tried the steps mentioned i.e. adding the appsettings explicitly but had no luck. o One of the contributors asked to deploy the function app using the web app deploy action • I also tried deploying using the web app deploy action, but surprisingly faced the same problem – the deploy action is stuck indefinitely. • I have also tried changing the function deploy action to action v1.3.1 as mentioned here - https://github.com/Azure/functions-action/issues/86, but this also produced the same result.

Please note that I am using a self-hosted runner.

Thanks!

[patelchandni]

@tinuoommen Thanks for reporting the issue. Is this a Linux or Windows function app?

[pjung-bosch]

I am experiencing the same issue. The action is stuck after a successful deployment from a self-hosted runner. In my case, it is a linux function app.

Here's a demo log (cancelled the job after ~4h): demo_log.txt

[Royston-Shufflebotham-i2]

I'm experiencing the same issue, and it's quite possible that issue #119 is also the same issue.

I did a bit of investigation by adding lots of debugging statements to a fork and managed to track the problem down...

The problem seems to be an infinite retry bug in an old version of typed-rest-client (pulled in via azure-actions-webclient) . It was fixed some time ago but this action is still using an old version. With the old version, certain network errors (e.g. 'not found') will cause the request to simply loop forever.

It was fixed here: https://github.com/microsoft/typed-rest-client/pull/248

typed-rest-client 1.8.2 or above contains the fix.

I've submitted PR #125 to bump the version of typed-rest-client.

[patelchandni]

@Royston-Shufflebotham-i2 Thank you for investigating and creating the PR. Due to set up requirements, I have created a new PR #126 based on your PR to run E2E tests cases.

[github-actions[bot]]

This issue is idle because it has been open for 14 days with no activity.

[Royston-Shufflebotham-i2]

Issue isn't idle. It's awaiting a review of PR #126.

[patelchandni]

@Royston-Shufflebotham-i2 the PR was merged and released. Please test and let us know your feedback. Thank you.

[Tbohunek]

We have actions version 1.4.6 and yet we still have this issue. Why could that be?

[Royston-Shufflebotham-i2]

@Tbohunek The problem I had was that the Azure deployment was failing due to my configuration errors, but rather than reporting that failure correctly, the action had hardcoded an old version of typed-rest-client which hung when it experienced errors.

[Tbohunek]

@Royston-Shufflebotham-i2 interesting. Would you be able/willing to tell more about these config errors you made? We ain't giving it much:

name: Deploy .NET to Function App

on:
  workflow_dispatch:

env:
  HTTP_PROXY: ${{ secrets.HTTP_PROXY }}
  HTTPS_PROXY: ${{ secrets.HTTPS_PROXY }}
  NO_PROXY: ${{ secrets.NO_PROXY }}

  AZURE_FUNCTION_APP_NAME: myfunction
  AZURE_FUNCTION_PROJECT_PATH: src/Functions/ResourceFunctions

jobs:
  build-and-deploy:
    runs-on: [self-hosted]

   steps:
      - name: Run Azure Functions Action
        uses: Azure/functions-action@v1.4.6
        id: fa
        with:
          app-name: ${{ env.AZURE_FUNCTION_APP_NAME }}
          package: ${{ env.AZURE_FUNCTION_PROJECT_PATH }}/output

But we run it on self-hosted Github Runner in Azure Vnet with Azure Firewall, which denies it over Action: Deny. Reason: Host header port 443 mismatch with network port 80 which I fail to understand. With TCPdump on the host we see IP actions-runner.internal.cloudapp.net.34374 > 0.0.12.56.http: Flags [P.], seq 1:137, ack 1, win 502, length 136: HTTP: CONNECT myfunction.scm.azurewebsites.net:443 HTTP/1.1 which again I fail to understand. Both of these keep appearing until we kill the Action.

Your help shall be rewarded.

[Tbohunek]

Here I found this...which sounds like the issue we have.. However, I don't see if there's something wrong on our side, or it's a problem of the Action code. Clues?

In both HTTP and TLS inspected HTTPS cases, the firewall ignores the packet's destination IP address and uses the DNS resolved IP address from the Host header. The firewall expects to get port number in the Host header, otherwise it assumes the standard port 80. If there's a port mismatch between the actual TCP port and the port in the host header, the traffic is dropped. DNS resolution is done by Azure DNS or by a custom DNS if configured on the firewall. 

[Royston-Shufflebotham-i2]

"Host header port 443 mismatch with network port 80" sounds like an issue from an HTTP/HTTPS proxy to me. e.g. you're going through a port 80 proxy but attempting to connect to an HTTPS server, thereby providing a port-443-based Host header to the proxy. No idea why. I know nothing about Azure Firewall.

In your situation I would wind things right back from such a complicated deployment and get a basic function working from a GitHub-hosted runner first, without all your *_PROXY env vars, then build it back up. My problem was just a simple function misconfiguration but bumping the network client package allowed the error to be reported.

[Tbohunek]

Thanks @Royston-Shufflebotham-i2. Github-hosted runner works, so the overall Action is correct. While we set *_PROXY env, traffic is NOT going through proxy - that wouldn't hit Azure Firewall. I don't know why and I don't see any proxy-related docs in this Action's docs. Other steps in the job DO go through proxy - they failed without env set.