search

Azure / azure-functions-host

The host/runtime that powers Azure Functions
https://functions.azure.com
MIT License
1.94k stars 442 forks source link

If downloading the extension bundle takes too long, the runtime doesn't start properly #4959

Open Programmerman1 opened 5 years ago

Programmerman1 commented 5 years ago

If you're on a slow Internet connection such that the extension bundle can't download within about 2 minutes, then the runtime fails.

Manually downloading the extension bundle (nearly 5 minutes on my connection) and expanding it where the ExtensionBundleManager looks for it works around the issue.

Console logs:

[hostname]:[project] [username]$ func start

                  %%%%%%
                 %%%%%%
            @   %%%%%%    @
          @@   %%%%%%      @@
       @@@    %%%%%%%%%%%    @@@
     @@      %%%%%%%%%%        @@
       @@         %%%%       @@
         @@      %%%       @@
           @@    %%      @@
                %%
                %

Azure Functions Core Tools (2.7.1585 Commit hash: 9a2b7239b937148e3b76e6f7999798629a82134a)
Function Runtime Version: 2.0.12641.0
[9/19/19 8:14:17 PM] Building host: startup suppressed:False, configuration suppressed: False
[9/19/19 8:14:17 PM] Looking for extension bundle Microsoft.Azure.Functions.ExtensionBundle at /var/folders/pl/qgvhtxwx7153kf2c8y9bc8kw0000gn/T/Functions/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle
[9/19/19 8:14:17 PM] Fetching information on versions of extension bundle Microsoft.Azure.Functions.ExtensionBundle available on https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/index.json
[9/19/19 8:14:18 PM] Downloading extension bundle from https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/1.0.0/Microsoft.Azure.Functions.ExtensionBundle.1.0.0.zip to /var/folders/pl/qgvhtxwx7153kf2c8y9bc8kw0000gn/T/5d2fd1c0-d8f7-4ced-82c9-93cf0de7f0bc/Microsoft.Azure.Functions.ExtensionBundle.1.0.0.zip
Hosting environment: Production
Content root path: /Users/[username]/[project]
Now listening on: http://0.0.0.0:7071
Application started. Press Ctrl+C to shut down.
Value cannot be null.
Parameter name: provider
Application is shutting down...
[9/19/19 8:16:31 PM] Stopping host...
[9/19/19 8:16:31 PM] Host shutdown completed.
ahmelsayed commented 5 years ago

/cc @soninaren

fabiocav commented 5 years ago

@soninaren can you please comment?

We need to make sure this scenario is properly handled and a better message is displayed. We also need the ability to resume the download so we're not hopelessly stuck in a situation like this.

mrpassiontea commented 4 years ago

Any new updates on this? I'm also having the issue of Value cannot be null and Parameter name: provider. I've checked my local.settings.json to have accurate connection strings and I've even created a new Typescript project with one httpTrigger function. I still receive this error. I really think it's because before the extension bundle is fully installed, a timeout occurs.

If there's a solution please let me know as this is really impacting my work.

I've created a thread on social.msdn about this issue here: https://social.msdn.microsoft.com/Forums/en-US/ef8c3c03-482d-4bc9-902b-78324305d9b0/cannot-connect-to-runtime-process-timeout-after-10000ms?forum=AzureFunctions

soninaren commented 4 years ago

@mrpassiontea Can you remove the extension bundle section from host.json and try again? By Removing bundle configuration the extension bundle download will not happen. That should help you isolate the issue.

mrpassiontea commented 4 years ago

@mrpassiontea Can you remove the extension bundle section from host.json and try again? By Removing bundle configuration the extension bundle download will not happen. That should help you isolate the issue.

So when including 

extensionBundle: {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[1.*, 2.0.0)"
}

in Host.json I get:

Now listening on: http://0.0.0.0:7071
Application started. Press Ctrl+C to shut down.
Value cannot be null.
Parameter name: provider
Application is shutting down...

When I remove the extensionBundle from Host.json the Function App runs but only the httpTrigger works and my three other functions have this error:

The 'Install' function is in error: The binding type(s) 'table' are not registered. Please ensure the type is correct and the binding extension is installed.
[3/3/20 12:34:37 PM] The 'X' function is in error: The binding type(s) 'serviceBus' are not registered. Please ensure the type is correct and the binding extension is installed.
[3/3/20 12:34:37 PM] The 'Y' function is in error: The binding type(s) 'serviceBusTrigger, sendGrid' are not registered. Please ensure the type is correct and the binding extension is installed.

az version results in:

az version
This command is in preview. It may be changed/removed in a future release.
{
  "azure-cli": "2.1.0",
  "azure-cli-command-modules-nspkg": "2.0.3",
  "azure-cli-core": "2.1.0",
  "azure-cli-nspkg": "3.0.4",
  "azure-cli-telemetry": "1.0.4",
  "extensions": {}
}
Programmerman1 commented 4 years ago

Here were my workaround steps. This was on a Mac but the concepts should carry over to other platforms.

If func host start fails to get rolling and you see in the console messages about downloading the extension bundle and then two minutes later that the provider cannot be null, you probably need to manually download the extension bundle.

[9/19/19 8:14:17 PM] Looking for extension bundle Microsoft.Azure.Functions.ExtensionBundle at /var/folders/pl/qgvhtxwx7153kf2c8y9bc8kw0000gn/T/Functions/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle

...

[9/19/19 8:14:18 PM] Downloading extension bundle from https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/1.0.0/Microsoft.Azure.Functions.ExtensionBundle.1.0.0.zip to /var/folders/pl/qgvhtxwx7153kf2c8y9bc8kw0000gn/T/5d2fd1c0-d8f7-4ced-82c9-93cf0de7f0bc/Microsoft.Azure.Functions.ExtensionBundle.1.0.0.zip
Value cannot be null.
Parameter name: provider
Application is shutting down...
[9/19/19 8:16:31 PM] Stopping host...
[9/19/19 8:16:31 PM] Host shutdown completed.
  1. Download the file referenced in the console logs (look for Downloading extension bundle)
  2. Expand the .zip file
  3. Make the directory referenced in the console logs (look for Looking for extension bundle)
  4. Make a folder for the version number (1.0.0 in the example)
  5. Move the contents of the .zip file into the version number folder
  6. func host start again
soninaren commented 4 years ago

The workaround steps suggested by @Programmerman1 should work. However you would need to download version 1.1.1 instead of 1.0.0 since that is the latest version of bundle available and core tools check for latest.

For windows the path would be %Temp%\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle

Also if you don't want to go through the manual steps, you can simple delete the ExtensionBundles folder and core tools should re-create it, and attempt a download again.

matthew-r-clark commented 4 years ago

I'm experiencing a similar issue with the same Value cannot be null. error and hangs for 2 minutes. The difference is that the runtime is able to find the extension bundle and doesn't attempt to download anything:

Azure Functions Core Tools (2.7.2184 Commit hash: 5afacc827c2848e4debc23bb96604f1ffce0
9cc7)
Function Runtime Version: 2.0.12961.0
[4/8/20 3:34:29 PM] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for la
nguage:powershell
[4/8/20 3:34:29 PM] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for la
nguage:java
[4/8/20 3:34:29 PM] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:python
[4/8/20 3:34:29 PM] Building host: startup suppressed: 'False', configuration suppressed: 'False', startup operation id: '84a6adb4-ae3b-40ea-a7c6-30a46e7320df'
[4/8/20 3:34:29 PM] Reading host configuration file '/home/[username]/[project]/host.json'
[4/8/20 3:34:29 PM] Host configuration file read:
[4/8/20 3:34:29 PM] {
[4/8/20 3:34:29 PM]   "version": "2.0",
[4/8/20 3:34:29 PM]   "extensionBundle": {
[4/8/20 3:34:29 PM]     "id": "Microsoft.Azure.Functions.ExtensionBundle",
[4/8/20 3:34:29 PM]     "version": "[1.*, 2.0.0)"
[4/8/20 3:34:29 PM]   }
[4/8/20 3:34:29 PM] }
[4/8/20 3:34:29 PM] Reading functions metadata
[4/8/20 3:34:29 PM] 1 functions found
[4/8/20 3:34:29 PM] Looking for extension bundle Microsoft.Azure.Functions.ExtensionBu
ndle at /tmp/Functions/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle
[4/8/20 3:34:29 PM] Found a matching extension bundle at /tmp/Functions/ExtensionBundl
es/Microsoft.Azure.Functions.ExtensionBundle/1.1.1
[4/8/20 3:34:29 PM] Fetching information on versions of extension bundle Microsoft.Azu
re.Functions.ExtensionBundle available on https://functionscdn.azureedge.net/public/Ex
tensionBundles/Microsoft.Azure.Functions.ExtensionBundle/index.json
Hosting environment: Production
Content root path: /home/[username]/[project]
Now listening on: http://0.0.0.0:7071
Application started. Press Ctrl+C to shut down.
Value cannot be null.
Parameter name: provider
Application is shutting down...
[4/8/20 3:36:44 PM] Stopping host...
[4/8/20 3:36:44 PM] Host shutdown completed.

I tried removing the /tmp/Functions/ExtensionBundles directory and it seems to do the same thing except it doesn't say it found the extension bundle. It still doesn't appear to be trying to download anything:

Azure Functions Core Tools (2.7.2184 Commit hash: 5afacc827c2848e4debc23bb96604f1ffce0
9cc7)
Function Runtime Version: 2.0.12961.0
[4/8/20 3:37:33 PM] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:powershell
[4/8/20 3:37:33 PM] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:java
[4/8/20 3:37:33 PM] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:python
[4/8/20 3:37:33 PM] Building host: startup suppressed: 'False', configuration suppressed: 'False', startup operation id: '6935c34a-a25a-415c-b9b7-fd911eb94458'
[4/8/20 3:37:33 PM] Reading host configuration file '/home/[username]/[project]/host.json'
[4/8/20 3:37:33 PM] Host configuration file read:
[4/8/20 3:37:33 PM] {
[4/8/20 3:37:33 PM]   "version": "2.0",
[4/8/20 3:37:33 PM]   "extensionBundle": {
[4/8/20 3:37:33 PM]     "id": "Microsoft.Azure.Functions.ExtensionBundle",
[4/8/20 3:37:33 PM]     "version": "[1.*, 2.0.0)"
[4/8/20 3:37:33 PM]   }
[4/8/20 3:37:33 PM] }
[4/8/20 3:37:34 PM] Reading functions metadata
[4/8/20 3:37:34 PM] 1 functions found
[4/8/20 3:37:34 PM] Looking for extension bundle Microsoft.Azure.Functions.ExtensionBundle at /tmp/Functions/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle
[4/8/20 3:37:34 PM] Fetching information on versions of extension bundle Microsoft.Azure.Functions.ExtensionBundle available on https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/index.json
Hosting environment: Production
Content root path: /home/[username]/[project]
Now listening on: http://0.0.0.0:7071
Application started. Press Ctrl+C to shut down.
Value cannot be null.
Parameter name: provider
Application is shutting down...
[4/8/20 3:39:47 PM] Stopping host...
[4/8/20 3:39:47 PM] Host shutdown completed.

The /tmp/Functions/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/1.1.1 directory includes:

local.settings.json contains the following within "Values":

matthew-r-clark commented 4 years ago

Well I deleted the following from host.json and it started working locally.

  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[1.*, 2.0.0)"
  }
miguelguadarrama commented 4 years ago

Same issue here. I tried the workaround by @Programmerman1 however it didn't quite work (System.Private.CoreLib: The file ....dll already exists), but I noticed something. It seems the download from azureedge.net CDN goes very, very slow, for some reason. I think it just times out and that's why I end up getting the error when running func host start.

Just out of curiosity I tried downloading it using a VPN connected to a different country and download was blazing fast (less than a second) and the function worked fine. So it's probably some network issues for the azureedge CDN in my region (East Asia).

However, what I don't really understand is why everyday I run func host start and it needs to re-download the same bundle version... I would think the same one from a few hours ago should still be valid?

rodrigo-gutierrez commented 4 years ago

Having same issue. Located in Japan. Attempts to hit https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/1.1.1/Microsoft.Azure.Functions.ExtensionBundle.1.1.1.zip time out after like 10 seconds and app just crashes:

A host error has occurred during startup operation '6ff53f3e-6f07-451c-98ac-808c0e6508f0'.
System.Net.Http: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
System.Private.CoreLib: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Value cannot be null.
Parameter name: provider
Application is shutting down...
HyeonGook commented 4 years ago

Also having same issue in South Korea.

func new and func start don't work during downloading extensions.

mrpassiontea commented 4 years ago

I haven't replied back to this thread for awhile now, but I was able to get it working after fiddling around with it.

What I noticed is that this is a common issue that occurs randomly. For instance, I originally commented on March 1 and after a few days was able to fix it. Fast-forward to yesterday and I received the same error.

The way I fixed yesterday's error was by removing the extension bundle in the host.json file and ran func start. That seemed to make it work. But by removing the extension bundle from your host.json, I believe this would cause an issue when you attempt deploy it.

Honestly, this is such a random issue and occurs more often than it should. Even if I don't make any changes to my files, suddenly this happens.

Things I've attempted:

It's frustrating because I can go back and forth between adding/removing the extendionBundle to the Host.json and it would either work or not work. Which shouldn't be the case. Right now it seems to be working without the extensionBundle within the Host.json. I'm pretty sure when I attempt to deploy my function, I would need to add the extensionBundle to the Host.json. I'll see if it'll cause problems when I get to that point.

Note: I've had this issue occur to me while I was in South Korea, Japan, and America (As I was in those places when this problem occurred.)

Crisfole commented 3 years ago

This occurred for me on terrible Hotel Wifi.

numocityadmin commented 3 years ago

In my case, the host.json had:

  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[1.*, 2.0.0)"
  }

This resulted in the Value cannot be null error.

I changed the extensionBundle in host.json to:

  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[2.*, 3.0.0)"
  }

And it worked :) I have no idea why the error has to be cryptic, why the old one had an issue only on some laptops, and why this solved the issue! If anyone can explain, I would be grateful!