Azure / azure-functions-sql-extension

Azure SQL bindings for Azure Functions ⚡️ supports Azure SQL Database, Azure SQL Managed Instance, and SQL Server 2016+
https://aka.ms/sqlbindings
MIT License
120 stars 61 forks source link

Insert data in Azure SQL database on a schedule fails using Azure Functions Python V2 bindings due to cold start database #1078

Open LouisDeconinck opened 4 months ago

LouisDeconinck commented 4 months ago

I have an Azure Function that scrapes a website every day on a daily time trigger, all results are stored in a dictionary. At the end I want to store all of those rows (~10.000) in the Azure SQL database. Usually when I manually run the function after having manually connected to the database everything goes perfectly, but when the scraping happens due to the time trigger I always get this error: Database 'X' on server 'x.database.windows.net' is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of '{11B1C15C-BA7C-4BF9-929E-33AE4F11FC72}'. image001

This sometimes also happens when using the Query editor. image002

I believe this is caused due to some kind of cold start of Azure SQL database? Now, this would not be a problem if I could implement a proper retry logic, unfortunately I have not succeeded. I have a feeling that r.set(rows_sql) is launched as a background task, and that the try statement does not check if it was successful, which it never is due to the cold start. Is there any way how I can deal with this scenario with the Azure SQL binding Python V2 for Azure functions?

Steps to Reproduce:

  1. Use a daily time trigger to insert data in an Azure SQL database.
  2. Database connection fails the first time (probably due to cold start).
lucyzhang929 commented 4 months ago

@LouisDeconinck SQL output bindings does not have built in support for retries (https://github.com/Azure/azure-functions-sql-extension/blob/main/docs/BindingsOverview.md#retry-support-for-output-bindings).

Since you are using a timer trigger, would something like this retry example help? https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-error-pages?tabs=fixed-delay%2Cisolated-process%2Cnode-v4%2Cpython-v2&pivots=programming-language-python#retry-examples