Azure Function doesn't capture exceptions on Azure

[empz]

I have a Queue Trigger Azure Function in Python with the following code:

import logging
import os

import azure.functions as func
import sentry_sdk
from sentry_sdk.api import capture_exception, flush, push_scope
from sentry_sdk.integrations.serverless import serverless_function

# Sentry configuration
sentry_dsn = "my-dsn"
environment = "DEV"
logger = logging.getLogger(__name__)
sentry_sdk.init(
    sentry_dsn,
    environment=environment,
    send_default_pii=True,
    request_bodies="always",
    with_locals=True,
)
sentry_sdk.utils.MAX_STRING_LENGTH = 2048

@serverless_function
def main(msg: func.QueueMessage) -> None:

    with push_scope() as scope:
        scope.set_tag("function.name", "ProcessHeadersFile")
        scope.set_context(
            "Queue Message",
            {
                "id": msg.id,
                "dequeue_count": msg.dequeue_count,
                "insertion_time": msg.insertion_time,
                "expiration_time": msg.expiration_time,
                "pop_receipt": msg.pop_receipt,
            },
        )

        try:
            # code that might raise exceptions here
            function_that_raise()
        except Exception as ex:
            print(ex)
            # Rethrow to fail the execution
            raise

def function_that_raise():
    return 5 / 0

Works locally, but not on Azure. I run multiple invocations, I get all the failures, but then nothing shows on Sentry.

I have also try manually capturing and flushing, but doesn't work either:

try:
    # code that might raise exceptions here
    function_that_raise()
except Exception as ex:
    capture_exception(ex)
    flush(2)
    raise

Using sentry-sdk==0.19.5.

How can I troubleshoot what's happening? What can be the causes that it works when running the function locally, but not when running on Azure?

[untitaker]

can you try with debug=True? That gives you logs for what's happening inside the sdk

On Thu, Jan 14, 2021 at 5:54 PM Emiliano Parizzi notifications@github.com wrote:

I have a Queue Trigger Azure Function in Python with the following code:

import loggingimport os import azure.functions as funcimport sentry_sdkfrom sentry_sdk.api import capture_exception, flush, push_scopefrom sentry_sdk.integrations.serverless import serverless_function

Sentry configurationsentry_dsn = "https://4903cfdfe3e24de5b34981fd11a0c787@o374073.ingest.sentry.io/5191540"environment = "DEV"logger = logging.getLogger(name)sentry_sdk.init(

sentry_dsn,
environment=environment,
send_default_pii=True,
request_bodies="always",
with_locals=True,

)sentry_sdk.utils.MAX_STRING_LENGTH = 2048

@serverless_functiondef main(msg: func.QueueMessage) -> None:

with push_scope() as scope:
    scope.set_tag("function.name", "ProcessHeadersFile")
    scope.set_context(
        "Queue Message",
        {
            "id": msg.id,
            "dequeue_count": msg.dequeue_count,
            "insertion_time": msg.insertion_time,
            "expiration_time": msg.expiration_time,
            "pop_receipt": msg.pop_receipt,
        },
    )

    try:
        # code that might raise exceptions here
        function_that_raise()
    except Exception as ex:
        print(ex)
        # Rethrow to fail the execution
        raise

def function_that_raise(): return 5 / 0

Works locally, but not on Azure. I run multiple invocations, I get all the failures, but then nothing shows on Sentry.

I have also try manually capturing and flushing, but doesn't work either:

try:

code that might raise exceptions here

function_that_raise()except Exception as ex:
capture_exception(ex)
flush(2)
raise

Using sentry-sdk==0.19.5.

How can I troubleshoot what's happening? What can be the causes that it works when running the function locally, but not when running on Azure?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/getsentry/sentry-python/issues/979, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGMPRNGA236DVUVIGJGACDSZ4OS3ANCNFSM4WCXA5IA .

[empz]

can you try with debug=True? That gives you logs for what's happening inside the sdk … On Thu, Jan 14, 2021 at 5:54 PM Emiliano Parizzi @.> wrote: I have a Queue Trigger Azure Function in Python with the following code: import loggingimport os import azure.functions as funcimport sentry_sdkfrom sentry_sdk.api import capture_exception, flush, push_scopefrom sentry_sdk.integrations.serverless import serverless_function # Sentry configurationsentry_dsn = @./5191540"environment = "DEV"logger = logging.getLogger(name)sentry_sdk.init( sentry_dsn, environment=environment, send_default_pii=True, request_bodies="always", with_locals=True, )sentry_sdk.utils.MAX_STRING_LENGTH = 2048 @serverless_functiondef main(msg: func.QueueMessage) -> None: with push_scope() as scope: scope.set_tag("function.name", "ProcessHeadersFile") scope.set_context( "Queue Message", { "id": msg.id, "dequeue_count": msg.dequeue_count, "insertion_time": msg.insertion_time, "expiration_time": msg.expiration_time, "pop_receipt": msg.pop_receipt, }, ) try: # code that might raise exceptions here function_that_raise() except Exception as ex: print(ex) # Rethrow to fail the execution raise def function_that_raise(): return 5 / 0 Works locally, but not on Azure. I run multiple invocations, I get all the failures, but then nothing shows on Sentry. I have also try manually capturing and flushing, but doesn't work either: try: # code that might raise exceptions here function_that_raise()except Exception as ex: capture_exception(ex) flush(2) raise Using sentry-sdk==0.19.5. How can I troubleshoot what's happening? What can be the causes that it works when running the function locally, but not when running on Azure? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#979>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGMPRNGA236DVUVIGJGACDSZ4OS3ANCNFSM4WCXA5IA .

Ok, done, but where am I supposed to view those logs...?

[empz]

The only Sentry related logs I can find in ApplicationInsights > Logs are the following:

"14/01/2021, 19:23:40.463","[sentry] DEBUG: Enabling integration threading",1
"14/01/2021, 19:23:40.462","[sentry] DEBUG: Enabling integration argv",1
"14/01/2021, 19:23:40.462","[sentry] DEBUG: Enabling integration modules",1
"14/01/2021, 19:23:40.461","[sentry] DEBUG: Enabling integration atexit",1
"14/01/2021, 19:23:40.461","[sentry] DEBUG: Enabling integration dedupe",1
"14/01/2021, 19:23:40.461","[sentry] DEBUG: Enabling integration excepthook",1
"14/01/2021, 19:23:40.457","[sentry] DEBUG: Enabling integration stdlib",1
"14/01/2021, 19:23:40.457","[sentry] DEBUG: Enabling integration logging",1
"14/01/2021, 19:23:40.456","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed",1
"14/01/2021, 19:23:40.453","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.",1
"14/01/2021, 19:23:40.445","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed",1
"14/01/2021, 19:23:40.442","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed",1
"14/01/2021, 19:23:40.433","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed",1
"14/01/2021, 19:23:40.427","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed",1
"14/01/2021, 19:23:40.417","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed",1
"14/01/2021, 19:23:40.413","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed",1
"14/01/2021, 19:23:40.406","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed",1
"14/01/2021, 19:23:40.402","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed",1
"14/01/2021, 19:23:40.397","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed",1
"14/01/2021, 19:23:40.385","[sentry] DEBUG: Setting up integrations (with default = True)",1
"14/01/2021, 19:23:40.365","[sentry] DEBUG: Enabling integration threading",1
"14/01/2021, 19:23:40.364","[sentry] DEBUG: Enabling integration argv",1
"14/01/2021, 19:23:40.364","[sentry] DEBUG: Enabling integration modules",1
"14/01/2021, 19:23:40.364","[sentry] DEBUG: Enabling integration atexit",1
"14/01/2021, 19:23:40.364","[sentry] DEBUG: Enabling integration dedupe",1
"14/01/2021, 19:23:40.364","[sentry] DEBUG: Enabling integration excepthook",1
"14/01/2021, 19:23:40.364","[sentry] DEBUG: Enabling integration stdlib",1
"14/01/2021, 19:23:40.358","[sentry] DEBUG: Enabling integration logging",1
"14/01/2021, 19:23:40.357","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed",1
"14/01/2021, 19:23:40.353","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.",1
"14/01/2021, 19:23:40.346","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed",1
"14/01/2021, 19:23:40.343","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed",1
"14/01/2021, 19:23:40.335","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed",1
"14/01/2021, 19:23:40.328","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed",1
"14/01/2021, 19:23:40.324","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed",1
"14/01/2021, 19:23:40.316","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed",1
"14/01/2021, 19:23:40.308","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed",1
"14/01/2021, 19:23:40.303","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed",1
"14/01/2021, 19:23:40.295","[sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed",1
"14/01/2021, 19:23:40.277","[sentry] DEBUG: Setting up integrations (with default = True)",1

But nothing about particular events being sent, rejected or not.

[untitaker]

@empz I don't know much about azure functions but the SDK will print them to stdout/err. Seems you found it anyway.

Are you sure those are all logs? Don't you see logs for when the exception occurs? Because then the SDK would print much more stuff, including when the HTTP request to Sentry is sent out and some extra message for a successful flush

[untitaker]

If you do init, capture_exception and flush in the same function all at once, does anything happen?

[empz]

I'm trying to get those logs to show up anywhere on Azure but it seems the Function App framework overrides the logging level and so DEBUG messages are filtered.

I'm now trying a very simple HttpTrigger function in the same Function App. Locally I can debug and put breakpoints inside the serverless_function decorator and I can see those messages that you're asking for.

[2021-01-15T09:57:44.631Z] Python HTTP trigger function processed a request.
[2021-01-15T10:00:42.963Z]  [sentry] DEBUG: Sending event, type:null level:error event_id:7332a45e51e5455f9263aff2344da2c1 project:xxxxxxx host:xxxxxxx.ingest.sentry.io
[2021-01-15T10:00:59.942Z]  [sentry] DEBUG: Flushing HTTP transport
[2021-01-15T10:01:15.199Z] Flushing HTTP transport
[2021-01-15T10:01:15.203Z]  [sentry] DEBUG: background worker got flush request
[2021-01-15T10:01:15.205Z] background worker got flush request
[2021-01-15T10:01:15.304Z]  [sentry] DEBUG: 0 event(s) pending on flush
[2021-01-15T10:01:15.304Z] 0 event(s) pending on flush
[2021-01-15T10:01:15.872Z]  [sentry] DEBUG: background worker flushed
[2021-01-15T10:01:15.875Z] background worker flushed
[2021-01-15T10:01:24.371Z] InvocationResponse received for invocation id: 977768ce-819a-4894-b865-4c04419bc6dc

Unfortunately I can't see any of that when running it on Azure. I just don't understand what's happening. I have another similar function that works just fine...

Azure is really bad at monitoring and the "Stream Logs" option in the function app will just show logs from the function underlying framework (.NET based), but not logs from the actual function... The only apparent way to get logs out of the function is to use ApplicationInsights. I have tried with no luck, as I can only see those integration related logs (which happens outside the main function invocation), during init() .

Are you suggesting that I move the sentry client into the main? I'm going to try that right now, but it should work with a single client shared between all the invocations.

[empz]

Ok so the following function is working:

@serverless_function
def main(req: func.HttpRequest) -> func.HttpResponse:
    # Sentry configuration
    sentry_dsn = "https://my-sentry-dsn"
    sentry_sdk.init(
        dsn=sentry_dsn,
        debug=True,
    )

    logging.info("Python HTTP trigger function processed a request.")

    name = req.params.get("name")
    if not name:
        try:
            req_body = req.get_json()
        except ValueError as e:
            capture_exception(e)
            flush(10)

            raise
        else:
            name = req_body.get("name")

    if name:
        return func.HttpResponse(
            f"Hello, {name}. This HTTP triggered function executed successfully."
        )
    else:
        return func.HttpResponse(
            "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
            status_code=200,
        )

But I'm getting only one event in Sentry, It doesn't have the handled property set, meaning it's the capture_exception and not the serverless_function decorator working. In the above example, even though I'm explicitly capturing an exception and flushing right after, I still re-raise it and that should be caught by the decorator, which doesn't seem to be happening...

I'm going to try to attach a file share to the function so I can pipe the sentry logger to a file there. I'm not a great python developer myself though. Would something like the following do the trick?

import logging
from sentry_sdk.utils import logger as sentry_logger

fh = logging.FileHandler("/some/mount/path/sentry.log")
sentry_logger.addHandler(fh)

UPDATE So piping the sentry_logger to a local file on my machine I do get all the logging now. And yes, I'm only getting one event because the second one (the serverless_function one) get's filtered by the DedupeIntegration.

[untitaker]

I think your FileHandler would work.

Can you do the following:

1. revert to teh "sane" setup with serverless_funciton, no manual exception capturing, and init() on the module level (no indentation basically)
2. add print(threading.current_thread()) at all points where you initialize the SDK (and also print something so you know which print has been hit

basically:

print("on module level: {}".format(threading.current_thread()))
init(...)

@serverless_function
def main(...) -> ...:
    print("in the function: {}".format(threading.current_thread()))
    raise Exception("foo")

SDK initialization is done per-thread. However, when the SDK is initialized on the thread that sentry_sdk is imported from (usually the main thread), the SDK initialization applies to all threads (with some caveats). Azure may do the following:

1. spawn a threadpool
2. import your app on thread A
3. run your handler on thread B
4. up until here this is how most webservers behave and we can deal with this. Azure would somehow trick the SDK into thinking thread A is not the main thread?

[untitaker]

And yes, I'm only getting one event because the second one (the serverless_function one) get's filtered by the DedupeIntegration.

This should be ok, right? We dedupe by exception identity

[empz]

Well, that simple "sane" function is being able to send events to Sentry. I couldn't get the prints (I also tried using logging.info) because Azure sucks. I could only get the traces inside the function and they look like this:

in the function: <Thread(ThreadPoolExecutor-0_0, started daemon 140702941513472)> |  
  | in the function: <Thread(ThreadPoolExecutor-1_0, started daemon 140529519499008)> |  
  | in the function: <Thread(ThreadPoolExecutor-0_0, started daemon 140702941513472)> |  
  | in the function: <Thread(ThreadPoolExecutor-1_0, started daemon 140529519499008)> |  
  | in the function: <Thread(ThreadPoolExecutor-0_0, started daemon 140702941513472)> |  
  | in the function: <Thread(ThreadPoolExecutor-1_0, started daemon 140529519499008)> |  
  | in the function: <Thread(ThreadPoolExecutor-0_0, started daemon 140702941513472)> |  
  | in the function: <Thread(ThreadPoolExecutor-1_0, started daemon 140529519499008)> |  
  | in the function: <Thread(ThreadPoolExecutor-1_0, started daemon 140529519499008)> |  
  | in the function: <Thread(ThreadPoolExecutor-0_0, started daemon 139715106207488)> |  
  | in the function: <Thread(ThreadPoolExecutor-0_0, started daemon 140376662337280)>

I'm going to try my real queue triggered function and see if it's working now...

[empz]

Nope, it just doesn't work...

Here's the full code. The only difference with the basic HttpTrigger that works is that this is a queue triggered and I'm setting some context and tags before raising...

import logging
import os
import json
import urllib

import azure.functions as func
import sentry_sdk
from sentry_sdk.integrations.serverless import serverless_function

from shared.data_logs_ingestion import DataLogsIngestionService
from shared.data_log_parser import DataLogParser
from shared.data_log_file import DataLogFile

logger = logging.getLogger(__name__)

# Sentry configuration
sentry_dsn = os.getenv("SENTRY_DSN")
environment = os.getenv("ENVIRONMENT", "PROD")

sentry_sdk.init(
    dsn=sentry_dsn,
    environment=os.environ["ENVIRONMENT"],
    debug=True,
)

# Load environment variables.
with_aad_app_key = os.getenv("ApplicationId") is not None
kusto_cluster_name = os.environ["KustoClusterName"]
storage_sas = os.environ["StorageSAS"]

# Init DataLogsIngestionService
ingestion_service = DataLogsIngestionService(kusto_cluster_name, with_aad_app_key)

@serverless_function
def main(msg: func.QueueMessage) -> None:
    sentry_sdk.set_tag("function.name", "ProcessHeadersFile")
    sentry_sdk.set_context(
        "Queue Message",
        {
            "id": msg.id,
            "dequeue_count": msg.dequeue_count,
            "insertion_time": msg.insertion_time,
            "expiration_time": msg.expiration_time,
            "pop_receipt": msg.pop_receipt,
        },
    )

    logger.info(
        "Python queue trigger function processed a queue item: %s",
        msg.get_body().decode("utf-8"),
    )

    raise Exception("ProcessHeadersFile")

[empz]

Ok, something about these lines is causing the issue...

sentry_sdk.set_tag("function.name", "ProcessHeadersFile")
sentry_sdk.set_context(
    "Queue Message",
    {
        "id": msg.id,
        "dequeue_count": msg.dequeue_count,
        "insertion_time": msg.insertion_time,
        "expiration_time": msg.expiration_time,
        "pop_receipt": msg.pop_receipt,
    },
)

What's wrong with that? Could that be causing an event not to be sent? But why would that only happen on Azure and not locally?

[untitaker]

this seems okay. Could you try wrapping all variables here in str() or repr() just to be sure?

[empz]

No, that wasn't it.... I really don't know what's happening, it's too random.

I see this exception on Azure:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.ProcessHeadersFile
 ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
Exception: AttributeError: 'NoneType' object has no attribute 'group'
Stack:   File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 355, in _handle__invocation_request
    call_result = await self._loop.run_in_executor(
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 542, in __run_sync_func
    return func(**params)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sentry_sdk/integrations/serverless.py", line 55, in inner
    _capture_and_reraise()
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sentry_sdk/integrations/serverless.py", line 80, in _capture_and_reraise
    reraise(*exc_info)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sentry_sdk/_compat.py", line 54, in reraise
    raise value
  File "/home/site/wwwroot/.python_packages/lib/site-packages/sentry_sdk/integrations/serverless.py", line 53, in inner
    return f(*args, **kwargs)
  File "/home/site/wwwroot/ProcessHeadersFile/__init__.py", line 70, in main
    data_log_file = DataLogFile(blob_uri)
  File "/home/site/wwwroot/shared/data_log_file.py", line 25, in __init__
    self.provider = self._get_provider_header()
  File "/home/site/wwwroot/shared/data_log_file.py", line 65, in _get_provider_header
    return m.group(1)

Which clearly shows that the serverless_function decorator is capturing the exception (_capture_and_reraise()) but then it doesn't get to Sentry at all. Not a single one of hundreds of failed executions.

[empz]

I've successfully attached the sentry_logger to a file handler in an Azure File Share and all I get are these...

Setting up integrations (with default = True)
Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
Enabling integration logging
Enabling integration logging
Enabling integration stdlib
Enabling integration stdlib
Enabling integration excepthook
Enabling integration excepthook
Enabling integration dedupe
Enabling integration dedupe
Enabling integration atexit
Enabling integration atexit
Enabling integration modules
Enabling integration modules
Enabling integration argv
Enabling integration argv
Enabling integration threading
Enabling integration threading

[untitaker]

Well that doesn't make much sense to me, it also seems that the log lines are written twice each

[empz]

Well that can caused by the function app spinning more servers, I guess...

[empz]

I'm off for the week but here's the latest.

The following is not working and no other logs than the ones I shared before are being generated by the sentry_logger.

import logging
import os
import json
import urllib
import socket

import azure.functions as func
import sentry_sdk
from sentry_sdk.integrations.serverless import serverless_function
from sentry_sdk.utils import logger as sentry_logger

# Sentry configuration
sentry_dsn = os.getenv("SENTRY_DSN")
environment = os.getenv("ENVIRONMENT", "PROD")

# Piping sentry_logger logs to a mounted Azure File Share
file_handler = logging.FileHandler(f"/sentry-logs/{socket.gethostname()}.log")
sentry_logger.addHandler(file_handler)

# Initializing sentry_sdk with debug=True
sentry_sdk.init(sentry_dsn, environment=environment, debug=True)

def main(msg: func.QueueMessage) -> None:

    try:
        raise Exception("Foo")
    except Exception as ex:
        sentry_sdk.capture_exception(ex)
        sentry_sdk.flush(10)

        raise ex

All I get are file names like SandboxHost-{number}.log with the exact following content:

Setting up integrations (with default = True)
Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
Enabling integration logging
Enabling integration stdlib
Enabling integration excepthook
Enabling integration dedupe
Enabling integration atexit
Enabling integration modules
Enabling integration argv
Enabling integration threading
Setting up integrations (with default = True)
Setting up integrations (with default = True)
Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
Enabling integration logging
Enabling integration logging
Enabling integration stdlib
Enabling integration stdlib
Enabling integration excepthook
Enabling integration excepthook
Enabling integration dedupe
Enabling integration dedupe
Enabling integration atexit
Enabling integration atexit
Enabling integration modules
Enabling integration modules
Enabling integration argv
Enabling integration argv
Enabling integration threading
Enabling integration threading

I can't waste any more time on this. Is there a way to force the SDK to just capture and send all exceptions, no matter which thread the thing is running on and no matter if that's going to block the execution for a while or degrade the performance or whatever. I just need to get the exceptions.

Thank you

[untitaker]

Okay, and sorry to see it's not working. Given what we've learned so far I think it's safest to initialize the SDK from within the function. Either that or:

sentry_sdk.init(...)
main_client = sentry_sdk.Hub.current

and then wrapping all of your functions in this:

with sentry_sdk.Hub(main_client):
   try:
       ...
   except Exception:
       sentry_sdk.capture_exception()

This does not disable thread-locality entirely. It rather explicitly forks off the client from the main one for each thread, such that set_context still sets data for the current function invocation only, not for the entire process. If this behavior also doesn't work for you, replace the with-stmt with with main_client:.

Given your experience we'll conduct follow-up work to fix the experience but I cannot give you any timelines for that.

[empz]

Thanks I'm going to try that. But first, I want to try something else. I think I know what's happening...

Azure Functions host framework (written in .NET Core) calls the function by using some kind of RPC (I think it's gRPC int the latest version). I think as soon as an uncaught exception is raised, the python process is ended and the host framework catches this inner exception and throws its own RpcException. Leaving no time for the process to flush the events to sentry.

This sample from the official Sentry SDK for Node Js Azure Function waits 2 seconds before returning.

"use strict";

const Sentry = require("@sentry/node");

Sentry.init({
  dsn: "https://examplePublicKey@o0.ingest.sentry.io/0",
});

module.exports = async function(context, req) {
  try {
    await notExistFunction();
  } catch (e) {
    Sentry.captureException(e);
    await Sentry.flush(2000);
  }

  context.res = {
    status: 200,
    body: "Hello from Azure Cloud Function!",
  };
};

The thing is that the Python SDK doesn't use await/async (I'm not familiar at all with await/async in Python). So there should some way to wait for flush as calling flush(10) doesn't seem to do anything.

[untitaker]

I think as soon as an uncaught exception is raised, the python process is ended

the question is, when does this happen? We explicitly catch and re-raise the exception, and the re-raise happens after flush. As you correctly deduced this basically means that flush does not sleep at all. I find that somewhat hard to believe unless Azure messes with the runtime a lot.

Perhaps replacing flush() with sleep(10) works then?

[empz]

Yeah, I thought about sleeping right after flush, but it doesn't work. I'm out of ideas here...

[empz]

with sentry_sdk.Hub(main_client): try: ... except Exception: sentry_sdk.capture_exception()

Sadly this is not working either for me..,.

Here's the full code of my function:

import logging
import socket
import os
import azure.functions as func

import sentry_sdk
from sentry_sdk.utils import logger as sentry_logger

# Sentry configuration
sentry_dsn = os.getenv("SENTRY_DSN")
environment = os.getenv("ENVIRONMENT", "PROD")

logs_path = os.getenv("LOGS_PATH", "logs")
file_handler = logging.FileHandler(
    f"{logs_path}/HttpTrigger/{socket.gethostname()}.log")
sentry_logger.addHandler(file_handler)

sentry_sdk.init(sentry_dsn, environment=environment, debug=True)
main_client = sentry_sdk.Hub.current

def main(req: func.HttpRequest) -> func.HttpResponse:
    with sentry_sdk.Hub(main_client):
        try:
            logging.info('Python HTTP trigger function processed a request.')

            raise Exception("Http Exception")

            # return func.HttpResponse(
            #     "This HTTP triggered function executed successfully.",
            #     status_code=200
            # )
        except Exception as ex:
            sentry_sdk.capture_exception()
            sentry_sdk.flush()

            return func.HttpResponse(
                str(ex),
                status_code=500
            )

What I don't understand is why I'm not getting all the logs on Azure even though I'm piping the sentry_logger directly to a FileStream. When running the function locally I get all the expected logs:

Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
Enabling integration logging
Enabling integration logging
Enabling integration stdlib
Enabling integration stdlib
Enabling integration excepthook
Enabling integration excepthook
Enabling integration dedupe
Enabling integration dedupe
Enabling integration atexit
Enabling integration atexit
Enabling integration modules
Enabling integration modules
Enabling integration argv
Enabling integration argv
Enabling integration threading
Enabling integration threading
Killing HTTP transport
Killing HTTP transport
background worker got kill request
background worker got kill request
Killing HTTP transport
Killing HTTP transport
background worker got kill request
background worker got kill request
Flushing HTTP transport
Flushing HTTP transport
background worker got flush request
background worker got flush request
Sending event, type:null level:error event_id:f3ad00f637e449b0a8f926b35fbfa126 project:5191540 host:o374073.ingest.sentry.io
Sending event, type:null level:error event_id:f3ad00f637e449b0a8f926b35fbfa126 project:5191540 host:o374073.ingest.sentry.io
0 event(s) pending on flush
0 event(s) pending on flush
background worker flushed
background worker flushed

But on Azure, only the integration logs are being written. Nothing about HTTP transport, background workers, sending events or flushing...

[untitaker]

@empz one other thing I suspect is that azure may shoot down threads spawned by user code. I have no way of verifying that at the moment.

sorry for the slow response

[untitaker]

@empz fyi we're going to have somebody look into this next week for real. Past weeks we were strapped for resources. Sorry again for slow responses and thanks for the patience so far.

[ahmedetefy]

Hey @empz Sorry for the late reply. I have a taken a look into this issue and I am unable to replicate the same issue.

I managed to create a QueueTrigger function with your initial code and it caught the exceptions, and the events were forwarded to Sentry as they should.

Here are the steps I took: 1- My initial thought was to create a HTTP Trigger function to test I followed this step-by-step tutorial https://docs.microsoft.com/en-us/azure/azure-functions/create-first-function-vs-code-python

and made minor modifications to your code to make it into a HTTP Trigger function Here is the code I used

import logging

import azure.functions as func
import sentry_sdk
from sentry_sdk.api import capture_exception, flush, push_scope
from sentry_sdk.integrations.serverless import serverless_function

# Sentry configuration
sentry_dsn = "your-dsn"
environment = "DEV"
logger = logging.getLogger(__name__)
sentry_sdk.init(
    sentry_dsn,
    environment=environment,
    send_default_pii=True,
    request_bodies="always",
    with_locals=True,
    debug=True
)
sentry_sdk.utils.MAX_STRING_LENGTH = 2048

@serverless_function
def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    msg = {
        "id": "e541a742-1646-4c50-1913-68f53f22f2ec",
        "dequeue_count": 5,
        "insertion_time": "2021-01-27T16:08:33.000000Z",
        "expiration_time":"2021-02-03T16:08:33.000000Z",
        "pop_receipt": "AgAAAAMAAABCBDFVAPCcPH8j01gE="
    }
    with push_scope() as scope:
        scope.set_tag("function.name", "ProcessHeadersFile")
        scope.set_context(
            "Queue Message",
            {
                "id": msg['id'],
                "dequeue_count": msg['dequeue_count'],
                "insertion_time": msg['insertion_time'],
                "expiration_time": msg['expiration_time'],
                "pop_receipt": msg['pop_receipt'],
            },
        )

        try:
            # code that might raise exceptions here
            function_that_raise()
        except Exception as ex:
            logging.error(ex)
            # Rethrow to fail the execution
            raise

def function_that_raise():
    return 5 / 0

Calling this HTTP Trigger function resulted in the exceptions being raised, caught and sent to Sentry as expected.

2- So I decided to try and replicate this on a Queue Trigger function and used your initial code snippet and it also worked. Here is the Azure QueueTrigger tutorial I followed: https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-storage-queue-triggered-function

and same code snippet

import logging
import os

import azure.functions as func
import sentry_sdk
from sentry_sdk.api import capture_exception, flush, push_scope
from sentry_sdk.integrations.serverless import serverless_function

# Sentry configuration
sentry_dsn = "my-dsn"
environment = "DEV"
logger = logging.getLogger(__name__)
sentry_sdk.init(
    sentry_dsn,
    environment=environment,
    send_default_pii=True,
    request_bodies="always",
    with_locals=True,
)
sentry_sdk.utils.MAX_STRING_LENGTH = 2048

@serverless_function
def main(msg: func.QueueMessage) -> None:

    with push_scope() as scope:
        scope.set_tag("function.name", "ProcessHeadersFile")
        scope.set_context(
            "Queue Message",
            {
                "id": msg.id,
                "dequeue_count": msg.dequeue_count,
                "insertion_time": msg.insertion_time,
                "expiration_time": msg.expiration_time,
                "pop_receipt": msg.pop_receipt,
            },
        )

        try:
            # code that might raise exceptions here
            function_that_raise()
        except Exception as ex:
            print(ex)
            # Rethrow to fail the execution
            raise

def function_that_raise():
    return 5 / 0

Upon publishing a message to the queue, the QueueTrigger function was called and the exception event was sent to Sentry

Please try to replicate this by following these steps and let me know if this issue persists

[empz]

@ahmedetefy Are you actually deploying these functions to Azure or just running them locally? Because the issue only happens when deployed to Azure. Locally everything works as expected.

[ahmedetefy]

@empz Yep the above comment was in reference to Azure deployed functions

[empz]

@ahmedetefy Could you share the function app settings? Windows or Linux? (or Docker?) Consumption or Premium plan? Region?

[ahmedetefy]

@empz I think by function app settings you probably mean this

{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "name": "msg",
      "type": "queueTrigger",
      "direction": "in",
      "queueName": "queue-items",
      "connection": "AzureWebJobsStorage"
    }
  ]
}

P.S. you also need to create a Queue that follows the same name as the "queueName" in this file, and after that you need to publish a message for your function to get triggered OS: Linux (default creation following the tutorials I referenced earlier) Plan: I was using the Azure free trial plan Region: West-Europe

Side note that might help, If the only logs you are seeing are the following

Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
Enabling integration logging
Enabling integration logging
Enabling integration stdlib
Enabling integration stdlib
Enabling integration excepthook
Enabling integration excepthook
Enabling integration dedupe
Enabling integration dedupe
Enabling integration atexit
Enabling integration atexit
Enabling integration modules
Enabling integration modules
Enabling integration argv
Enabling integration argv
Enabling integration threading
Enabling integration threading

There is a chance that your actual Queue Trigger function is not being triggered (since these logs only show initialization logs before your function is actually triggered) either due to not adding items on the queue or a mismatch between the naming of the queue your function app and the actual created queue

Hope this helps!

[empz]

@ahmedetefy I appreciate the thorough response, but I know how to use a Queue Trigger Azure Function and I know it's throwing exceptions because I can see the exceptions in ApplicationInsights. Sentry is not being able to catch it for some reason.

During the whole troubleshooting, there was times that Sentry was indeed able to catch it, but that didn't lasted.

I simply have no idea what's happening but I'm pretty sure is Azure's fault. I just wish there was a way to force capturing the exception via some explicit method call to avoid any weird stuff Azure might be doing...

[untitaker]

During the whole troubleshooting, there was times that Sentry was indeed able to catch it, but that didn't lasted.

can you elaborate on this? Like, did everything actually work 5 out of 100 times and if so, what was the log output for those 5 times?

[j-wil]

I'm running into the same issue when integrating sentry_sdk into a python azure function with an Eventhub trigger.

Here is my function:

from typing import List
from datetime import datetime
import requests
import logging
import json
import time
import os
import sys
import sentry_sdk
from sentry_sdk.api import capture_exception, flush, push_scope
from sentry_sdk.integrations.serverless import serverless_function
from sentry_sdk.utils import logger as sentry_logger

import azure.functions as func

sentry_sdk.init(
        sentry_dsn,
        environment="dev",
        send_default_pii=True,
        request_bodies="always",
        with_locals=True,
        debug=True
    )
fh = logging.FileHandler("sentry.log")
sentry_logger.addHandler(fh)

@serverless_function
def main(events: func.EventHubEvent):
    time.sleep(10)
    division_by_zero = 1 / 0

When I run locally here are the logs:

func host start
Found Python version 3.8.5 (python3).

Azure Functions Core Tools
Core Tools Version:       3.0.3568 Commit hash: e30a0ede85fd498199c28ad699ab2548593f759b  (64-bit)
Function Runtime Version: 3.0.15828.0

Functions:

        IoTConnect-stream-processor: eventHubTrigger

For detailed output, run func with --verbose flag.
[2021-08-02T18:42:49.349Z] Worker process started and initialized.
[2021-08-02T18:42:50.544Z]  [sentry] DEBUG: Setting up integrations (with default = True)
[2021-08-02T18:42:50.544Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
[2021-08-02T18:42:50.544Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
[2021-08-02T18:42:50.545Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
[2021-08-02T18:42:50.545Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
[2021-08-02T18:42:50.545Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
[2021-08-02T18:42:50.545Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
[2021-08-02T18:42:50.545Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
[2021-08-02T18:42:50.546Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
[2021-08-02T18:42:50.546Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
[2021-08-02T18:42:50.546Z]  [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
[2021-08-02T18:42:50.547Z]  [sentry] DEBUG: Setting up previously not enabled integration logging
[2021-08-02T18:42:50.547Z]  [sentry] DEBUG: Setting up previously not enabled integration stdlib
[2021-08-02T18:42:50.547Z]  [sentry] DEBUG: Setting up previously not enabled integration excepthook
[2021-08-02T18:42:50.547Z]  [sentry] DEBUG: Setting up previously not enabled integration dedupe
[2021-08-02T18:42:50.547Z]  [sentry] DEBUG: Setting up previously not enabled integration atexit
[2021-08-02T18:42:50.548Z]  [sentry] DEBUG: Setting up previously not enabled integration modules
[2021-08-02T18:42:50.548Z]  [sentry] DEBUG: Setting up previously not enabled integration argv
[2021-08-02T18:42:50.548Z]  [sentry] DEBUG: Setting up previously not enabled integration threading
[2021-08-02T18:42:50.548Z]  [sentry] DEBUG: Setting up previously not enabled integration sqlalchemy
[2021-08-02T18:42:50.548Z]  [sentry] DEBUG: Enabling integration logging
[2021-08-02T18:42:50.548Z]  [sentry] DEBUG: Enabling integration stdlib
[2021-08-02T18:42:50.548Z]  [sentry] DEBUG: Enabling integration excepthook
[2021-08-02T18:42:50.548Z]  [sentry] DEBUG: Enabling integration dedupe
[2021-08-02T18:42:50.549Z]  [sentry] DEBUG: Enabling integration atexit
[2021-08-02T18:42:50.549Z]  [sentry] DEBUG: Enabling integration modules
[2021-08-02T18:42:50.549Z]  [sentry] DEBUG: Enabling integration argv
[2021-08-02T18:42:50.549Z]  [sentry] DEBUG: Enabling integration threading
[2021-08-02T18:42:50.549Z]  [sentry] DEBUG: Enabling integration sqlalchemy
[2021-08-02T18:42:56.102Z] Host lock lease acquired by instance ID '000000000000000000000000AC82A56D'.
[2021-08-02T18:43:01.556Z] Executing 'Functions.IoTConnect-stream-processor' (Reason='(null)', Id=727083c5-f465-48aa-963e-79588a810ce6)
[2021-08-02T18:43:11.701Z]  [sentry] DEBUG: Flushing HTTP transport
[2021-08-02T18:43:11.701Z]  [sentry] DEBUG: Sending event, type:null level:error event_id:30c37f4c2da7481eacbef59582e2b4ec project:5887260 host:o352802.ingest.sentry.io
[2021-08-02T18:43:11.701Z]  [sentry] DEBUG: background worker got flush request
[2021-08-02T18:43:11.801Z]  [sentry] DEBUG: 1 event(s) pending on flush
[2021-08-02T18:43:11.810Z]  [sentry] DEBUG: background worker flushed
[2021-08-02T18:43:11.854Z] Executed 'Functions.IoTConnect-stream-processor' (Failed, Id=727083c5-f465-48aa-963e-79588a810ce6, Duration=10335ms)
[2021-08-02T18:43:11.854Z] System.Private.CoreLib: Exception while executing function: Functions.IoTConnect-stream-processor. System.Private.CoreLib: Result: Failure
[2021-08-02T18:43:11.854Z] Exception: ZeroDivisionError: division by zero
[2021-08-02T18:43:11.854Z] Stack:   File "/usr/lib/azure-functions-core-tools-3/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 398, in _handle__invocation_request
[2021-08-02T18:43:11.854Z]     call_result = await self._loop.run_in_executor(
[2021-08-02T18:43:11.854Z]   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
[2021-08-02T18:43:11.854Z]     result = self.fn(*self.args, **self.kwargs)
[2021-08-02T18:43:11.854Z]   File "/usr/lib/azure-functions-core-tools-3/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 602, in _run_sync_func
[2021-08-02T18:43:11.854Z]     return ExtensionManager.get_sync_invocation_wrapper(context,
[2021-08-02T18:43:11.854Z]   File "/usr/lib/azure-functions-core-tools-3/workers/python/3.8/LINUX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper
[2021-08-02T18:43:11.854Z]     result = function(**args)
[2021-08-02T18:43:11.854Z]   File "/home/jim/projects/Elementum/serverless/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/serverless.py", line 55, in inner
[2021-08-02T18:43:11.854Z]     _capture_and_reraise()
[2021-08-02T18:43:11.854Z]   File "/home/jim/projects/Elementum/serverless/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/serverless.py", line 80, in _capture_and_reraise
[2021-08-02T18:43:11.854Z]     reraise(*exc_info)
[2021-08-02T18:43:11.854Z]   File "/home/jim/projects/Elementum/serverless/.venv/lib/python3.8/site-packages/sentry_sdk/_compat.py", line 54, in reraise
[2021-08-02T18:43:11.854Z]     raise value
[2021-08-02T18:43:11.854Z]   File "/home/jim/projects/Elementum/serverless/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/serverless.py", line 53, in inner
[2021-08-02T18:43:11.854Z]     return f(*args, **kwargs)
[2021-08-02T18:43:11.854Z]   File "/home/jim/projects/Elementum/serverless/IoTConnect-stream-processor/__init__.py", line 37, in main
[2021-08-02T18:43:11.854Z]     division_by_zero = 1 / 0
[2021-08-02T18:43:11.854Z] .
[2021-08-02T18:43:12.084Z] Executing 'Functions.IoTConnect-stream-processor' (Reason='(null)', Id=4e92b955-a85d-4e06-94e4-b0602a2f19b9)

And this is the sentry.log contents

Flushing HTTP transport
background worker got flush request
Sending event, type:null level:error event_id:30c37f4c2da7481eacbef59582e2b4ec project:5887260 host:o352802.ingest.sentry.io
1 event(s) pending on flush
background worker flushed
Flushing HTTP transport
Sending event, type:null level:error event_id:29f144ae5fcc43a8a08bbd2b6c168470 project:5887260 host:o352802.ingest.sentry.io
background worker got flush request
background worker flushed
atexit: got shutdown signal
Sending event, type:null level:error event_id:9487ad4189d342b0b3eacbc259c411d5 project:5887260 host:o352802.ingest.sentry.io
atexit: shutting down client
Flushing HTTP transport
background worker got flush request
background worker flushed
Killing HTTP transport
background worker got kill request

You can see the event getting captured but it never shows up in sentry.

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[antonpirker]

I think of putting this into the internal backlog with very low priority, because this seems like a very hard problem to reproduce or debug.

@empz as a lot of time passed, have you managed to fix this problem by any chance?

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀