search

Azure / azure-functions-python-worker

Python worker for Azure Functions.
http://aka.ms/azurefunctions
MIT License
336 stars 104 forks source link

[Bug] start_from_time parameter isn't working with Cosmos DB trigger #1610

Open syberen opened 2 weeks ago

syberen commented 2 weeks ago

Expected Behavior

The function is triggered with documents added from specified date and time

Actual Behavior

First of all, it is unclear what the type is of the start_from_time parameter (linked issue). Depending on the type of input a different error message occurs.

When I try a string i.e. "2023-06-01T00:00:00Z" the following error occurs:

The listener for function 'Functions.risicoscores_silver_calculation_models_producer' was unable to start. Microsoft.Azure.Cosmos.Client: StartTime cannot have DateTimeKind.Unspecified (Parameter 'value').

When I try a datetime i.e. datetime.fromisoformat("2023-06-01T00:00:00Z"), the following error occurs:

Worker failed to index functions
[2024-11-12T12:01:43.006Z] Result: Failure
[2024-11-12T12:01:43.006Z] Exception: TypeError: Object of type datetime is not JSON serializable
[2024-11-12T12:01:43.006Z] Stack:   File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure_functions_worker/dispatcher.py", line 345, in _handle__functions_metadata_request
[2024-11-12T12:01:43.006Z]     fx_metadata_results = self.index_functions(function_path)
[2024-11-12T12:01:43.006Z]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-12T12:01:43.006Z]   File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure_functions_worker/dispatcher.py", line 622, in index_functions
[2024-11-12T12:01:43.007Z]     fx_metadata_results = loader.process_indexed_function(
[2024-11-12T12:01:43.007Z]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-12T12:01:43.007Z]   File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure_functions_worker/loader.py", line 142, in process_indexed_function
[2024-11-12T12:01:43.007Z]     raw_bindings=indexed_function.get_raw_bindings(),
[2024-11-12T12:01:43.007Z]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-12T12:01:43.007Z]   File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure/functions/decorators/function_app.py", line 161, in get_raw_bindings
[2024-11-12T12:01:43.007Z]     return [json.dumps(b.get_dict_repr(), cls=StringifyEnumJsonEncoder)
[2024-11-12T12:01:43.007Z]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-12T12:01:43.007Z]   File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure/functions/decorators/function_app.py", line 161, in <listcomp>
[2024-11-12T12:01:43.007Z]     return [json.dumps(b.get_dict_repr(), cls=StringifyEnumJsonEncoder)
[2024-11-12T12:01:43.007Z]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-12T12:01:43.007Z]   File "/usr/lib/python3.11/json/__init__.py", line 238, in dumps
[2024-11-12T12:01:43.007Z]     **kw).encode(obj)
[2024-11-12T12:01:43.008Z]           ^^^^^^^^^^^
[2024-11-12T12:01:43.008Z]   File "/usr/lib/python3.11/json/encoder.py", line 200, in encode
[2024-11-12T12:01:43.008Z]     chunks = self.iterencode(o, _one_shot=True)
[2024-11-12T12:01:43.008Z]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-12T12:01:43.008Z]   File "/usr/lib/python3.11/json/encoder.py", line 258, in iterencode
[2024-11-12T12:01:43.008Z]     return _iterencode(o, 0)
[2024-11-12T12:01:43.008Z]            ^^^^^^^^^^^^^^^^^
[2024-11-12T12:01:43.008Z]   File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure/functions/decorators/utils.py", line 175, in default
[2024-11-12T12:01:43.008Z]     return super().default(o)
[2024-11-12T12:01:43.008Z]            ^^^^^^^^^^^^^^^^^^
[2024-11-12T12:01:43.008Z]   File "/usr/lib/python3.11/json/encoder.py", line 180, in default
[2024-11-12T12:01:43.008Z]     raise TypeError(f'Object of type {o.__class__.__name__} '

Steps to Reproduce

  1. Create an azure function with a cosmos db trigger
  2. Add a start_from_time parameter to the trigger decorator
  3. Start the function

Relevant code being tried

# Either
trigger_start_time = datetime.fromisoformat("2023-06-01T00:00:00Z") 
# Or
trigger_start_time = "2023-06-01T00:00:00Z"

@app.cosmos_db_trigger(
    arg_name="redacted",
    container_name="redacted",
    database_name="redacted",
    connection="redacted",
    lease_container_name="ChangeFeed_Leases",
    max_items_per_invocation=200,
    start_from_time=trigger_start_time,  # type: ignore
    lease_container_prefix="redacted"
    ),
)

Relevant log output

requirements.txt file

Where are you facing this problem?

Local - Core Tools

Function app name

No response

Additional Information

No response

hallvictoria commented 1 week ago

The type is supposed to be str. Can you try using slashes instead of dashes? For example: "2024/11/19T00:00:00Z"

I was able to repro that issue with both V1 and V2 models, so it doesn't look like it's an issue with the decorators. We don't do any parsing or altering of the value passed in the decorator, so it seems like the extension is expecting a particular format. A Node function app works similarly here.

Tbf, I'm not sure why the slashes would be the expected format here. If this also works for you, we can update the docs to mention this. If this format is difficult, we can look into adding some logic in the decorators that will convert a start_from_time input value of "2024-11-19T00:00:00Z" into "2024/11/19T00:00:00Z", but I'd have to confirm with the CosmosDB extension team what the expected format is.

syberen commented 1 week ago

Hi, thanks for you response. Switching to slashes does seem to work indeed!