search

Azure / azure-functions-durable-python

Python library for using the Durable Functions bindings.
MIT License
134 stars 54 forks source link

Eternal orchestration doesn't get cleaned up #514

Open rokcarl opened 2 months ago

rokcarl commented 2 months ago

I'm trying to create an eternal orchestration, as per the docs. When I triggered the orchestration, the eternal runs started. But then they never stop. I can understand that they don't restart after a restart of the app, but even when I change the methods, the eternal run is continued:

[2024-07-05T10:00:48.996Z] The function 'cleanup_orchestrator' doesn't exist, is disabled, or is not an orchestrator function. Additional info: The following are the known orchestrator functions: 'cleanup_orchestrator2'.. InstanceId: 18682d4729e348708b55528067f65d45. Function: cleanup_orchestrator. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 3.

I ran this by having the code below and running func start, running the http trigger (using curl), then stopping, renaming the function and running func start again.

function_app.py ```python import datetime import logging import azure.durable_functions as df import azure.functions as func app = df.DFApp(http_auth_level=func.AuthLevel.FUNCTION) @app.route(route="cleanup") @app.durable_client_input(client_name="client") async def cleanup_http(req: func.HttpRequest, client: df.DurableOrchestrationClient) -> func.HttpResponse: logging.info("[cleanup] http trigger") instance_id = await client.start_new("cleanup_orchestrator", instance_id="cleanupv4", client_input={}) return client.create_check_status_response(req, instance_id) @app.orchestration_trigger(context_name="context") def cleanup_orchestrator(context: df.DurableOrchestrationContext): logging.info("[cleanup] orchestrator called") yield context.call_activity("cleanup") logging.info("[cleanup] activity called") next_run = context.current_utc_datetime + datetime.timedelta(minutes=1) yield context.create_timer(next_run) logging.info("[cleanup] timer created") context.continue_as_new(None) # reset history @app.activity_trigger(input_name="inputs") def cleanup(inputs: dict): logging.info("[cleanup] work started and done") ```
full log ``` ❯ func start Found Python version 3.10.10 (python3). Azure Functions Core Tools Core Tools Version: 4.0.5801 Commit hash: N/A +5ac2f09758b98257e728dd1b5576ce5ea9ef68ff (64-bit) Function Runtime Version: 4.34.1.22669 [2024-07-05T10:00:42.915Z] Worker process started and initialized. Functions: cleanup_http: http://localhost:7071/api/cleanup cleanup2: activityTrigger cleanup_orchestrator2: orchestrationTrigger For detailed output, run func with --verbose flag. [2024-07-05T10:00:47.874Z] Host lock lease acquired by instance ID '0000000000000000000000001DA5AB02'. [2024-07-05T10:00:48.996Z] The function 'cleanup_orchestrator' doesn't exist, is disabled, or is not an orchestrator function. Additional info: The following are the known orchestrator functions: 'cleanup_orchestrator2'.. InstanceId: 18682d4729e348708b55528067f65d45. Function: cleanup_orchestrator. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 3. ```

I would expect that upon running func start, any eternal orchestrator that needs to run, would first check if the run is still needed.

You might think, no big deal, but if I change my code a bit to this and re-run func start, I get this orchestrator run three times, and as the instance ID tells me, it's three different versions, two of them phantom ones, long gone from my code (check for cleanupv4, cleanupv5 and cleanupv6 in the below logs):

new logs ``` ❯ func start Found Python version 3.10.10 (python3). Azure Functions Core Tools Core Tools Version: 4.0.5801 Commit hash: N/A +5ac2f09758b98257e728dd1b5576ce5ea9ef68ff (64-bit) Function Runtime Version: 4.34.1.22669 [2024-07-05T10:54:36.804Z] Worker process started and initialized. Functions: cleanup: activityTrigger cleanup_orchestrator: orchestrationTrigger timer: timerTrigger For detailed output, run func with --verbose flag. [2024-07-05T10:54:37.370Z] Executing 'Functions.timer' (Reason='Timer fired at 2024-07-05T12:54:37.2748780+02:00', Id=447c426e-0734-4185-93ed-2593f013ed8b) [2024-07-05T10:54:37.375Z] Trigger Details: UnscheduledInvocationReason: RunOnStartup [2024-07-05T10:54:37.679Z] [cleanup] timer [2024-07-05T10:54:37.952Z] Executed 'Functions.timer' (Failed, Id=447c426e-0734-4185-93ed-2593f013ed8b, Duration=663ms) [2024-07-05T10:54:37.952Z] System.Private.CoreLib: Exception while executing function: Functions.timer. System.Private.CoreLib: Result: Failure [2024-07-05T10:54:37.952Z] Exception: Exception: {'Message': 'Something went wrong while processing your request', 'ExceptionMessage': 'An Orchestration instance with the status Running already exists.', 'ExceptionType': 'System.InvalidOperationException', 'StackTrace': ' at DurableTask.AzureStorage.AzureStorageOrchestrationService.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1693\n at DurableTask.Core.TaskHubClient.InternalCreateOrchestrationInstanceWithRaisedEventAsync(String orchestrationName, String orchestrationVersion, String orchestrationInstanceId, Object orchestrationInput, IDictionary`2 orchestrationTags, OrchestrationStatus[] dedupeStatuses, String eventName, Object eventData, Nullable`1 startAt) in /_/src/DurableTask.Core/TaskHubClient.cs:line 645\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient.Microsoft.Azure.WebJobs.Extensions.DurableTask.IDurableOrchestrationClient.StartNewAsync[T](String orchestratorFunctionName, String instanceId, T input) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\ContextImplementations\\DurableClient.cs:line 215\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleStartOrchestratorRequestAsync(HttpRequestMessage request, String functionName, String instanceId) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 875\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleRequestAsync(HttpRequestMessage request) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 306'} [2024-07-05T10:54:37.952Z] Stack: File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/dispatcher.py", line 602, in _handle__invocation_request [2024-07-05T10:54:37.952Z] await self._run_async_func(fi_context, fi.func, args) [2024-07-05T10:54:37.952Z] File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/dispatcher.py", line 939, in _run_async_func [2024-07-05T10:54:37.952Z] return await ExtensionManager.get_async_invocation_wrapper( [2024-07-05T10:54:37.952Z] File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5801/workers/python/3.10/OSX/X64/azure_functions_worker/extension.py", line 147, in get_async_invocation_wrapper [2024-07-05T10:54:37.952Z] result = await function(**args) [2024-07-05T10:54:37.952Z] File "/Users/rokcarl/delo/enpal/automation-maltrix-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/decorators/durable_app.py", line 199, in df_client_middleware [2024-07-05T10:54:37.952Z] return await user_code(*args, **kwargs) [2024-07-05T10:54:37.952Z] File "/Users/rokcarl/delo/enpal/automation-maltrix-v2/code/function_app.py", line 25, in cleanup_timer [2024-07-05T10:54:37.952Z] await client.start_new("cleanup_orchestrator", instance_id=instance_id, client_input={}) [2024-07-05T10:54:37.952Z] File "/Users/rokcarl/delo/enpal/automation-maltrix-v2/code/.venv/lib/python3.10/site-packages/azure/durable_functions/models/DurableOrchestrationClient.py", line 89, in start_new [2024-07-05T10:54:37.952Z] raise Exception(ex_message) [2024-07-05T10:54:37.952Z] . [2024-07-05T10:54:41.691Z] Host lock lease acquired by instance ID '0000000000000000000000001DA5AB02'. [2024-07-05T10:54:43.128Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=0cc81ec8-68ae-4cb2-9576-02d87b1dd8cb) [2024-07-05T10:54:43.128Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=f6a862b0-1449-47d5-8846-7fc0fdf43f5d) [2024-07-05T10:54:43.187Z] [cleanup] timer created [2024-07-05T10:54:43.188Z] [cleanup] orchestrator called, instance id cleanupv6 [2024-07-05T10:54:43.188Z] [cleanup] activity called [2024-07-05T10:54:43.191Z] [cleanup] orchestrator called, instance id cleanupv5 [2024-07-05T10:54:43.191Z] [cleanup] activity called [2024-07-05T10:54:43.191Z] [cleanup] timer created [2024-07-05T10:54:43.298Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=0cc81ec8-68ae-4cb2-9576-02d87b1dd8cb, Duration=197ms) [2024-07-05T10:54:43.298Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=f6a862b0-1449-47d5-8846-7fc0fdf43f5d, Duration=197ms) [2024-07-05T10:54:43.328Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=0ec5023b-1937-4161-a793-b22a03418f86) [2024-07-05T10:54:43.328Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=bd5e0619-1d13-4095-b18e-ef60f112514f) [2024-07-05T10:54:43.367Z] [cleanup] orchestrator called, instance id cleanupv6 [2024-07-05T10:54:43.367Z] [cleanup] orchestrator called, instance id cleanupv5 [2024-07-05T10:54:43.377Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=0ec5023b-1937-4161-a793-b22a03418f86, Duration=51ms) [2024-07-05T10:54:43.377Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=bd5e0619-1d13-4095-b18e-ef60f112514f, Duration=51ms) [2024-07-05T10:54:43.461Z] Executing 'Functions.cleanup' (Reason='(null)', Id=57cc715b-53cb-4174-80e4-9afce77b73c0) [2024-07-05T10:54:43.461Z] Executing 'Functions.cleanup' (Reason='(null)', Id=0826481f-9eb0-4428-8d37-63b45a66f98a) [2024-07-05T10:54:43.468Z] [cleanup] work started and done [2024-07-05T10:54:43.469Z] [cleanup] work started and done [2024-07-05T10:54:43.472Z] Executed 'Functions.cleanup' (Succeeded, Id=57cc715b-53cb-4174-80e4-9afce77b73c0, Duration=15ms) [2024-07-05T10:54:43.472Z] Executed 'Functions.cleanup' (Succeeded, Id=0826481f-9eb0-4428-8d37-63b45a66f98a, Duration=15ms) [2024-07-05T10:54:43.560Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=0320adec-17d3-4dd4-a32b-6c51e3a50fbc) [2024-07-05T10:54:43.560Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=fbd89619-6134-4dee-9138-e13040d4cd46) [2024-07-05T10:54:43.567Z] [cleanup] orchestrator called, instance id cleanupv6 [2024-07-05T10:54:43.567Z] [cleanup] activity called [2024-07-05T10:54:43.567Z] [cleanup] orchestrator called, instance id cleanupv5 [2024-07-05T10:54:43.567Z] [cleanup] activity called [2024-07-05T10:54:43.572Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=fbd89619-6134-4dee-9138-e13040d4cd46, Duration=13ms) [2024-07-05T10:54:43.572Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=0320adec-17d3-4dd4-a32b-6c51e3a50fbc, Duration=13ms) [2024-07-05T10:55:08.512Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=b8277603-a4ba-4266-aa4a-df53ba189866) [2024-07-05T10:55:08.603Z] [cleanup] orchestrator called, instance id cleanupv4 [2024-07-05T10:55:08.603Z] [cleanup] timer created [2024-07-05T10:55:08.604Z] [cleanup] activity called [2024-07-05T10:55:08.635Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=b8277603-a4ba-4266-aa4a-df53ba189866, Duration=174ms) [2024-07-05T10:55:08.650Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=aa9097e7-5d47-44a5-adf2-d5319662e734) [2024-07-05T10:55:08.659Z] [cleanup] orchestrator called, instance id cleanupv4 [2024-07-05T10:55:08.663Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=aa9097e7-5d47-44a5-adf2-d5319662e734, Duration=14ms) [2024-07-05T10:55:08.704Z] Executing 'Functions.cleanup' (Reason='(null)', Id=9acdb8ea-b1fc-4636-b232-ee50f918d57e) [2024-07-05T10:55:08.712Z] [cleanup] work started and done [2024-07-05T10:55:08.714Z] Executed 'Functions.cleanup' (Succeeded, Id=9acdb8ea-b1fc-4636-b232-ee50f918d57e, Duration=12ms) [2024-07-05T10:55:08.742Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=74a74890-111b-4b15-b8e8-822caf213e19) [2024-07-05T10:55:08.746Z] [cleanup] orchestrator called, instance id cleanupv4 [2024-07-05T10:55:08.746Z] [cleanup] activity called [2024-07-05T10:55:08.748Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=74a74890-111b-4b15-b8e8-822caf213e19, Duration=6ms) [2024-07-05T10:56:02.654Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=998dc337-9b0b-442a-bcb6-70d5082fb872) [2024-07-05T10:56:02.672Z] [cleanup] orchestrator called, instance id cleanupv6 [2024-07-05T10:56:02.672Z] [cleanup] timer created [2024-07-05T10:56:02.672Z] [cleanup] activity called [2024-07-05T10:56:02.680Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=998dc337-9b0b-442a-bcb6-70d5082fb872, Duration=29ms) [2024-07-05T10:56:02.684Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=bb62b856-7092-450f-b405-2b99b79487a4) [2024-07-05T10:56:02.686Z] [cleanup] orchestrator called, instance id cleanupv6 [2024-07-05T10:56:02.687Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=bb62b856-7092-450f-b405-2b99b79487a4, Duration=3ms) [2024-07-05T10:56:02.705Z] Executing 'Functions.cleanup' (Reason='(null)', Id=a40a1be0-af13-4cbf-a4e5-6348a21b5104) [2024-07-05T10:56:02.708Z] [cleanup] work started and done [2024-07-05T10:56:02.709Z] Executed 'Functions.cleanup' (Succeeded, Id=a40a1be0-af13-4cbf-a4e5-6348a21b5104, Duration=3ms) [2024-07-05T10:56:02.721Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=2b16e029-0ed1-4d1f-9876-1ca725e31ae5) [2024-07-05T10:56:02.723Z] [cleanup] orchestrator called, instance id cleanupv6 [2024-07-05T10:56:02.723Z] [cleanup] activity called [2024-07-05T10:56:02.724Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=2b16e029-0ed1-4d1f-9876-1ca725e31ae5, Duration=3ms) [2024-07-05T10:56:02.959Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=88b65d47-8662-4cdd-9f9d-725c6e3212c8) [2024-07-05T10:56:02.968Z] [cleanup] timer created [2024-07-05T10:56:02.968Z] [cleanup] activity called [2024-07-05T10:56:02.968Z] [cleanup] orchestrator called, instance id cleanupv5 [2024-07-05T10:56:02.971Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=88b65d47-8662-4cdd-9f9d-725c6e3212c8, Duration=14ms) [2024-07-05T10:56:02.973Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=9cf8b471-9d20-4703-a857-8cf08f659ff0) [2024-07-05T10:56:02.976Z] [cleanup] orchestrator called, instance id cleanupv5 [2024-07-05T10:56:02.976Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=9cf8b471-9d20-4703-a857-8cf08f659ff0, Duration=3ms) [2024-07-05T10:56:02.987Z] Executing 'Functions.cleanup' (Reason='(null)', Id=8324bc64-2e37-4f04-ad4c-c45b7b00759d) [2024-07-05T10:56:02.988Z] [cleanup] work started and done [2024-07-05T10:56:02.989Z] Executed 'Functions.cleanup' (Succeeded, Id=8324bc64-2e37-4f04-ad4c-c45b7b00759d, Duration=2ms) [2024-07-05T10:56:02.998Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=b6897950-d935-419c-958f-e7a17c8f3cb0) [2024-07-05T10:56:03.001Z] [cleanup] orchestrator called, instance id cleanupv5 [2024-07-05T10:56:03.001Z] [cleanup] activity called [2024-07-05T10:56:03.002Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=b6897950-d935-419c-958f-e7a17c8f3cb0, Duration=3ms) [2024-07-05T10:56:28.610Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=f012c1ac-e5b7-423c-8275-51e11a8208c1) [2024-07-05T10:56:28.628Z] [cleanup] activity called [2024-07-05T10:56:28.628Z] [cleanup] orchestrator called, instance id cleanupv4 [2024-07-05T10:56:28.629Z] [cleanup] timer created [2024-07-05T10:56:28.635Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=f012c1ac-e5b7-423c-8275-51e11a8208c1, Duration=29ms) [2024-07-05T10:56:28.638Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=be518ff8-65d5-42dd-8efd-b859973a0dec) [2024-07-05T10:56:28.644Z] [cleanup] orchestrator called, instance id cleanupv4 [2024-07-05T10:56:28.647Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=be518ff8-65d5-42dd-8efd-b859973a0dec, Duration=9ms) [2024-07-05T10:56:28.668Z] Executing 'Functions.cleanup' (Reason='(null)', Id=da40d417-52fd-41d8-ac1d-5b947250b6ce) [2024-07-05T10:56:28.671Z] [cleanup] work started and done [2024-07-05T10:56:28.672Z] Executed 'Functions.cleanup' (Succeeded, Id=da40d417-52fd-41d8-ac1d-5b947250b6ce, Duration=6ms) [2024-07-05T10:56:28.683Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=c9d19a43-9426-47f1-9f92-55003712d7d8) [2024-07-05T10:56:28.687Z] [cleanup] orchestrator called, instance id cleanupv4 [2024-07-05T10:56:28.687Z] [cleanup] activity called [2024-07-05T10:56:28.689Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=c9d19a43-9426-47f1-9f92-55003712d7d8, Duration=5ms) [2024-07-05T10:57:28.631Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=b63c03cd-53dc-4a19-b0ff-70a7f2330b56) [2024-07-05T10:57:28.637Z] [cleanup] orchestrator called, instance id cleanupv6 [2024-07-05T10:57:28.637Z] [cleanup] activity called [2024-07-05T10:57:28.637Z] [cleanup] timer created [2024-07-05T10:57:28.641Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=b63c03cd-53dc-4a19-b0ff-70a7f2330b56, Duration=10ms) [2024-07-05T10:57:28.644Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=a287568e-a376-4b3d-8a5f-3324b7ffc4b3) [2024-07-05T10:57:28.648Z] [cleanup] orchestrator called, instance id cleanupv6 [2024-07-05T10:57:28.649Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=a287568e-a376-4b3d-8a5f-3324b7ffc4b3, Duration=6ms) [2024-07-05T10:57:28.665Z] Executing 'Functions.cleanup' (Reason='(null)', Id=25a444fc-b52e-410f-a93a-5c1cb0311bcc) [2024-07-05T10:57:28.668Z] [cleanup] work started and done [2024-07-05T10:57:28.669Z] Executed 'Functions.cleanup' (Succeeded, Id=25a444fc-b52e-410f-a93a-5c1cb0311bcc, Duration=4ms) [2024-07-05T10:57:28.688Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=46e3d9ed-911a-4d6b-83be-e66405c9a40a) [2024-07-05T10:57:28.693Z] [cleanup] activity called [2024-07-05T10:57:28.693Z] [cleanup] orchestrator called, instance id cleanupv6 [2024-07-05T10:57:28.694Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=46e3d9ed-911a-4d6b-83be-e66405c9a40a, Duration=8ms) [2024-07-05T10:57:29.246Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=b7f945fb-5f2f-418d-b2bb-11c07b20260c) [2024-07-05T10:57:29.261Z] [cleanup] orchestrator called, instance id cleanupv5 [2024-07-05T10:57:29.261Z] [cleanup] activity called [2024-07-05T10:57:29.261Z] [cleanup] timer created [2024-07-05T10:57:29.265Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=b7f945fb-5f2f-418d-b2bb-11c07b20260c, Duration=24ms) [2024-07-05T10:57:29.271Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=20c5b024-16dc-4a21-a102-c0423770677e) [2024-07-05T10:57:29.277Z] [cleanup] orchestrator called, instance id cleanupv5 [2024-07-05T10:57:29.279Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=20c5b024-16dc-4a21-a102-c0423770677e, Duration=10ms) [2024-07-05T10:57:29.295Z] Executing 'Functions.cleanup' (Reason='(null)', Id=1303ea47-6f9d-4a41-b199-ba07ff9f3abe) [2024-07-05T10:57:29.309Z] [cleanup] work started and done [2024-07-05T10:57:29.315Z] Executed 'Functions.cleanup' (Succeeded, Id=1303ea47-6f9d-4a41-b199-ba07ff9f3abe, Duration=21ms) [2024-07-05T10:57:29.342Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=570af916-7afc-492c-87d7-47735971bfb3) [2024-07-05T10:57:29.348Z] [cleanup] orchestrator called, instance id cleanupv5 [2024-07-05T10:57:29.348Z] [cleanup] activity called [2024-07-05T10:57:29.349Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=570af916-7afc-492c-87d7-47735971bfb3, Duration=7ms) [2024-07-05T10:57:52.989Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=d3121e15-931c-4a15-be2e-87caa0408c70) [2024-07-05T10:57:53.003Z] [cleanup] orchestrator called, instance id cleanupv4 [2024-07-05T10:57:53.003Z] [cleanup] activity called [2024-07-05T10:57:53.003Z] [cleanup] timer created [2024-07-05T10:57:53.008Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=d3121e15-931c-4a15-be2e-87caa0408c70, Duration=21ms) [2024-07-05T10:57:53.011Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=ffd42a52-68bc-490b-a23e-02510945844c) [2024-07-05T10:57:53.017Z] [cleanup] orchestrator called, instance id cleanupv4 [2024-07-05T10:57:53.017Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=ffd42a52-68bc-490b-a23e-02510945844c, Duration=6ms) [2024-07-05T10:57:53.037Z] Executing 'Functions.cleanup' (Reason='(null)', Id=d450e991-7643-4ad1-aac8-59012fab7cd9) [2024-07-05T10:57:53.043Z] [cleanup] work started and done [2024-07-05T10:57:53.045Z] Executed 'Functions.cleanup' (Succeeded, Id=d450e991-7643-4ad1-aac8-59012fab7cd9, Duration=10ms) [2024-07-05T10:57:53.067Z] Executing 'Functions.cleanup_orchestrator' (Reason='(null)', Id=10441fb1-de52-41a1-9b3b-cef5d343b1c9) [2024-07-05T10:57:53.072Z] [cleanup] orchestrator called, instance id cleanupv4 [2024-07-05T10:57:53.072Z] [cleanup] activity called [2024-07-05T10:57:53.075Z] Executed 'Functions.cleanup_orchestrator' (Succeeded, Id=10441fb1-de52-41a1-9b3b-cef5d343b1c9, Duration=8ms) ```
cgillum commented 2 months ago

Eternal orchestrations will run forever or until you terminate them. Creating a new instance of the same orchestration with a different ID will just create another eternal orchestration instance that runs side-by-side with any existing eternal instances.

My suggestion would be to update your cleanup_http function to check for the old versions of this eternal orchestration and then terminate them if they exist. That will help ensure that you only have one eternal orchestration running at a time.

rokcarl commented 1 month ago

Yes, obviously.

It's possible to manually stop/remove phantom orchestrations, but first off, it's weird & not ideal. It's like pressing a button that is no longer there and then manually removing the action of pressing that button. Second, how would the SDLC look like? Imagine having a cleanup that needs to run all the time. First you add code that runs it. Then you change how the cleanup works and how often it works, what do you do then? And what do you do if someone deletes the code with this cleanup?

While I consider this workflow far from ideal, do you have any code examples of managing this?