search

Azure / Azure-Sentinel

Cloud-native SIEM for intelligent security analytics for your entire enterprise.
https://azure.microsoft.com/en-us/services/azure-sentinel/
MIT License
4.62k stars 3.03k forks source link

Trend Micro Vision One (Function App) - ModuleNotFoundError #10622

Closed hgtok closed 5 months ago

hgtok commented 5 months ago

Describe the bug Function App failing with ModuleNotFoundError: No module named '_cffi_backend'.

To Reproduce Steps to reproduce the behavior:

  1. Deploy Trend Micro Vision One (Function App) Connector
  2. Navigate to Function App
  3. Click on 'Log stream'
  4. Monitor for 5 - 10 mins
  5. See error

Expected behavior No errors and Trend Micro Vision One logs are ingested into Sentinel

Additional context ModuleNotFoundError is one of the first errors. Multiple errors as attached.

Log stream 
2024-06-07T01:59:15Z   [Information]   The next 5 occurrences of the 'timer_trigger_oat' schedule (Cron: '0 0,5,10,15,20,25,30,35,40,45,50,55 * * * *') will be:
06/07/2024 02:00:00+00:00 (06/07/2024 02:00:00Z)
06/07/2024 02:05:00+00:00 (06/07/2024 02:05:00Z)
06/07/2024 02:10:00+00:00 (06/07/2024 02:10:00Z)
06/07/2024 02:15:00+00:00 (06/07/2024 02:15:00Z)
06/07/2024 02:20:00+00:00 (06/07/2024 02:20:00Z)
2024-06-07T01:59:15Z   [Verbose]   Timer for 'timer_trigger_oat' started with interval '00:00:44.9950583'.
2024-06-07T01:59:15Z   [Information]   The next 5 occurrences of the 'timer_trigger' schedule (Cron: '0 0,5,10,15,20,25,30,35,40,45,50,55 * * * *') will be:
06/07/2024 02:00:00+00:00 (06/07/2024 02:00:00Z)
06/07/2024 02:05:00+00:00 (06/07/2024 02:05:00Z)
06/07/2024 02:10:00+00:00 (06/07/2024 02:10:00Z)
06/07/2024 02:15:00+00:00 (06/07/2024 02:15:00Z)
06/07/2024 02:20:00+00:00 (06/07/2024 02:20:00Z)
2024-06-07T01:59:15Z   [Verbose]   Timer for 'timer_trigger' started with interval '00:00:44.9894566'.
2024-06-07T01:59:15Z   [Verbose]   Timer listener started (timer_trigger)
2024-06-07T01:59:15Z   [Verbose]   Timer listener started (timer_trigger_oat)
2024-06-07T01:59:23Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-07T01:59:23Z   [Verbose]   [HostMonitor] Worker status: ID=ca1825f7-60e9-4e18-a557-02d5885316b8, Latency=1ms
2024-06-07T01:59:23Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 87): History=(0,0,0,0,0), AvgCpuLoad=0, MaxCpuLoad=0
2024-06-07T01:59:23Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 61): History=(1,0,2,0,1), AvgCpuLoad=1, MaxCpuLoad=2
2024-06-07T01:59:23Z   [Verbose]   [HostMonitor] Host aggregate CPU load 1
2024-06-07T01:59:23Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-07T01:59:37Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-07T01:59:37Z   [Verbose]   [HostMonitor] Worker status: ID=ca1825f7-60e9-4e18-a557-02d5885316b8, Latency=1ms
2024-06-07T01:59:37Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 87): History=(0,0,0,1,0), AvgCpuLoad=0.2, MaxCpuLoad=1
2024-06-07T01:59:37Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 61): History=(0,0,1,1,1), AvgCpuLoad=1, MaxCpuLoad=1
2024-06-07T01:59:37Z   [Verbose]   [HostMonitor] Host aggregate CPU load 1
2024-06-07T01:59:37Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-07T01:59:44Z   [Verbose]   Function 'oat_pipeline_task_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue'.
2024-06-07T01:59:50Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-07T01:59:50Z   [Verbose]   [HostMonitor] Worker status: ID=ca1825f7-60e9-4e18-a557-02d5885316b8, Latency=6ms
2024-06-07T01:59:50Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 87): History=(0,0,0,0,0), AvgCpuLoad=0, MaxCpuLoad=0
2024-06-07T01:59:50Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 61): History=(2,1,0,0,2), AvgCpuLoad=1, MaxCpuLoad=2
2024-06-07T01:59:50Z   [Verbose]   [HostMonitor] Host aggregate CPU load 1
2024-06-07T01:59:50Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-07T01:59:51Z   [Verbose]   Function 'queue_trigger_rca' will wait 60000 ms before polling queue 'rca-queue'.
2024-06-07T01:59:56Z   [Verbose]   Function 'oat_pipeline_file_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue-poison'.
2024-06-07T01:59:59Z   [Verbose]   Function 'oat_pipeline_file_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue'.
2024-06-07T02:00:00Z   [Verbose]   Function 'queue_trigger_wb_poison' will wait 60000 ms before polling queue 'workbench-queue-poison'.
2024-06-07T02:00:00Z   [Information]   Executing 'Functions.timer_trigger_oat' (Reason='Timer fired at 2024-06-07T02:00:00.0012937+00:00', Id=32b3764d-dd2b-44c7-9657-dd7ddfd09462)
2024-06-07T02:00:00Z   [Verbose]   Sending invocation id: '32b3764d-dd2b-44c7-9657-dd7ddfd09462
2024-06-07T02:00:00Z   [Verbose]   Posting invocation id:32b3764d-dd2b-44c7-9657-dd7ddfd09462 on workerId:ca1825f7-60e9-4e18-a557-02d5885316b8
2024-06-07T02:00:00Z   [Information]   Executing 'Functions.timer_trigger' (Reason='Timer fired at 2024-06-07T02:00:00.0009762+00:00', Id=fe049497-b777-4e1c-a106-95125a26832d)
2024-06-07T02:00:00Z   [Verbose]   Sending invocation id: 'fe049497-b777-4e1c-a106-95125a26832d
2024-06-07T02:00:00Z   [Verbose]   Posting invocation id:fe049497-b777-4e1c-a106-95125a26832d on workerId:ca1825f7-60e9-4e18-a557-02d5885316b8
2024-06-07T02:00:00Z   [Error]   Executed 'Functions.timer_trigger_oat' (Failed, Id=32b3764d-dd2b-44c7-9657-dd7ddfd09462, Duration=10ms)
2024-06-07T02:00:00Z   [Error]   Executed 'Functions.timer_trigger' (Failed, Id=fe049497-b777-4e1c-a106-95125a26832d, Duration=19ms)
2024-06-07T02:00:00Z   [Verbose]   Function 'timer_trigger_oat' updated status: Last='2024-06-07T02:00:00.0012629+00:00', Next='2024-06-07T02:05:00.0000000+00:00', LastUpdated='2024-06-07T02:00:00.0012629+00:00'
2024-06-07T02:00:00Z   [Verbose]   Timer for 'timer_trigger_oat' started with interval '00:04:59.9302891'.
2024-06-07T02:00:00Z   [Verbose]   Function 'timer_trigger' updated status: Last='2024-06-07T02:00:00.0004466+00:00', Next='2024-06-07T02:05:00.0000000+00:00', LastUpdated='2024-06-07T02:00:00.0004466+00:00'
2024-06-07T02:00:00Z   [Verbose]   Timer for 'timer_trigger' started with interval '00:04:59.9295213'.
2024-06-07T02:00:03Z   [Verbose]   Function 'oat_pipeline_task_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue-poison'.
2024-06-07T02:00:03Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-07T02:00:03Z   [Verbose]   [HostMonitor] Worker status: ID=ca1825f7-60e9-4e18-a557-02d5885316b8, Latency=1ms
2024-06-07T02:00:03Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 87): History=(0,0,0,0,1), AvgCpuLoad=0.2, MaxCpuLoad=1
2024-06-07T02:00:03Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 61): History=(1,1,7,2,1), AvgCpuLoad=2, MaxCpuLoad=7
2024-06-07T02:00:03Z   [Verbose]   [HostMonitor] Host aggregate CPU load 3
2024-06-07T02:00:03Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-07T02:00:09Z   [Verbose]   Function 'queue_trigger_wb' will wait 60000 ms before polling queue 'workbench-queue'.
2024-06-07T02:00:44Z   [Verbose]   Function 'oat_pipeline_task_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue'.
2024-06-07T02:00:51Z   [Verbose]   Function 'queue_trigger_rca' will wait 60000 ms before polling queue 'rca-queue'.
2024-06-07T02:00:56Z   [Verbose]   Function 'oat_pipeline_file_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue-poison'.
2024-06-07T02:00:59Z   [Verbose]   Function 'oat_pipeline_file_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue'.
2024-06-07T02:01:00Z   [Verbose]   Function 'queue_trigger_wb_poison' will wait 60000 ms before polling queue 'workbench-queue-poison'.
2024-06-07T02:01:03Z   [Verbose]   Function 'oat_pipeline_task_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue-poison'.
2024-06-07T02:01:09Z   [Verbose]   Function 'queue_trigger_wb' will wait 60000 ms before polling queue 'workbench-queue'.
2024-06-07T02:01:15Z   [Verbose]   Handling WorkerErrorEvent for runtime:python, workerId:python. Failed with: System.TimeoutException: The operation has timed out.
   at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.PendingItem.OnTimeout() in /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs:line 1764
2024-06-07T02:01:15Z   [Verbose]   Attempting to dispose webhost or jobhost channel for workerId: 'ca1825f7-60e9-4e18-a557-02d5885316b8', runtime: 'python'
2024-06-07T02:01:15Z   [Verbose]   No initialized worker channels for runtime 'python'. Delaying future invocations
2024-06-07T02:01:15Z   [Verbose]   Restarting worker channel for runtime: 'python'
2024-06-07T02:01:16Z   [Verbose]   Adding jobhost language worker channel for runtime: python. workerId:284b81b2-80c5-4049-a1bf-e720b91e81df
2024-06-07T02:01:16Z   [Information]   Worker process started and initialized.
2024-06-07T02:01:16Z   [Error]   ModuleNotFoundError: No module named '_cffi_backend'
2024-06-07T02:01:16Z   [Information]   thread '' panicked at /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/err/mod.rs:790:5:
2024-06-07T02:01:16Z   [Error]   Python API call failed
2024-06-07T02:01:16Z   [Information]   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2024-06-07T02:01:16Z   [Information]   Traceback (most recent call last):
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call
2024-06-07T02:01:16Z   [Information]       return func(*args, **kwargs)
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 214, in load_function
2024-06-07T02:01:16Z   [Information]       mod = importlib.import_module(fullmodname)
2024-06-07T02:01:16Z   [Information]     File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-06-07T02:01:16Z   [Information]       return _bootstrap._gcd_import(name[level:], package, level)
2024-06-07T02:01:16Z   [Information]     File "", line 1030, in _gcd_import
2024-06-07T02:01:16Z   [Information]     File "", line 1007, in _find_and_load
2024-06-07T02:01:16Z   [Information]     File "", line 986, in _find_and_load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 680, in _load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 850, in exec_module
2024-06-07T02:01:16Z   [Information]     File "", line 228, in _call_with_frames_removed
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/oat_pipeline_file_qt/__init__.py", line 2, in 
2024-06-07T02:01:16Z   [Information]       from shared_code import configurations, utils
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/shared_code/utils.py", line 3, in 
2024-06-07T02:01:16Z   [Information]       import jwt
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/__init__.py", line 1, in 
2024-06-07T02:01:16Z   [Information]       from .api_jwk import PyJWK, PyJWKSet
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/api_jwk.py", line 3, in 
2024-06-07T02:01:16Z   [Information]       from .algorithms import get_default_algorithms
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/algorithms.py", line 6, in 
2024-06-07T02:01:16Z   [Information]       from .utils import (
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/utils.py", line 7, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat._oid import ObjectIdentifier
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/_oid.py", line 9, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.bindings._rust import (
2024-06-07T02:01:16Z   [Error]   ImportError: PyO3 modules may only be initialized once per interpreter process
2024-06-07T02:01:16Z   [Information]   Traceback (most recent call last):
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call
2024-06-07T02:01:16Z   [Information]       return func(*args, **kwargs)
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 214, in load_function
2024-06-07T02:01:16Z   [Information]       mod = importlib.import_module(fullmodname)
2024-06-07T02:01:16Z   [Information]     File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-06-07T02:01:16Z   [Information]       return _bootstrap._gcd_import(name[level:], package, level)
2024-06-07T02:01:16Z   [Information]     File "", line 1030, in _gcd_import
2024-06-07T02:01:16Z   [Information]     File "", line 1007, in _find_and_load
2024-06-07T02:01:16Z   [Information]     File "", line 986, in _find_and_load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 680, in _load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 850, in exec_module
2024-06-07T02:01:16Z   [Information]     File "", line 228, in _call_with_frames_removed
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/oat_pipeline_task_poison_qt/__init__.py", line 4, in 
2024-06-07T02:01:16Z   [Information]       from azure.storage.queue import QueueClient, TextBase64EncodePolicy
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/queue/__init__.py", line 8, in 
2024-06-07T02:01:16Z   [Information]       from ._queue_client import QueueClient
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/queue/_queue_client.py", line 27, in 
2024-06-07T02:01:16Z   [Information]       from ._message_encoding import NoEncodePolicy, NoDecodePolicy
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/queue/_message_encoding.py", line 14, in 
2024-06-07T02:01:16Z   [Information]       from ._shared.encryption import decrypt_queue_message, encrypt_queue_message
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/queue/_shared/encryption.py", line 16, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.primitives.ciphers import Cipher
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/primitives/ciphers/__init__.py", line 11, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.primitives.ciphers.base import (
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/primitives/ciphers/base.py", line 10, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.exceptions import (
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/exceptions.py", line 9, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
2024-06-07T02:01:16Z   [Error]   ImportError: PyO3 modules may only be initialized once per interpreter process
2024-06-07T02:01:16Z   [Information]   Traceback (most recent call last):
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call
2024-06-07T02:01:16Z   [Information]       return func(*args, **kwargs)
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 214, in load_function
2024-06-07T02:01:16Z   [Information]       mod = importlib.import_module(fullmodname)
2024-06-07T02:01:16Z   [Information]     File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-06-07T02:01:16Z   [Information]       return _bootstrap._gcd_import(name[level:], package, level)
2024-06-07T02:01:16Z   [Information]     File "", line 1030, in _gcd_import
2024-06-07T02:01:16Z   [Information]     File "", line 1007, in _find_and_load
2024-06-07T02:01:16Z   [Information]     File "", line 986, in _find_and_load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 680, in _load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 850, in exec_module
2024-06-07T02:01:16Z   [Information]     File "", line 228, in _call_with_frames_removed
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/oat_pipeline_task_qt/__init__.py", line 6, in 
2024-06-07T02:01:16Z   [Information]       from shared_code import configurations, utils
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/shared_code/utils.py", line 3, in 
2024-06-07T02:01:16Z   [Information]       import jwt
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/__init__.py", line 1, in 
2024-06-07T02:01:16Z   [Information]       from .api_jwk import PyJWK, PyJWKSet
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/api_jwk.py", line 3, in 
2024-06-07T02:01:16Z   [Information]       from .algorithms import get_default_algorithms
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/algorithms.py", line 6, in 
2024-06-07T02:01:16Z   [Information]       from .utils import (
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/utils.py", line 7, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat._oid import ObjectIdentifier
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/_oid.py", line 9, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.bindings._rust import (
2024-06-07T02:01:16Z   [Error]   ImportError: PyO3 modules may only be initialized once per interpreter process
2024-06-07T02:01:16Z   [Information]   Traceback (most recent call last):
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call
2024-06-07T02:01:16Z   [Information]       return func(*args, **kwargs)
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 214, in load_function
2024-06-07T02:01:16Z   [Information]       mod = importlib.import_module(fullmodname)
2024-06-07T02:01:16Z   [Information]     File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-06-07T02:01:16Z   [Information]       return _bootstrap._gcd_import(name[level:], package, level)
2024-06-07T02:01:16Z   [Information]     File "", line 1030, in _gcd_import
2024-06-07T02:01:16Z   [Information]     File "", line 1007, in _find_and_load
2024-06-07T02:01:16Z   [Information]     File "", line 986, in _find_and_load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 680, in _load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 850, in exec_module
2024-06-07T02:01:16Z   [Information]     File "", line 228, in _call_with_frames_removed
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/queue_trigger_rca/__init__.py", line 10, in 
2024-06-07T02:01:16Z   [Information]       from shared_code import utils, configurations, utils, transform_utils
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/shared_code/utils.py", line 3, in 
2024-06-07T02:01:16Z   [Information]       import jwt
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/__init__.py", line 1, in 
2024-06-07T02:01:16Z   [Information]       from .api_jwk import PyJWK, PyJWKSet
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/api_jwk.py", line 3, in 
2024-06-07T02:01:16Z   [Information]       from .algorithms import get_default_algorithms
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/algorithms.py", line 6, in 
2024-06-07T02:01:16Z   [Information]       from .utils import (
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/utils.py", line 7, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat._oid import ObjectIdentifier
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/_oid.py", line 9, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.bindings._rust import (
2024-06-07T02:01:16Z   [Error]   ImportError: PyO3 modules may only be initialized once per interpreter process
2024-06-07T02:01:16Z   [Information]   Traceback (most recent call last):
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call
2024-06-07T02:01:16Z   [Information]       return func(*args, **kwargs)
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 214, in load_function
2024-06-07T02:01:16Z   [Information]       mod = importlib.import_module(fullmodname)
2024-06-07T02:01:16Z   [Information]     File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-06-07T02:01:16Z   [Information]       return _bootstrap._gcd_import(name[level:], package, level)
2024-06-07T02:01:16Z   [Information]     File "", line 1030, in _gcd_import
2024-06-07T02:01:16Z   [Information]     File "", line 1007, in _find_and_load
2024-06-07T02:01:16Z   [Information]     File "", line 986, in _find_and_load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 680, in _load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 850, in exec_module
2024-06-07T02:01:16Z   [Information]     File "", line 228, in _call_with_frames_removed
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/queue_trigger_wb/__init__.py", line 9, in 
2024-06-07T02:01:16Z   [Information]       from shared_code import utils, configurations, transform_utils
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/shared_code/utils.py", line 3, in 
2024-06-07T02:01:16Z   [Information]       import jwt
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/__init__.py", line 1, in 
2024-06-07T02:01:16Z   [Information]       from .api_jwk import PyJWK, PyJWKSet
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/api_jwk.py", line 3, in 
2024-06-07T02:01:16Z   [Information]       from .algorithms import get_default_algorithms
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/algorithms.py", line 6, in 
2024-06-07T02:01:16Z   [Information]       from .utils import (
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/jwt/utils.py", line 7, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat._oid import ObjectIdentifier
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/_oid.py", line 9, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.bindings._rust import (
2024-06-07T02:01:16Z   [Error]   ImportError: PyO3 modules may only be initialized once per interpreter process
2024-06-07T02:01:16Z   [Information]   Traceback (most recent call last):
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call
2024-06-07T02:01:16Z   [Information]       return func(*args, **kwargs)
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 214, in load_function
2024-06-07T02:01:16Z   [Information]       mod = importlib.import_module(fullmodname)
2024-06-07T02:01:16Z   [Information]     File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-06-07T02:01:16Z   [Information]       return _bootstrap._gcd_import(name[level:], package, level)
2024-06-07T02:01:16Z   [Information]     File "", line 1030, in _gcd_import
2024-06-07T02:01:16Z   [Information]     File "", line 1007, in _find_and_load
2024-06-07T02:01:16Z   [Information]     File "", line 986, in _find_and_load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 680, in _load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 850, in exec_module
2024-06-07T02:01:16Z   [Information]     File "", line 228, in _call_with_frames_removed
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/queue_trigger_wb_poison/__init__.py", line 6, in 
2024-06-07T02:01:16Z   [Information]       from azure.storage.queue import QueueClient, TextBase64EncodePolicy
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/queue/__init__.py", line 8, in 
2024-06-07T02:01:16Z   [Information]       from ._queue_client import QueueClient
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/queue/_queue_client.py", line 27, in 
2024-06-07T02:01:16Z   [Information]       from ._message_encoding import NoEncodePolicy, NoDecodePolicy
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/queue/_message_encoding.py", line 14, in 
2024-06-07T02:01:16Z   [Information]       from ._shared.encryption import decrypt_queue_message, encrypt_queue_message
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/storage/queue/_shared/encryption.py", line 16, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.primitives.ciphers import Cipher
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/primitives/ciphers/__init__.py", line 11, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.primitives.ciphers.base import (
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/primitives/ciphers/base.py", line 10, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.exceptions import (
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/.python_packages/lib/site-packages/cryptography/exceptions.py", line 9, in 
2024-06-07T02:01:16Z   [Information]       from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
2024-06-07T02:01:16Z   [Error]   ImportError: PyO3 modules may only be initialized once per interpreter process
2024-06-07T02:01:16Z   [Information]   Traceback (most recent call last):
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call
2024-06-07T02:01:16Z   [Information]       return func(*args, **kwargs)
2024-06-07T02:01:16Z   [Information]     File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 214, in load_function
2024-06-07T02:01:16Z   [Information]       mod = importlib.import_module(fullmodname)
2024-06-07T02:01:16Z   [Information]     File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-06-07T02:01:16Z   [Information]       return _bootstrap._gcd_import(name[level:], package, level)
2024-06-07T02:01:16Z   [Information]     File "", line 1030, in _gcd_import
2024-06-07T02:01:16Z   [Information]     File "", line 1007, in _find_and_load
2024-06-07T02:01:16Z   [Information]     File "", line 986, in _find_and_load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 680, in _load_unlocked
2024-06-07T02:01:16Z   [Information]     File "", line 850, in exec_module
2024-06-07T02:01:16Z   [Information]     File "", line 228, in _call_with_frames_removed
2024-06-07T02:01:16Z   [Information]     File "/home/site/wwwroot/timer_trigger/__init__.py", line 7, in 
2024-06-07T02:01:44Z   [Verbose]   Function 'oat_pipeline_task_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue'.
2024-06-07T02:01:51Z   [Verbose]   Function 'queue_trigger_rca' will wait 60000 ms before polling queue 'rca-queue'.
2024-06-07T02:01:56Z   [Verbose]   Function 'oat_pipeline_file_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue-poison'.
2024-06-07T02:01:59Z   [Verbose]   Function 'oat_pipeline_file_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue'.
2024-06-07T02:02:00Z   [Verbose]   Function 'queue_trigger_wb_poison' will wait 60000 ms before polling queue 'workbench-queue-poison'.
2024-06-07T02:02:03Z   [Verbose]   Function 'oat_pipeline_task_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue-poison'.
2024-06-07T02:02:09Z   [Verbose]   Function 'queue_trigger_wb' will wait 60000 ms before polling queue 'workbench-queue'.
2024-06-07T02:02:13Z   [Verbose]   Received request to drain the host
2024-06-07T02:02:13Z   [Information]   DrainMode mode enabled
2024-06-07T02:02:13Z   [Information]   Calling StopAsync on the registered listeners
2024-06-07T02:02:13Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'oat_pipeline_file_poison_qt'
2024-06-07T02:02:13Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'oat_pipeline_file_poison_qt'
2024-06-07T02:02:13Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'oat_pipeline_file_qt'
2024-06-07T02:02:13Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'oat_pipeline_file_qt'
2024-06-07T02:02:13Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'oat_pipeline_task_poison_qt'
2024-06-07T02:02:13Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'oat_pipeline_task_poison_qt'
2024-06-07T02:02:13Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'oat_pipeline_task_qt'
2024-06-07T02:02:13Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'oat_pipeline_task_qt'
2024-06-07T02:02:13Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'queue_trigger_rca'
2024-06-07T02:02:13Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'queue_trigger_rca'
2024-06-07T02:02:13Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'queue_trigger_wb'
2024-06-07T02:02:13Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'queue_trigger_wb'
2024-06-07T02:02:13Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'queue_trigger_wb_poison'
2024-06-07T02:02:13Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function 'queue_trigger_wb_poison'
2024-06-07T02:02:13Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener' for function 'timer_trigger'
2024-06-07T02:02:13Z   [Verbose]   Timer listener stopped (timer_trigger)
2024-06-07T02:02:13Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener' for function 'timer_trigger_oat'
2024-06-07T02:02:13Z   [Verbose]   Timer listener stopped (timer_trigger_oat)
2024-06-07T02:02:13Z   [Verbose]   Singleton lock released (visiononesmczq5hcdrn5g/Host.Functions.timer_trigger.Listener)
2024-06-07T02:02:13Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener' for function 'timer_trigger'
2024-06-07T02:02:13Z   [Verbose]   Singleton lock released (visiononesmczq5hcdrn5g/Host.Functions.timer_trigger_oat.Listener)
2024-06-07T02:02:13Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener' for function 'timer_trigger_oat'
2024-06-07T02:02:13Z   [Information]   Call to StopAsync complete, registered listeners are now stopped
v-sudkharat commented 5 months ago

Hi @hgtok, Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 20-06-2024. Thanks!

v-sudkharat commented 5 months ago

Hi @hgtok, Could you please check the function app by updating the WEBSITE_RUN_FROM_PACKAGE with below shared URL in the function app. and let us know the response.

image

Link- https://github.com/Azure/Azure-Sentinel/raw/489e084c58543e7e2737ae6863ff5b378f184dca/Solutions/Trend%20Micro%20Vision%20One/Data%20Connectors/AzureFunctionTrendMicroXDR.zip

Once updating it, please restart the function app. Please let us know if your issue gets resolved. Thanks!

hgtok commented 5 months ago

Works! We can see data ingested now. However, we still spot some [Error] showing up in the logs. Pls advise.

Details 
2024-06-13T08:28:43Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-13T08:28:43Z   [Verbose]   [HostMonitor] Worker status: ID=3417dee0-3037-4ae3-ab42-2d3dd6f5b76f, Latency=2ms
2024-06-13T08:28:43Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 89): History=(0,0,0,0,0), AvgCpuLoad=0, MaxCpuLoad=0
2024-06-13T08:28:43Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 60): History=(1,1,0,3,2), AvgCpuLoad=1, MaxCpuLoad=3
2024-06-13T08:28:43Z   [Verbose]   [HostMonitor] Host aggregate CPU load 1
2024-06-13T08:28:43Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-13T08:28:49Z   [Verbose]   Received request to drain the host
2024-06-13T08:28:57Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-13T08:28:57Z   [Verbose]   [HostMonitor] Worker status: ID=3417dee0-3037-4ae3-ab42-2d3dd6f5b76f, Latency=5ms
2024-06-13T08:28:57Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 89): History=(0,0,0,0,0), AvgCpuLoad=0, MaxCpuLoad=0
2024-06-13T08:28:57Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 60): History=(0,0,2,0,1), AvgCpuLoad=1, MaxCpuLoad=2
2024-06-13T08:28:57Z   [Verbose]   [HostMonitor] Host aggregate CPU load 1
2024-06-13T08:28:57Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-13T08:29:06Z   [Information]   Host lock lease acquired by instance ID '0000000000000000000000004D4E09B4'.
2024-06-13T08:29:06Z   [Verbose]   Function 'queue_trigger_wb_poison' will wait 60000 ms before polling queue 'workbench-queue-poison'.
2024-06-13T08:29:10Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-13T08:29:10Z   [Verbose]   [HostMonitor] Worker status: ID=3417dee0-3037-4ae3-ab42-2d3dd6f5b76f, Latency=1ms
2024-06-13T08:29:10Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 89): History=(0,0,0,0,0), AvgCpuLoad=0, MaxCpuLoad=0
2024-06-13T08:29:10Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 60): History=(1,2,0,1,1), AvgCpuLoad=1, MaxCpuLoad=2
2024-06-13T08:29:10Z   [Verbose]   [HostMonitor] Host aggregate CPU load 1
2024-06-13T08:29:10Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-13T08:29:11Z   [Verbose]   Function 'oat_pipeline_file_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue-poison'.
2024-06-13T08:29:17Z   [Verbose]   Function 'oat_pipeline_file_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue'.
2024-06-13T08:29:22Z   [Verbose]   Poll for function 'queue_trigger_wb' on queue 'workbench-queue' with ClientRequestId '4c4d2ddd-6d66-46c7-99d7-1d04ab1dd184' found 0 messages in 5 ms.
2024-06-13T08:29:22Z   [Verbose]   Function 'queue_trigger_wb' will wait 60000 ms before polling queue 'workbench-queue'.
2024-06-13T08:29:23Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-13T08:29:23Z   [Verbose]   [HostMonitor] Worker status: ID=3417dee0-3037-4ae3-ab42-2d3dd6f5b76f, Latency=1ms
2024-06-13T08:29:23Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 89): History=(0,0,0,1,0), AvgCpuLoad=0.2, MaxCpuLoad=1
2024-06-13T08:29:23Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 60): History=(1,1,0,1,1), AvgCpuLoad=1, MaxCpuLoad=1
2024-06-13T08:29:23Z   [Verbose]   [HostMonitor] Host aggregate CPU load 1
2024-06-13T08:29:23Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-13T08:29:25Z   [Verbose]   Function 'oat_pipeline_task_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue'.
2024-06-13T08:29:28Z   [Verbose]   Function 'oat_pipeline_task_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue-poison'.
2024-06-13T08:29:29Z   [Verbose]   Function 'queue_trigger_rca' will wait 60000 ms before polling queue 'rca-queue'.
2024-06-13T08:29:36Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-13T08:29:36Z   [Verbose]   [HostMonitor] Worker status: ID=3417dee0-3037-4ae3-ab42-2d3dd6f5b76f, Latency=1ms
2024-06-13T08:29:36Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 89): History=(1,0,0,0,0), AvgCpuLoad=0.2, MaxCpuLoad=1
2024-06-13T08:29:36Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 60): History=(0,2,0,1,1), AvgCpuLoad=1, MaxCpuLoad=2
2024-06-13T08:29:36Z   [Verbose]   [HostMonitor] Host aggregate CPU load 1
2024-06-13T08:29:36Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-13T08:29:50Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-13T08:29:50Z   [Verbose]   [HostMonitor] Worker status: ID=3417dee0-3037-4ae3-ab42-2d3dd6f5b76f, Latency=1ms
2024-06-13T08:29:50Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 89): History=(0,0,0,0,0), AvgCpuLoad=0, MaxCpuLoad=0
2024-06-13T08:29:50Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 60): History=(1,0,2,0,0), AvgCpuLoad=1, MaxCpuLoad=2
2024-06-13T08:29:50Z   [Verbose]   [HostMonitor] Host aggregate CPU load 1
2024-06-13T08:29:50Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-13T08:30:00Z   [Information]   Executing 'Functions.timer_trigger' (Reason='Timer fired at 2024-06-13T08:29:59.9997464+00:00', Id=cf0b4c43-be80-4313-97c7-6e46b00befa1)
2024-06-13T08:30:00Z   [Verbose]   Sending invocation id: 'cf0b4c43-be80-4313-97c7-6e46b00befa1
2024-06-13T08:30:00Z   [Verbose]   Posting invocation id:cf0b4c43-be80-4313-97c7-6e46b00befa1 on workerId:3417dee0-3037-4ae3-ab42-2d3dd6f5b76f
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,004", "message": "Client-Request-ID=1f91989c-295f-11ef-83f1-00155dcb728c Outgoing request: Method=POST, Path=/Tables, Query={'timeout': None}, Headers={'Content-Type': 'application/json', 'Prefer': 'return-no-content', 'Accept': 'application/json;odata=minimalmetadata', 'DataServiceVersion': '3.0;NetFx', 'MaxDataServiceVersion': '3.0', 'Content-Length': '35', 'x-ms-version': '2018-03-28', 'User-Agent': 'Azure-Storage/1.4.2-None (Python CPython 3.9.19; Linux 5.10.102.2-microsoft-standard)', 'x-ms-client-request-id': '1f91989c-295f-11ef-83f1-00155dcb728c', 'x-ms-date': 'Thu, 13 Jun 2024 08:30:00 GMT', 'Authorization': 'REDACTED'}.", "trace_id": "056f4b66-52a6-4fcb-8707-9ea58e88a6bb", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "azure.cosmosdb.table.common.storageclient", "func_name": "_perform_request", "level": "INFO"}
2024-06-13T08:30:00Z   [Information]   Executing 'Functions.timer_trigger_oat' (Reason='Timer fired at 2024-06-13T08:30:00.0078035+00:00', Id=295f8954-c89a-43c7-8173-ee7d1092c60c)
2024-06-13T08:30:00Z   [Verbose]   Sending invocation id: '295f8954-c89a-43c7-8173-ee7d1092c60c
2024-06-13T08:30:00Z   [Verbose]   Posting invocation id:295f8954-c89a-43c7-8173-ee7d1092c60c on workerId:3417dee0-3037-4ae3-ab42-2d3dd6f5b76f
2024-06-13T08:30:00Z   [Error]   Executed 'Functions.timer_trigger_oat' (Failed, Id=295f8954-c89a-43c7-8173-ee7d1092c60c, Duration=2ms)
2024-06-13T08:30:00Z   [Verbose]   Function 'timer_trigger_oat' updated status: Last='2024-06-13T08:30:00.0077877+00:00', Next='2024-06-13T08:35:00.0000000+00:00', LastUpdated='2024-06-13T08:30:00.0077877+00:00'
2024-06-13T08:30:00Z   [Verbose]   Timer for 'timer_trigger_oat' started with interval '00:04:59.9660111'.
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,080", "message": "Client-Request-ID=1f91989c-295f-11ef-83f1-00155dcb728c Receiving Response: Server-Timestamp=Thu, 13 Jun 2024 08:29:59 GMT, Server-Request-ID=ecb216b2-1002-004a-776b-bd3a85000000, HTTP Status Code=409, Message=Conflict, Headers={'cache-control': 'no-cache', 'transfer-encoding': 'chunked', 'content-type': 'application/json;odata=minimalmetadata;streaming=true;charset=utf-8', 'server': 'Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0', 'x-ms-request-id': 'ecb216b2-1002-004a-776b-bd3a85000000', 'x-ms-version': '2018-03-28', 'x-content-type-options': 'nosniff', 'preference-applied': 'return-no-content', 'date': 'Thu, 13 Jun 2024 08:29:59 GMT'}.", "trace_id": "056f4b66-52a6-4fcb-8707-9ea58e88a6bb", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "azure.cosmosdb.table.common.storageclient", "func_name": "_perform_request", "level": "INFO"}
2024-06-13T08:30:00Z   [Error]   {"asctime": "2024-06-13 08:30:00,080", "message": "Client-Request-ID=1f91989c-295f-11ef-83f1-00155dcb728c Retry policy did not allow for a retry: Server-Timestamp=Thu, 13 Jun 2024 08:29:59 GMT, Server-Request-ID=ecb216b2-1002-004a-776b-bd3a85000000, HTTP status code=409, Exception=Conflict{\"odata.error\":{\"code\":\"TableAlreadyExists\",\"message\":{\"lang\":\"en-US\",\"value\":\"The table specified already exists.\\nRequestId:ecb216b2-1002-004a-776b-bd3a85000000\\nTime:2024-06-13T08:30:00.0766201Z\"}}}.", "trace_id": "056f4b66-52a6-4fcb-8707-9ea58e88a6bb", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "azure.cosmosdb.table.common.storageclient", "func_name": "_perform_request", "level": "ERROR"}
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,080", "message": "Client-Request-ID=1f9d4318-295f-11ef-83f1-00155dcb728c Outgoing request: Method=GET, Path=/XdrConnectorStatus(PartitionKey='last_success_time',RowKey='1e34cfa5-7857-4991-80e9-faf985b61f88'), Query={'$select': None, 'timeout': None}, Headers={'Accept': 'application/json;odata=minimalmetadata', 'DataServiceVersion': '3.0;NetFx', 'MaxDataServiceVersion': '3.0', 'x-ms-version': '2018-03-28', 'User-Agent': 'Azure-Storage/1.4.2-None (Python CPython 3.9.19; Linux 5.10.102.2-microsoft-standard)', 'x-ms-client-request-id': '1f9d4318-295f-11ef-83f1-00155dcb728c', 'x-ms-date': 'Thu, 13 Jun 2024 08:30:00 GMT', 'Authorization': 'REDACTED'}.", "trace_id": "056f4b66-52a6-4fcb-8707-9ea58e88a6bb", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "azure.cosmosdb.table.common.storageclient", "func_name": "_perform_request", "level": "INFO"}
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,080", "message": "Client-Request-ID=1f91989c-295f-11ef-83f1-00155dcb728c Operation failed: checking if the operation should be retried. Current retry count=0, Server-Timestamp=Thu, 13 Jun 2024 08:29:59 GMT, Server-Request-ID=ecb216b2-1002-004a-776b-bd3a85000000, HTTP status code=409, Exception=Conflict{\"odata.error\":{\"code\":\"TableAlreadyExists\",\"message\":{\"lang\":\"en-US\",\"value\":\"The table specified already exists.\\nRequestId:ecb216b2-1002-004a-776b-bd3a85000000\\nTime:2024-06-13T08:30:00.0766201Z\"}}}.", "trace_id": "056f4b66-52a6-4fcb-8707-9ea58e88a6bb", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "azure.cosmosdb.table.common.storageclient", "func_name": "_perform_request", "level": "INFO"}
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,116", "message": "Client-Request-ID=1f9d4318-295f-11ef-83f1-00155dcb728c Receiving Response: Server-Timestamp=Thu, 13 Jun 2024 08:29:59 GMT, Server-Request-ID=ecb216b5-1002-004a-786b-bd3a85000000, HTTP Status Code=200, Message=OK, Headers={'cache-control': 'no-cache', 'transfer-encoding': 'chunked', 'content-type': 'application/json;odata=minimalmetadata;streaming=true;charset=utf-8', 'etag': 'W/\"datetime\\'2024-06-13T08%3A25%3A01.0941678Z\\'\"', 'server': 'Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0', 'x-ms-request-id': 'ecb216b5-1002-004a-786b-bd3a85000000', 'x-ms-version': '2018-03-28', 'x-content-type-options': 'nosniff', 'date': 'Thu, 13 Jun 2024 08:29:59 GMT'}.", "trace_id": "056f4b66-52a6-4fcb-8707-9ea58e88a6bb", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "azure.cosmosdb.table.common.storageclient", "func_name": "_perform_request", "level": "INFO"}
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,116", "message": "start to poll workbench events from 2024-06-13T08:25:00.000Z to 2024-06-13T08:30:00.000Z.", "trace_id": "056f4b66-52a6-4fcb-8707-9ea58e88a6bb", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "root", "func_name": "main", "level": "INFO"}
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,118", "message": "Get workbench list url: https://api.sg.xdr.trendmicro.com/v2.0/siem/events", "trace_id": "4d71b871-b7fb-469b-8ed1-c76d5865c399", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "root", "func_name": "get_workbench_list", "level": "INFO"}
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,478", "message": "Get workbench list response: {\"info\":{\"code\":3008000,\"msg\":\"Retrieve workbench summary information successfully.\"},\"data\":{\"totalCount\":0,\"modelList\":[\"Demo - Copying of NTDS File\",\"Demo - Credential Dumping via Registry\",\"Disabling of Gatekeeper\",\"Eicar Test File Detection\",\"Possible Brute Force via Multiple Failed Logons via Windows Event\",\"Possible Disabling of Antivirus Software\",\"Suspicious Multiple Failed Logons via Windows Event\",\"Suspicious Ransomware Behavior\",\"Targeted Attack Detection: Fileless Credential Dumping\",\"Unknown Threat Detection and Mitigation via Predictive Machine Learning\",\"[Heuristic Attribute] Backdoor File Detection\",\"[Heuristic Attribute] Impair Defenses\",\"[Heuristic Attribute] Possible Unsecured Credentials\",\"[Heuristic Attribute] Trojan Spy File Detection\"],\"workbenchRecords\":[]}}Get workbench list trace: task id: 9057da04-0fd6-4069-9fa0-6ffdd331ceab, trace id: 4d71b871-b7fb-469b-8ed1-c76d5865c399.", "trace_id": "4d71b871-b7fb-469b-8ed1-c76d5865c399", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "root", "func_name": "get_workbench_list", "level": "INFO"}
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,480", "message": "0 workbench events received.", "trace_id": "4d71b871-b7fb-469b-8ed1-c76d5865c399", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "root", "func_name": "main", "level": "INFO"}
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,481", "message": "Client-Request-ID=1fda65d6-295f-11ef-83f1-00155dcb728c Outgoing request: Method=MERGE, Path=/XdrConnectorStatus(PartitionKey='last_success_time',RowKey='1e34cfa5-7857-4991-80e9-faf985b61f88'), Query={'timeout': None}, Headers={'Content-Type': 'application/json', 'Accept': 'application/json;odata=minimalmetadata', 'DataServiceVersion': '3.0;NetFx', 'MaxDataServiceVersion': '3.0', 'Content-Length': '136', 'x-ms-version': '2018-03-28', 'User-Agent': 'Azure-Storage/1.4.2-None (Python CPython 3.9.19; Linux 5.10.102.2-microsoft-standard)', 'x-ms-client-request-id': '1fda65d6-295f-11ef-83f1-00155dcb728c', 'x-ms-date': 'Thu, 13 Jun 2024 08:30:00 GMT', 'Authorization': 'REDACTED'}.", "trace_id": "4d71b871-b7fb-469b-8ed1-c76d5865c399", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "azure.cosmosdb.table.common.storageclient", "func_name": "_perform_request", "level": "INFO"}
2024-06-13T08:30:00Z   [Information]   {"asctime": "2024-06-13 08:30:00,493", "message": "Client-Request-ID=1fda65d6-295f-11ef-83f1-00155dcb728c Receiving Response: Server-Timestamp=Thu, 13 Jun 2024 08:29:59 GMT, Server-Request-ID=ecb216f7-1002-004a-2f6b-bd3a85000000, HTTP Status Code=204, Message=No Content, Headers={'cache-control': 'no-cache', 'content-length': '0', 'etag': 'W/\"datetime\\'2024-06-13T08%3A30%3A00.4852035Z\\'\"', 'server': 'Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0', 'x-ms-request-id': 'ecb216f7-1002-004a-2f6b-bd3a85000000', 'x-ms-version': '2018-03-28', 'x-content-type-options': 'nosniff', 'date': 'Thu, 13 Jun 2024 08:29:59 GMT'}.", "trace_id": "4d71b871-b7fb-469b-8ed1-c76d5865c399", "task_id": "9057da04-0fd6-4069-9fa0-6ffdd331ceab", "version": "TMXDRSentinelAddon/1.1.0", "logger_name": "azure.cosmosdb.table.common.storageclient", "func_name": "_perform_request", "level": "INFO"}
2024-06-13T08:30:01Z   [Information]   Executed 'Functions.timer_trigger' (Succeeded, Id=cf0b4c43-be80-4313-97c7-6e46b00befa1, Duration=644ms)
2024-06-13T08:30:01Z   [Verbose]   Function 'timer_trigger' updated status: Last='2024-06-13T08:30:00.0000000+00:00', Next='2024-06-13T08:35:00.0000000+00:00', LastUpdated='2024-06-13T08:30:00.0000000+00:00'
2024-06-13T08:30:01Z   [Verbose]   Timer for 'timer_trigger' started with interval '00:04:59.3326092'.
2024-06-13T08:30:03Z   [Verbose]   [HostMonitor] Checking worker statuses (Count=1)
2024-06-13T08:30:03Z   [Verbose]   [HostMonitor] Worker status: ID=3417dee0-3037-4ae3-ab42-2d3dd6f5b76f, Latency=1ms
2024-06-13T08:30:03Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 89): History=(0,0,2,4,0), AvgCpuLoad=1.2, MaxCpuLoad=4
2024-06-13T08:30:03Z   [Verbose]   [HostMonitor] Host process CPU stats (PID 60): History=(1,1,2,3,1), AvgCpuLoad=2, MaxCpuLoad=3
2024-06-13T08:30:03Z   [Verbose]   [HostMonitor] Host aggregate CPU load 3
2024-06-13T08:30:03Z   [Information]   Executing StatusCodeResult, setting HTTP status code 200
2024-06-13T08:30:06Z   [Verbose]   Function 'queue_trigger_wb_poison' will wait 60000 ms before polling queue 'workbench-queue-poison'.
2024-06-13T08:30:11Z   [Verbose]   Function 'oat_pipeline_file_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue-poison'.
2024-06-13T08:30:17Z   [Verbose]   Function 'oat_pipeline_file_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue'.
2024-06-13T08:30:22Z   [Verbose]   Poll for function 'queue_trigger_wb' on queue 'workbench-queue' with ClientRequestId '8cc9bc96-48ab-491f-ad79-7334ef309982' found 0 messages in 9 ms.
2024-06-13T08:30:22Z   [Verbose]   Function 'queue_trigger_wb' will wait 60000 ms before polling queue 'workbench-queue'.
2024-06-13T08:30:25Z   [Verbose]   Function 'oat_pipeline_task_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue'.
2024-06-13T08:30:28Z   [Verbose]   Function 'oat_pipeline_task_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-task-queue-poison'.
2024-06-13T08:30:29Z   [Verbose]   Function 'queue_trigger_rca' will wait 60000 ms before polling queue 'rca-queue'.
2024-06-13T08:31:06Z   [Verbose]   Function 'queue_trigger_wb_poison' will wait 60000 ms before polling queue 'workbench-queue-poison'.
2024-06-13T08:31:11Z   [Verbose]   Function 'oat_pipeline_file_poison_qt' will wait 60000 ms before polling queue 'oat-pipeline-file-queue-poison'.
v-sudkharat commented 5 months ago

@hgtok, Could you please share the invocation logs with us so we can check for that error. Thanks!

hgtok commented 5 months ago

the logs are included in my previous comment. pls take a look.

v-sudkharat commented 5 months ago

Comment added here- https://github.com/Azure/Azure-Sentinel/issues/10653#issuecomment-2172790813