A Solution Accelerator for the RAG pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences. This includes most common requirements and best practices.
I have initiated (seperately) both "code" and "container" based deployments from the Local Development/Deployment approach, on Windows WSL using Dev Container. They succesfully deployed. However the main app is consisently failing to start in the App Service. I've redployed multiple times, in new resource groups, without any success.
Expected behavior
Both the admin and main chat app are running based on a standard, uncustomized local deployment
If applicable, add screenshots to help explain your problem.
Logs
Ok
2024-09-12T12:38:50.128
INFO - Pulling image: mcr.microsoft.com/appsvc/middleware:stage5
Ok
2024-09-12T12:38:50.408
INFO - stage5 Pulling from appsvc/middleware
Ok
2024-09-12T12:38:50.428
INFO - Digest: sha256:6b3c135340f42bbf62290b3ff21556271c6e979753e7cafacae3e74191535b7e
Ok
2024-09-12T12:38:50.429
INFO - Status: Image is up to date for mcr.microsoft.com/appsvc/middleware:stage5
Ok
2024-09-12T12:38:50.444
INFO - Pull Image successful, Time taken: 0 Seconds
Ok
2024-09-12T12:38:50.544
INFO - Starting container for site
Ok
2024-09-12T12:38:50.545
INFO - docker run -d --expose=8181 --name web-**_e59fbcc5_middleware -e WEBSITE_CORS_ALLOWED_ORIGINS=https://portal.azure.com,https://ms.portal.azure.com -e WEBSITE_CORS_SUPPORT_CREDENTIALS=False -e WEBSITE_SITE_NAME=web-c -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=web-****.azurewebsites.net -e WEBSITE_INSTANCE_ID=32a926d1a80c14b7a13d64095168e8c96cae058331d08d583124b88b7de0a309 -e HTTP_LOGGING_ENABLED=12024-09-12T12:38:51.606Z INFO - Initiating warmup request to container web-_0_e59fbcc5_msiProxy for site web-cgz7qvevojqp6
Ok
2024-09-12T12:38:51.635
INFO - Container web-cgz7*0_e59fbcc5_msiProxy for site web-**** initialized successfully and is ready to serve requests.
Ok
2024-09-12T12:38:51.636
INFO - Initiating warmup request to container web-cg**6_0_e59fbcc5 for site web-cgz7qvevojqp6
Error
2024-09-12T12:39:49.558
ERROR - Container *ojqp6_0_e59fbcc5 for site **has exited, failing site start
Ok
2024-09-12T12:39:49.558
INFO - Initiating warmup request to container *_middleware for site web-cgz7qvevojqp6
Ok
2024-09-12T12:39:50.289
INFO - Container web-cgz7qvevojqp6_0_e59fbcc5_middleware for site ** initialized successfully and is ready to serve requests.
Ok
2024-09-12T12:39:50.331
ERROR - Container ***5 didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging.
Ok
2024-09-12T12:39:50.351
INFO - Stopping site ***** because it failed during startup.
Ok
2024-09-12T12:39:50.351
APPLICATION LOGS
2024-09-12T12:44:05.232332694Z
2024-09-12T12:44:05.232365395Z / \ _____ ___
2024-09-12T12:44:05.232371095Z / /\ \_ / | _ _/ \
2024-09-12T12:44:05.232375595Z / | \/ /| | /| | \/\ /
2024-09-12T12:44:05.232379795Z __| /__ __/ |_| \ >
2024-09-12T12:44:05.232384195Z \/ \/ \/
2024-09-12T12:44:05.232388195Z A P P S E R V I C E O N L I N U X
2024-09-12T12:44:05.232392195Z
2024-09-12T12:44:05.232396095Z Documentation: http://aka.ms/webapp-linux
2024-09-12T12:44:05.232400095Z Python 3.11.8
2024-09-12T12:44:05.232403995Z Note: Any data outside '/home' is not persisted
2024-09-12T12:44:07.247320554Z Starting OpenBSD Secure Shell server: sshd.
2024-09-12T12:44:07.259004280Z WEBSITES_INCLUDE_CLOUD_CERTS is not set to true.
2024-09-12T12:44:07.365299328Z App Command Line not configured, will attempt auto-detect
2024-09-12T12:44:07.365330028Z Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite
2024-09-12T12:44:07.405556963Z Found build manifest file at '/home/site/wwwroot/oryx-manifest.toml'. Deserializing it...
2024-09-12T12:44:07.409694207Z Build Operation ID: 7f022b7e419fd428
2024-09-12T12:44:07.413181045Z Oryx Version: 0.2.20240501.1, Commit: f83f88d3cfb8bb6d3e2765e1dcd218eb0814a095, ReleaseTagName: 20240501.1
2024-09-12T12:44:07.413211045Z Output is compressed. Extracting it...
2024-09-12T12:44:07.416175877Z Extracting '/home/site/wwwroot/output.tar.gz' to directory '/tmp/8dcd26f27168ecd'...
2024-09-12T12:44:31.114636292Z App path is set to '/tmp/8dcd26f27168ecd'
2024-09-12T12:44:31.260690066Z Detected an app based on Flask
2024-09-12T12:44:31.260886468Z Generating gunicorn command for 'app:app'
2024-09-12T12:44:31.357977913Z Writing output script to '/opt/startup/startup.sh'
2024-09-12T12:44:31.425621742Z Using packages from virtual environment antenv located at /tmp/8dcd26f27168ecd/antenv.
2024-09-12T12:44:31.425660242Z Updated PYTHONPATH to '/opt/startup/app_logs:/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages'
2024-09-12T12:44:32.573330103Z [2024-09-12 12:44:32 +0000] [81] [INFO] Starting gunicorn 22.0.0
2024-09-12T12:44:32.588917671Z [2024-09-12 12:44:32 +0000] [81] [INFO] Listening at: http://0.0.0.0:8000 (81)
2024-09-12T12:44:32.588951671Z [2024-09-12 12:44:32 +0000] [81] [INFO] Using worker: sync
2024-09-12T12:44:32.595079137Z [2024-09-12 12:44:32 +0000] [84] [INFO] Booting worker with pid: 84
2024-09-12T12:44:32.672615072Z [2024-09-12 12:44:32 +0000] [85] [INFO] Booting worker with pid: 85
2024-09-12T12:44:32.786003293Z [2024-09-12 12:44:32 +0000] [86] [INFO] Booting worker with pid: 86
2024-09-12T12:44:32.840630881Z [2024-09-12 12:44:32 +0000] [87] [INFO] Booting worker with pid: 87
2024-09-12T12:44:32.927612517Z [2024-09-12 12:44:32 +0000] [88] [INFO] Booting worker with pid: 88
2024-09-12T12:44:33.001185209Z [2024-09-12 12:44:32 +0000] [89] [INFO] Booting worker with pid: 89
2024-09-12T12:44:33.062721871Z [2024-09-12 12:44:33 +0000] [90] [INFO] Booting worker with pid: 90
2024-09-12T12:44:33.154343657Z [2024-09-12 12:44:33 +0000] [91] [INFO] Booting worker with pid: 91
2024-09-12T12:44:33.248754473Z [2024-09-12 12:44:33 +0000] [92] [INFO] Booting worker with pid: 92
2024-09-12T12:44:59.898021010Z [2024-09-12 12:44:59 +0000] [91] [ERROR] Exception in worker process
2024-09-12T12:44:59.898078411Z Traceback (most recent call last):
2024-09-12T12:44:59.898090311Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
2024-09-12T12:44:59.898100011Z worker.init_process()
2024-09-12T12:44:59.898108511Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process
2024-09-12T12:44:59.898117311Z self.load_wsgi()
2024-09-12T12:44:59.898125411Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
2024-09-12T12:44:59.898134012Z self.wsgi = self.app.wsgi()
2024-09-12T12:44:59.898142012Z ^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898150112Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi
2024-09-12T12:44:59.898158612Z self.callable = self.load()
2024-09-12T12:44:59.898166612Z ^^^^^^^^^^^
2024-09-12T12:44:59.898174612Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
2024-09-12T12:44:59.898183212Z return self.load_wsgiapp()
2024-09-12T12:44:59.898191212Z ^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898199112Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
2024-09-12T12:44:59.898207912Z return util.import_app(self.app_uri)
2024-09-12T12:44:59.898215912Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898224112Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/util.py", line 371, in import_app
2024-09-12T12:44:59.898232613Z mod = importlib.import_module(module)
2024-09-12T12:44:59.898240613Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898248613Z File "/opt/python/3.11.8/lib/python3.11/importlib/init.py", line 126, in import_module
2024-09-12T12:44:59.898256913Z return _bootstrap._gcd_import(name[level:], package, level)
2024-09-12T12:44:59.898265713Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898273913Z File "", line 1204, in _gcd_import
2024-09-12T12:44:59.898283213Z File "", line 1176, in _find_and_load
2024-09-12T12:44:59.898306613Z File "", line 1147, in _find_and_load_unlocked
2024-09-12T12:44:59.898315613Z File "", line 690, in _load_unlocked
2024-09-12T12:44:59.898324014Z File "", line 940, in exec_module
2024-09-12T12:44:59.898332414Z File "", line 241, in _call_with_frames_removed
2024-09-12T12:44:59.898340814Z File "/tmp/8dcd26f27168ecd/app.py", line 22, in
2024-09-12T12:44:59.898349414Z from create_app import create_app # noqa: E402
2024-09-12T12:44:59.898357514Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898367114Z File "/tmp/8dcd26f27168ecd/create_app.py", line 17, in
2024-09-12T12:44:59.898375814Z from backend.batch.utilities.helpers.orchestrator_helper import Orchestrator
2024-09-12T12:44:59.898384114Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/orchestrator_helper.py", line 5, in
2024-09-12T12:44:59.898392714Z from ..orchestrator.strategies import get_orchestrator
2024-09-12T12:44:59.898400814Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/strategies.py", line 2, in
2024-09-12T12:44:59.898409514Z from .open_ai_functions import OpenAIFunctionsOrchestrator
2024-09-12T12:44:59.898417614Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/open_ai_functions.py", line 5, in
2024-09-12T12:44:59.898426215Z from .orchestrator_base import OrchestratorBase
2024-09-12T12:44:59.898434215Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/orchestrator_base.py", line 5, in
2024-09-12T12:44:59.898442815Z from ..loggers.conversation_logger import ConversationLogger
2024-09-12T12:44:59.898450915Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/loggers/conversation_logger.py", line 1, in
2024-09-12T12:44:59.898459415Z from ..helpers.azure_search_helper import AzureSearchHelper
2024-09-12T12:44:59.898467515Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/azure_search_helper.py", line 29, in
2024-09-12T12:44:59.898476015Z from .llm_helper import LLMHelper
2024-09-12T12:44:59.898484015Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/llm_helper.py", line 5, in
2024-09-12T12:44:59.898492415Z from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
2024-09-12T12:44:59.898500915Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/init.py", line 3, in
2024-09-12T12:44:59.898509315Z from semantic_kernel.kernel import Kernel
2024-09-12T12:44:59.898517316Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/kernel.py", line 31, in
2024-09-12T12:44:59.898533016Z from semantic_kernel.functions.function_result import FunctionResult
2024-09-12T12:44:59.898541416Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/init.py", line 5, in
2024-09-12T12:44:59.898550016Z from semantic_kernel.functions.kernel_function import KernelFunction
2024-09-12T12:44:59.898558216Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in
2024-09-12T12:44:59.898566716Z class KernelFunction(KernelBaseModel):
2024-09-12T12:44:59.898575516Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 202, in new
2024-09-12T12:44:59.898584016Z complete_model_class(
2024-09-12T12:44:59.898591916Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 572, in complete_model_class
2024-09-12T12:44:59.898600416Z generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper),
2024-09-12T12:44:59.898608717Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898616917Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature
2024-09-12T12:44:59.898625417Z merged_params = _generate_signature_parameters(init, fields, config_wrapper)
2024-09-12T12:44:59.898633517Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898641717Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters
2024-09-12T12:44:59.898650217Z kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)}
2024-09-12T12:44:59.898658417Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898666717Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/fields.py", line 546, in get_default
2024-09-12T12:44:59.898675017Z return _utils.smart_deepcopy(self.default)
2024-09-12T12:44:59.898682917Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898690917Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy
2024-09-12T12:44:59.898699417Z return deepcopy(obj) # slowest way when we actually might need a deepcopy
2024-09-12T12:44:59.898707618Z ^^^^^^^^^^^^^
2024-09-12T12:44:59.898715618Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:44:59.898723918Z y = _reconstruct(x, memo, *rv)
2024-09-12T12:44:59.898731818Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898739818Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:44:59.898754818Z state = deepcopy(state, memo)
2024-09-12T12:44:59.898763018Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898771018Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:44:59.898779418Z y = copier(x, memo)
2024-09-12T12:44:59.898787318Z ^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898795318Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
......................
.....................................
2024-09-12T12:45:02.925627225Z from ..helpers.azure_search_helper import AzureSearchHelper
2024-09-12T12:45:02.925631725Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/azure_search_helper.py", line 29, in
2024-09-12T12:45:02.925640625Z from .llm_helper import LLMHelper
2024-09-12T12:45:02.925645125Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/llm_helper.py", line 5, in
2024-09-12T12:45:02.925649825Z from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
2024-09-12T12:45:02.925654225Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/init.py", line 3, in
2024-09-12T12:45:02.925658925Z from semantic_kernel.kernel import Kernel
2024-09-12T12:45:02.925663225Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/kernel.py", line 31, in
2024-09-12T12:45:02.925667925Z from semantic_kernel.functions.function_result import FunctionResult
2024-09-12T12:45:02.925672425Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/init.py", line 5, in
2024-09-12T12:45:02.925677125Z from semantic_kernel.functions.kernel_function import KernelFunction
2024-09-12T12:45:02.925681625Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in
2024-09-12T12:45:02.925686326Z class KernelFunction(KernelBaseModel):
2024-09-12T12:45:02.925691126Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 202, in new
2024-09-12T12:45:02.925695926Z complete_model_class(
2024-09-12T12:45:02.925700226Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 572, in complete_model_class
2024-09-12T12:45:02.925704926Z generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper),
2024-09-12T12:45:02.925709426Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925714126Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature
2024-09-12T12:45:02.925718726Z merged_params = _generate_signature_parameters(init, fields, config_wrapper)
2024-09-12T12:45:02.925723326Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925727826Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters
2024-09-12T12:45:02.925732526Z kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)}
2024-09-12T12:45:02.925737026Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925741526Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/fields.py", line 546, in get_default
2024-09-12T12:45:02.925746126Z return _utils.smart_deepcopy(self.default)
2024-09-12T12:45:02.925754526Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925759126Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy
2024-09-12T12:45:02.925763726Z return deepcopy(obj) # slowest way when we actually might need a deepcopy
2024-09-12T12:45:02.925768226Z ^^^^^^^^^^^^^
2024-09-12T12:45:02.925772526Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:02.925777126Z y = _reconstruct(x, memo, rv)
2024-09-12T12:45:02.925781427Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925785827Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:02.925790327Z state = deepcopy(state, memo)
2024-09-12T12:45:02.925794727Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925799027Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:02.925803527Z y = copier(x, memo)
2024-09-12T12:45:02.925807827Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925812127Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:02.925817227Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:02.925821827Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925826327Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:02.925830827Z y = _reconstruct(x, memo, rv)
2024-09-12T12:45:02.925835227Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925839627Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:02.925844127Z state = deepcopy(state, memo)
2024-09-12T12:45:02.925848427Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925852927Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:02.925857427Z y = copier(x, memo)
2024-09-12T12:45:02.925861827Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925866127Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in _deepcopy_tuple
2024-09-12T12:45:02.925873628Z y = [deepcopy(a, memo) for a in x]
2024-09-12T12:45:02.925878128Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925882428Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in
2024-09-12T12:45:02.925887028Z y = [deepcopy(a, memo) for a in x]
2024-09-12T12:45:02.925891428Z ^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925895828Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:02.925900328Z y = copier(x, memo)
2024-09-12T12:45:02.925904728Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925913228Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:02.925917828Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:02.925922228Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925926728Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:02.925931228Z y = _reconstruct(x, memo, *rv)
2024-09-12T12:45:02.925935528Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925940028Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:02.925944528Z state = deepcopy(state, memo)
2024-09-12T12:45:02.925948828Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925953228Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:02.925957728Z y = copier(x, memo)
2024-09-12T12:45:02.925962028Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925966428Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:02.925970929Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:02.925975329Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925979729Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 161, in deepcopy
2024-09-12T12:45:02.925984229Z rv = reductor(4)
2024-09-12T12:45:02.925988529Z ^^^^^^^^^^^
2024-09-12T12:45:02.925992929Z TypeError: cannot pickle '_thread.RLock' object
2024-09-12T12:45:02.927983150Z [2024-09-12 12:45:02 +0000] [89] [INFO] Worker exiting (pid: 89)
2024-09-12T12:45:03.042975662Z [2024-09-12 12:45:03 +0000] [88] [ERROR] Exception in worker process
2024-09-12T12:45:03.043013162Z Traceback (most recent call last):
2024-09-12T12:45:03.043020262Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
2024-09-12T12:45:03.043025962Z worker.init_process()
2024-09-12T12:45:03.043031162Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process
2024-09-12T12:45:03.043036362Z self.load_wsgi()
2024-09-12T12:45:03.043041362Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
2024-09-12T12:45:03.043046463Z self.wsgi = self.app.wsgi()
2024-09-12T12:45:03.043051363Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043056163Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi
2024-09-12T12:45:03.043061163Z self.callable = self.load()
2024-09-12T12:45:03.043066063Z ^^^^^^^^^^^
2024-09-12T12:45:03.043070963Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
2024-09-12T12:45:03.043088963Z return self.load_wsgiapp()
2024-09-12T12:45:03.043093863Z ^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043098563Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
2024-09-12T12:45:03.043103563Z return util.import_app(self.app_uri)
2024-09-12T12:45:03.043108363Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043112863Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/util.py", line 371, in import_app
2024-09-12T12:45:03.043117763Z mod = importlib.import_module(module)
2024-09-12T12:45:03.043122563Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043127163Z File "/opt/python/3.11.8/lib/python3.11/importlib/init.py", line 126, in import_module
2024-09-12T12:45:03.043132163Z return _bootstrap._gcd_import(name[level:], package, level)
2024-09-12T12:45:03.043137063Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043141964Z File "", line 1204, in _gcd_import
2024-09-12T12:45:03.043147264Z File "", line 1176, in _find_and_load
2024-09-12T12:45:03.043152264Z File "", line 1147, in _find_and_load_unlocked
2024-09-12T12:45:03.043157264Z File "", line 690, in _load_unlocked
2024-09-12T12:45:03.043162164Z File "", line 940, in exec_module
2024-09-12T12:45:03.043167264Z File "", line 241, in _call_with_frames_removed
2024-09-12T12:45:03.043172264Z File "/tmp/8dcd26f27168ecd/app.py", line 22, in
2024-09-12T12:45:03.043177264Z from create_app import create_app # noqa: E402
2024-09-12T12:45:03.043182164Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043188264Z File "/tmp/8dcd26f27168ecd/create_app.py", line 17, in
2024-09-12T12:45:03.043193364Z from backend.batch.utilities.helpers.orchestrator_helper import Orchestrator
2024-09-12T12:45:03.043197964Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/orchestrator_helper.py", line 5, in
2024-09-12T12:45:03.043202764Z from ..orchestrator.strategies import get_orchestrator
2024-09-12T12:45:03.043207264Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/strategies.py", line 2, in
2024-09-12T12:45:03.043212164Z from .open_ai_functions import OpenAIFunctionsOrchestrator
2024-09-12T12:45:03.043216564Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/open_ai_functions.py", line 5, in
2024-09-12T12:45:03.043221364Z from .orchestrator_base import OrchestratorBase
2024-09-12T12:45:03.043225864Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/orchestrator_base.py", line 5, in
2024-09-12T12:45:03.043235565Z from ..loggers.conversation_logger import ConversationLogger
2024-09-12T12:45:03.043244365Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/loggers/conversation_logger.py", line 1, in
2024-09-12T12:45:03.043249165Z from ..helpers.azure_search_helper import AzureSearchHelper
2024-09-12T12:45:03.043253865Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/azure_search_helper.py", line 29, in
2024-09-12T12:45:03.043258765Z from .llm_helper import LLMHelper
2024-09-12T12:45:03.043263665Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/llm_helper.py", line 5, in
2024-09-12T12:45:03.043268565Z from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
2024-09-12T12:45:03.043273165Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/init.py", line 3, in
2024-09-12T12:45:03.043278065Z from semantic_kernel.kernel import Kernel
2024-09-12T12:45:03.043282665Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/kernel.py", line 31, in
2024-09-12T12:45:03.043287465Z from semantic_kernel.functions.function_result import FunctionResult
2024-09-12T12:45:03.043292065Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/init.py", line 5, in
2024-09-12T12:45:03.043296965Z from semantic_kernel.functions.kernel_function import KernelFunction
2024-09-12T12:45:03.043301665Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in
2024-09-12T12:45:03.043306565Z class KernelFunction(KernelBaseModel):
2024-09-12T12:45:03.043311665Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 202, in new
2024-09-12T12:45:03.043316565Z complete_model_class(
2024-09-12T12:45:03.043321165Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 572, in complete_model_class
2024-09-12T12:45:03.043325965Z generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper),
2024-09-12T12:45:03.043330666Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043335366Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature
2024-09-12T12:45:03.043340266Z merged_params = _generate_signature_parameters(init, fields, config_wrapper)
2024-09-12T12:45:03.043344866Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043349566Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters
2024-09-12T12:45:03.043358666Z kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)}
2024-09-12T12:45:03.043363466Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043367966Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/fields.py", line 546, in get_default
2024-09-12T12:45:03.043372666Z return _utils.smart_deepcopy(self.default)
2024-09-12T12:45:03.043377166Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043381666Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy
2024-09-12T12:45:03.043386566Z return deepcopy(obj) # slowest way when we actually might need a deepcopy
2024-09-12T12:45:03.043391466Z ^^^^^^^^^^^^^
2024-09-12T12:45:03.043395866Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:03.043400366Z y = _reconstruct(x, memo, rv)
2024-09-12T12:45:03.043404866Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043409266Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:03.043413866Z state = deepcopy(state, memo)
2024-09-12T12:45:03.043418266Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043422766Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:03.043427467Z y = copier(x, memo)
2024-09-12T12:45:03.043432067Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043436367Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:03.043441367Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:03.043445767Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043450267Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:03.043454867Z y = _reconstruct(x, memo, rv)
2024-09-12T12:45:03.043459367Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043463867Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:03.043468367Z state = deepcopy(state, memo)
2024-09-12T12:45:03.043472767Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043477067Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:03.043481667Z y = copier(x, memo)
2024-09-12T12:45:03.043486067Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043490467Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in _deepcopy_tuple
2024-09-12T12:45:03.043495167Z y = [deepcopy(a, memo) for a in x]
2024-09-12T12:45:03.043499567Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043508267Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in
2024-09-12T12:45:03.043513367Z y = [deepcopy(a, memo) for a in x]
2024-09-12T12:45:03.043517867Z ^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043522368Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:03.043526968Z y = copier(x, memo)
2024-09-12T12:45:03.043531468Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043535868Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:03.043540468Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:03.043545068Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043549568Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:03.043554168Z y = _reconstruct(x, memo, *rv)
2024-09-12T12:45:03.043558668Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043563168Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:03.043567668Z state = deepcopy(state, memo)
2024-09-12T12:45:03.043572168Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043576668Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:03.043581368Z y = copier(x, memo)
2024-09-12T12:45:03.043585768Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043590068Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:03.043594668Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:03.043599168Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043603668Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 161, in deepcopy
2024-09-12T12:45:03.043608268Z rv = reductor(4)
2024-09-12T12:45:03.043612668Z ^^^^^^^^^^^
2024-09-12T12:45:03.043616969Z TypeError: cannot pickle '_thread.RLock' object
2024-09-12T12:45:03.043621369Z [2024-09-12 12:45:03 +0000] [88] [INFO] Worker exiting (pid: 88)
2024-09-12T12:45:04.843246335Z [2024-09-12 12:45:04 +0000] [81] [ERROR] Worker (pid:92) exited with code 3
2024-09-12T12:45:04.843288836Z Traceback (most recent call last):
2024-09-12T12:45:04.843295436Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 209, in run
2024-09-12T12:45:04.843301136Z self.sleep()
2024-09-12T12:45:04.843306036Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 360, in sleep
2024-09-12T12:45:04.843311136Z ready = select.select([self.PIPE[0]], [], [], 1.0)
2024-09-12T12:45:04.843315936Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:04.843320836Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
2024-09-12T12:45:04.843338936Z self.reap_workers()
2024-09-12T12:45:04.843343637Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
2024-09-12T12:45:04.843348837Z raise HaltServer(reason, self.WORKER_BOOT_ERROR)
2024-09-12T12:45:04.843353437Z gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
2024-09-12T12:45:04.843358437Z
2024-09-12T12:45:04.843362737Z During handling of the above exception, another exception occurred:
2024-09-12T12:45:04.843367237Z
2024-09-12T12:45:04.843371637Z Traceback (most recent call last):
2024-09-12T12:45:04.843376037Z File "/opt/python/3.11.8/bin/gunicorn", line 8, in
2024-09-12T12:45:04.843380837Z sys.exit(run())
2024-09-12T12:45:04.843385237Z ^^^^^
2024-09-12T12:45:04.843389637Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
2024-09-12T12:45:04.843394337Z WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]", prog=prog).run()
2024-09-12T12:45:04.843398837Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 236, in run
2024-09-12T12:45:04.843403437Z super().run()
2024-09-12T12:45:04.843407837Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 72, in run
2024-09-12T12:45:04.843412337Z Arbiter(self).run()
2024-09-12T12:45:04.843416737Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 229, in run
2024-09-12T12:45:04.843421237Z self.halt(reason=inst.reason, exit_status=inst.exit_status)
2024-09-12T12:45:04.843425837Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 342, in halt
2024-09-12T12:45:04.843430537Z self.stop()
2024-09-12T12:45:04.860256615Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 396, in stop
2024-09-12T12:45:04.860795520Z time.sleep(0.1)
2024-09-12T12:45:04.861067323Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
2024-09-12T12:45:04.861422527Z self.reap_workers()
2024-09-12T12:45:04.861657630Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
2024-09-12T12:45:04.862053834Z raise HaltServer(reason, self.WORKER_BOOT_ERROR)
2024-09-12T12:45:04.862302136Z gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Tasks
To be filled in by the engineer picking up the issue
Describe the bug
I have initiated (seperately) both "code" and "container" based deployments from the Local Development/Deployment approach, on Windows WSL using Dev Container. They succesfully deployed. However the main app is consisently failing to start in the App Service. I've redployed multiple times, in new resource groups, without any success.
Expected behavior
Both the admin and main chat app are running based on a standard, uncustomized local deployment
How does this bug make you feel?
Share a gif from giphy to tells us how you'd feel
Debugging information
Steps to reproduce
Steps to reproduce the behavior:
Screenshots
If applicable, add screenshots to help explain your problem.
Logs
Ok 2024-09-12T12:38:50.128 INFO - Pulling image: mcr.microsoft.com/appsvc/middleware:stage5 Ok 2024-09-12T12:38:50.408 INFO - stage5 Pulling from appsvc/middleware Ok 2024-09-12T12:38:50.428 INFO - Digest: sha256:6b3c135340f42bbf62290b3ff21556271c6e979753e7cafacae3e74191535b7e Ok 2024-09-12T12:38:50.429 INFO - Status: Image is up to date for mcr.microsoft.com/appsvc/middleware:stage5 Ok 2024-09-12T12:38:50.444 INFO - Pull Image successful, Time taken: 0 Seconds Ok 2024-09-12T12:38:50.544 INFO - Starting container for site Ok 2024-09-12T12:38:50.545 INFO - docker run -d --expose=8181 --name web-**_e59fbcc5_middleware -e WEBSITE_CORS_ALLOWED_ORIGINS=https://portal.azure.com,https://ms.portal.azure.com -e WEBSITE_CORS_SUPPORT_CREDENTIALS=False -e WEBSITE_SITE_NAME=web-c -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=web-****.azurewebsites.net -e WEBSITE_INSTANCE_ID=32a926d1a80c14b7a13d64095168e8c96cae058331d08d583124b88b7de0a309 -e HTTP_LOGGING_ENABLED=12024-09-12T12:38:51.606Z INFO - Initiating warmup request to container web-_0_e59fbcc5_msiProxy for site web-cgz7qvevojqp6 Ok 2024-09-12T12:38:51.635 INFO - Container web-cgz7*0_e59fbcc5_msiProxy for site web-**** initialized successfully and is ready to serve requests. Ok 2024-09-12T12:38:51.636 INFO - Initiating warmup request to container web-cg**6_0_e59fbcc5 for site web-cgz7qvevojqp6 Error 2024-09-12T12:39:49.558 ERROR - Container *ojqp6_0_e59fbcc5 for site **has exited, failing site start Ok 2024-09-12T12:39:49.558 INFO - Initiating warmup request to container *_middleware for site web-cgz7qvevojqp6 Ok 2024-09-12T12:39:50.289 INFO - Container web-cgz7qvevojqp6_0_e59fbcc5_middleware for site ** initialized successfully and is ready to serve requests. Ok 2024-09-12T12:39:50.331 ERROR - Container ***5 didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging. Ok 2024-09-12T12:39:50.351 INFO - Stopping site ***** because it failed during startup. Ok 2024-09-12T12:39:50.351
APPLICATION LOGS
2024-09-12T12:44:05.232332694Z", line 1204, in _gcd_import
2024-09-12T12:44:59.898283213Z File "", line 1176, in _find_and_load
2024-09-12T12:44:59.898306613Z File "", line 1147, in _find_and_load_unlocked
2024-09-12T12:44:59.898315613Z File "", line 690, in _load_unlocked
2024-09-12T12:44:59.898324014Z File "", line 940, in exec_module
2024-09-12T12:44:59.898332414Z File "", line 241, in _call_with_frames_removed
2024-09-12T12:44:59.898340814Z File "/tmp/8dcd26f27168ecd/app.py", line 22, in
2024-09-12T12:44:59.898349414Z from create_app import create_app # noqa: E402
2024-09-12T12:44:59.898357514Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898367114Z File "/tmp/8dcd26f27168ecd/create_app.py", line 17, in
2024-09-12T12:44:59.898375814Z from backend.batch.utilities.helpers.orchestrator_helper import Orchestrator
2024-09-12T12:44:59.898384114Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/orchestrator_helper.py", line 5, in
2024-09-12T12:44:59.898392714Z from ..orchestrator.strategies import get_orchestrator
2024-09-12T12:44:59.898400814Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/strategies.py", line 2, in
2024-09-12T12:44:59.898409514Z from .open_ai_functions import OpenAIFunctionsOrchestrator
2024-09-12T12:44:59.898417614Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/open_ai_functions.py", line 5, in
2024-09-12T12:44:59.898426215Z from .orchestrator_base import OrchestratorBase
2024-09-12T12:44:59.898434215Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/orchestrator_base.py", line 5, in
2024-09-12T12:44:59.898442815Z from ..loggers.conversation_logger import ConversationLogger
2024-09-12T12:44:59.898450915Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/loggers/conversation_logger.py", line 1, in
2024-09-12T12:44:59.898459415Z from ..helpers.azure_search_helper import AzureSearchHelper
2024-09-12T12:44:59.898467515Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/azure_search_helper.py", line 29, in
2024-09-12T12:44:59.898476015Z from .llm_helper import LLMHelper
2024-09-12T12:44:59.898484015Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/llm_helper.py", line 5, in
2024-09-12T12:44:59.898492415Z from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
2024-09-12T12:44:59.898500915Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/init.py", line 3, in
2024-09-12T12:44:59.898509315Z from semantic_kernel.kernel import Kernel
2024-09-12T12:44:59.898517316Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/kernel.py", line 31, in
2024-09-12T12:44:59.898533016Z from semantic_kernel.functions.function_result import FunctionResult
2024-09-12T12:44:59.898541416Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/init.py", line 5, in
2024-09-12T12:44:59.898550016Z from semantic_kernel.functions.kernel_function import KernelFunction
2024-09-12T12:44:59.898558216Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in
2024-09-12T12:44:59.898566716Z class KernelFunction(KernelBaseModel):
2024-09-12T12:44:59.898575516Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 202, in new
2024-09-12T12:44:59.898584016Z complete_model_class(
2024-09-12T12:44:59.898591916Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 572, in complete_model_class
2024-09-12T12:44:59.898600416Z generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper),
2024-09-12T12:44:59.898608717Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898616917Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature
2024-09-12T12:44:59.898625417Z merged_params = _generate_signature_parameters(init, fields, config_wrapper)
2024-09-12T12:44:59.898633517Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898641717Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters
2024-09-12T12:44:59.898650217Z kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)}
2024-09-12T12:44:59.898658417Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898666717Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/fields.py", line 546, in get_default
2024-09-12T12:44:59.898675017Z return _utils.smart_deepcopy(self.default)
2024-09-12T12:44:59.898682917Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898690917Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy
2024-09-12T12:44:59.898699417Z return deepcopy(obj) # slowest way when we actually might need a deepcopy
2024-09-12T12:44:59.898707618Z ^^^^^^^^^^^^^
2024-09-12T12:44:59.898715618Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:44:59.898723918Z y = _reconstruct(x, memo, *rv)
2024-09-12T12:44:59.898731818Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898739818Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:44:59.898754818Z state = deepcopy(state, memo)
2024-09-12T12:44:59.898763018Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898771018Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:44:59.898779418Z y = copier(x, memo)
2024-09-12T12:44:59.898787318Z ^^^^^^^^^^^^^^^
2024-09-12T12:44:59.898795318Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:44:05.232365395Z / \ __ ___ ___
2024-09-12T12:44:05.232371095Z / /\ \_ / | _ _/ \ 2024-09-12T12:44:05.232375595Z / | \/ /| | /| | \/\ / 2024-09-12T12:44:05.232379795Z __| /__ __/ |_| \ > 2024-09-12T12:44:05.232384195Z \/ \/ \/ 2024-09-12T12:44:05.232388195Z A P P S E R V I C E O N L I N U X 2024-09-12T12:44:05.232392195Z 2024-09-12T12:44:05.232396095Z Documentation: http://aka.ms/webapp-linux 2024-09-12T12:44:05.232400095Z Python 3.11.8 2024-09-12T12:44:05.232403995Z Note: Any data outside '/home' is not persisted 2024-09-12T12:44:07.247320554Z Starting OpenBSD Secure Shell server: sshd. 2024-09-12T12:44:07.259004280Z WEBSITES_INCLUDE_CLOUD_CERTS is not set to true. 2024-09-12T12:44:07.365299328Z App Command Line not configured, will attempt auto-detect 2024-09-12T12:44:07.365330028Z Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite 2024-09-12T12:44:07.405556963Z Found build manifest file at '/home/site/wwwroot/oryx-manifest.toml'. Deserializing it... 2024-09-12T12:44:07.409694207Z Build Operation ID: 7f022b7e419fd428 2024-09-12T12:44:07.413181045Z Oryx Version: 0.2.20240501.1, Commit: f83f88d3cfb8bb6d3e2765e1dcd218eb0814a095, ReleaseTagName: 20240501.1 2024-09-12T12:44:07.413211045Z Output is compressed. Extracting it... 2024-09-12T12:44:07.416175877Z Extracting '/home/site/wwwroot/output.tar.gz' to directory '/tmp/8dcd26f27168ecd'... 2024-09-12T12:44:31.114636292Z App path is set to '/tmp/8dcd26f27168ecd' 2024-09-12T12:44:31.260690066Z Detected an app based on Flask 2024-09-12T12:44:31.260886468Z Generating
gunicorn
command for 'app:app' 2024-09-12T12:44:31.357977913Z Writing output script to '/opt/startup/startup.sh' 2024-09-12T12:44:31.425621742Z Using packages from virtual environment antenv located at /tmp/8dcd26f27168ecd/antenv. 2024-09-12T12:44:31.425660242Z Updated PYTHONPATH to '/opt/startup/app_logs:/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages' 2024-09-12T12:44:32.573330103Z [2024-09-12 12:44:32 +0000] [81] [INFO] Starting gunicorn 22.0.0 2024-09-12T12:44:32.588917671Z [2024-09-12 12:44:32 +0000] [81] [INFO] Listening at: http://0.0.0.0:8000 (81) 2024-09-12T12:44:32.588951671Z [2024-09-12 12:44:32 +0000] [81] [INFO] Using worker: sync 2024-09-12T12:44:32.595079137Z [2024-09-12 12:44:32 +0000] [84] [INFO] Booting worker with pid: 84 2024-09-12T12:44:32.672615072Z [2024-09-12 12:44:32 +0000] [85] [INFO] Booting worker with pid: 85 2024-09-12T12:44:32.786003293Z [2024-09-12 12:44:32 +0000] [86] [INFO] Booting worker with pid: 86 2024-09-12T12:44:32.840630881Z [2024-09-12 12:44:32 +0000] [87] [INFO] Booting worker with pid: 87 2024-09-12T12:44:32.927612517Z [2024-09-12 12:44:32 +0000] [88] [INFO] Booting worker with pid: 88 2024-09-12T12:44:33.001185209Z [2024-09-12 12:44:32 +0000] [89] [INFO] Booting worker with pid: 89 2024-09-12T12:44:33.062721871Z [2024-09-12 12:44:33 +0000] [90] [INFO] Booting worker with pid: 90 2024-09-12T12:44:33.154343657Z [2024-09-12 12:44:33 +0000] [91] [INFO] Booting worker with pid: 91 2024-09-12T12:44:33.248754473Z [2024-09-12 12:44:33 +0000] [92] [INFO] Booting worker with pid: 92 2024-09-12T12:44:59.898021010Z [2024-09-12 12:44:59 +0000] [91] [ERROR] Exception in worker process 2024-09-12T12:44:59.898078411Z Traceback (most recent call last): 2024-09-12T12:44:59.898090311Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker 2024-09-12T12:44:59.898100011Z worker.init_process() 2024-09-12T12:44:59.898108511Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process 2024-09-12T12:44:59.898117311Z self.load_wsgi() 2024-09-12T12:44:59.898125411Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi 2024-09-12T12:44:59.898134012Z self.wsgi = self.app.wsgi() 2024-09-12T12:44:59.898142012Z ^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898150112Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi 2024-09-12T12:44:59.898158612Z self.callable = self.load() 2024-09-12T12:44:59.898166612Z ^^^^^^^^^^^ 2024-09-12T12:44:59.898174612Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load 2024-09-12T12:44:59.898183212Z return self.load_wsgiapp() 2024-09-12T12:44:59.898191212Z ^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898199112Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp 2024-09-12T12:44:59.898207912Z return util.import_app(self.app_uri) 2024-09-12T12:44:59.898215912Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898224112Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/util.py", line 371, in import_app 2024-09-12T12:44:59.898232613Z mod = importlib.import_module(module) 2024-09-12T12:44:59.898240613Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898248613Z File "/opt/python/3.11.8/lib/python3.11/importlib/init.py", line 126, in import_module 2024-09-12T12:44:59.898256913Z return _bootstrap._gcd_import(name[level:], package, level) 2024-09-12T12:44:59.898265713Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898273913Z File "......................
..................................... 2024-09-12T12:45:02.925627225Z from ..helpers.azure_search_helper import AzureSearchHelper 2024-09-12T12:45:02.925631725Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/azure_search_helper.py", line 29, in
2024-09-12T12:45:02.925640625Z from .llm_helper import LLMHelper
2024-09-12T12:45:02.925645125Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/llm_helper.py", line 5, in
2024-09-12T12:45:02.925649825Z from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
2024-09-12T12:45:02.925654225Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/init.py", line 3, in
2024-09-12T12:45:02.925658925Z from semantic_kernel.kernel import Kernel
2024-09-12T12:45:02.925663225Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/kernel.py", line 31, in
2024-09-12T12:45:02.925667925Z from semantic_kernel.functions.function_result import FunctionResult
2024-09-12T12:45:02.925672425Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/init.py", line 5, in
2024-09-12T12:45:02.925677125Z from semantic_kernel.functions.kernel_function import KernelFunction
2024-09-12T12:45:02.925681625Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in
2024-09-12T12:45:02.925686326Z class KernelFunction(KernelBaseModel):
2024-09-12T12:45:02.925691126Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 202, in new
2024-09-12T12:45:02.925695926Z complete_model_class(
2024-09-12T12:45:02.925700226Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 572, in complete_model_class
2024-09-12T12:45:02.925704926Z generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper),
2024-09-12T12:45:02.925709426Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925714126Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature
2024-09-12T12:45:02.925718726Z merged_params = _generate_signature_parameters(init, fields, config_wrapper)
2024-09-12T12:45:02.925723326Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925727826Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters
2024-09-12T12:45:02.925732526Z kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)}
2024-09-12T12:45:02.925737026Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925741526Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/fields.py", line 546, in get_default
2024-09-12T12:45:02.925746126Z return _utils.smart_deepcopy(self.default)
2024-09-12T12:45:02.925754526Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925759126Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy
2024-09-12T12:45:02.925763726Z return deepcopy(obj) # slowest way when we actually might need a deepcopy
2024-09-12T12:45:02.925768226Z ^^^^^^^^^^^^^
2024-09-12T12:45:02.925772526Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:02.925777126Z y = _reconstruct(x, memo, rv)
2024-09-12T12:45:02.925781427Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925785827Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:02.925790327Z state = deepcopy(state, memo)
2024-09-12T12:45:02.925794727Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925799027Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:02.925803527Z y = copier(x, memo)
2024-09-12T12:45:02.925807827Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925812127Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:02.925817227Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:02.925821827Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925826327Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:02.925830827Z y = _reconstruct(x, memo, rv)
2024-09-12T12:45:02.925835227Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925839627Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:02.925844127Z state = deepcopy(state, memo)
2024-09-12T12:45:02.925848427Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925852927Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:02.925857427Z y = copier(x, memo)
2024-09-12T12:45:02.925861827Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925866127Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in _deepcopy_tuple
2024-09-12T12:45:02.925873628Z y = [deepcopy(a, memo) for a in x]
2024-09-12T12:45:02.925878128Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925882428Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in
2024-09-12T12:45:02.925887028Z y = [deepcopy(a, memo) for a in x]
2024-09-12T12:45:02.925891428Z ^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925895828Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:02.925900328Z y = copier(x, memo)
2024-09-12T12:45:02.925904728Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925913228Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:02.925917828Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:02.925922228Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925926728Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:02.925931228Z y = _reconstruct(x, memo, *rv)
2024-09-12T12:45:02.925935528Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925940028Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:02.925944528Z state = deepcopy(state, memo)
2024-09-12T12:45:02.925948828Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925953228Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:02.925957728Z y = copier(x, memo)
2024-09-12T12:45:02.925962028Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925966428Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:02.925970929Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:02.925975329Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:02.925979729Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 161, in deepcopy
2024-09-12T12:45:02.925984229Z rv = reductor(4)
2024-09-12T12:45:02.925988529Z ^^^^^^^^^^^
2024-09-12T12:45:02.925992929Z TypeError: cannot pickle '_thread.RLock' object
2024-09-12T12:45:02.927983150Z [2024-09-12 12:45:02 +0000] [89] [INFO] Worker exiting (pid: 89)
2024-09-12T12:45:03.042975662Z [2024-09-12 12:45:03 +0000] [88] [ERROR] Exception in worker process
2024-09-12T12:45:03.043013162Z Traceback (most recent call last):
2024-09-12T12:45:03.043020262Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
2024-09-12T12:45:03.043025962Z worker.init_process()
2024-09-12T12:45:03.043031162Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process
2024-09-12T12:45:03.043036362Z self.load_wsgi()
2024-09-12T12:45:03.043041362Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
2024-09-12T12:45:03.043046463Z self.wsgi = self.app.wsgi()
2024-09-12T12:45:03.043051363Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043056163Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi
2024-09-12T12:45:03.043061163Z self.callable = self.load()
2024-09-12T12:45:03.043066063Z ^^^^^^^^^^^
2024-09-12T12:45:03.043070963Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
2024-09-12T12:45:03.043088963Z return self.load_wsgiapp()
2024-09-12T12:45:03.043093863Z ^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043098563Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
2024-09-12T12:45:03.043103563Z return util.import_app(self.app_uri)
2024-09-12T12:45:03.043108363Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043112863Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/util.py", line 371, in import_app
2024-09-12T12:45:03.043117763Z mod = importlib.import_module(module)
2024-09-12T12:45:03.043122563Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043127163Z File "/opt/python/3.11.8/lib/python3.11/importlib/init.py", line 126, in import_module
2024-09-12T12:45:03.043132163Z return _bootstrap._gcd_import(name[level:], package, level)
2024-09-12T12:45:03.043137063Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043141964Z File "", line 1204, in _gcd_import
2024-09-12T12:45:03.043147264Z File "", line 1176, in _find_and_load
2024-09-12T12:45:03.043152264Z File "", line 1147, in _find_and_load_unlocked
2024-09-12T12:45:03.043157264Z File "", line 690, in _load_unlocked
2024-09-12T12:45:03.043162164Z File "", line 940, in exec_module
2024-09-12T12:45:03.043167264Z File "", line 241, in _call_with_frames_removed
2024-09-12T12:45:03.043172264Z File "/tmp/8dcd26f27168ecd/app.py", line 22, in
2024-09-12T12:45:03.043177264Z from create_app import create_app # noqa: E402
2024-09-12T12:45:03.043182164Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043188264Z File "/tmp/8dcd26f27168ecd/create_app.py", line 17, in
2024-09-12T12:45:03.043193364Z from backend.batch.utilities.helpers.orchestrator_helper import Orchestrator
2024-09-12T12:45:03.043197964Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/orchestrator_helper.py", line 5, in
2024-09-12T12:45:03.043202764Z from ..orchestrator.strategies import get_orchestrator
2024-09-12T12:45:03.043207264Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/strategies.py", line 2, in
2024-09-12T12:45:03.043212164Z from .open_ai_functions import OpenAIFunctionsOrchestrator
2024-09-12T12:45:03.043216564Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/open_ai_functions.py", line 5, in
2024-09-12T12:45:03.043221364Z from .orchestrator_base import OrchestratorBase
2024-09-12T12:45:03.043225864Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/orchestrator_base.py", line 5, in
2024-09-12T12:45:03.043235565Z from ..loggers.conversation_logger import ConversationLogger
2024-09-12T12:45:03.043244365Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/loggers/conversation_logger.py", line 1, in
2024-09-12T12:45:03.043249165Z from ..helpers.azure_search_helper import AzureSearchHelper
2024-09-12T12:45:03.043253865Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/azure_search_helper.py", line 29, in
2024-09-12T12:45:03.043258765Z from .llm_helper import LLMHelper
2024-09-12T12:45:03.043263665Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/llm_helper.py", line 5, in
2024-09-12T12:45:03.043268565Z from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
2024-09-12T12:45:03.043273165Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/init.py", line 3, in
2024-09-12T12:45:03.043278065Z from semantic_kernel.kernel import Kernel
2024-09-12T12:45:03.043282665Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/kernel.py", line 31, in
2024-09-12T12:45:03.043287465Z from semantic_kernel.functions.function_result import FunctionResult
2024-09-12T12:45:03.043292065Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/init.py", line 5, in
2024-09-12T12:45:03.043296965Z from semantic_kernel.functions.kernel_function import KernelFunction
2024-09-12T12:45:03.043301665Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in
2024-09-12T12:45:03.043306565Z class KernelFunction(KernelBaseModel):
2024-09-12T12:45:03.043311665Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 202, in new
2024-09-12T12:45:03.043316565Z complete_model_class(
2024-09-12T12:45:03.043321165Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 572, in complete_model_class
2024-09-12T12:45:03.043325965Z generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper),
2024-09-12T12:45:03.043330666Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043335366Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature
2024-09-12T12:45:03.043340266Z merged_params = _generate_signature_parameters(init, fields, config_wrapper)
2024-09-12T12:45:03.043344866Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043349566Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters
2024-09-12T12:45:03.043358666Z kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)}
2024-09-12T12:45:03.043363466Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043367966Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/fields.py", line 546, in get_default
2024-09-12T12:45:03.043372666Z return _utils.smart_deepcopy(self.default)
2024-09-12T12:45:03.043377166Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043381666Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy
2024-09-12T12:45:03.043386566Z return deepcopy(obj) # slowest way when we actually might need a deepcopy
2024-09-12T12:45:03.043391466Z ^^^^^^^^^^^^^
2024-09-12T12:45:03.043395866Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:03.043400366Z y = _reconstruct(x, memo, rv)
2024-09-12T12:45:03.043404866Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043409266Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:03.043413866Z state = deepcopy(state, memo)
2024-09-12T12:45:03.043418266Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043422766Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:03.043427467Z y = copier(x, memo)
2024-09-12T12:45:03.043432067Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043436367Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:03.043441367Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:03.043445767Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043450267Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:03.043454867Z y = _reconstruct(x, memo, rv)
2024-09-12T12:45:03.043459367Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043463867Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:03.043468367Z state = deepcopy(state, memo)
2024-09-12T12:45:03.043472767Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043477067Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:03.043481667Z y = copier(x, memo)
2024-09-12T12:45:03.043486067Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043490467Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in _deepcopy_tuple
2024-09-12T12:45:03.043495167Z y = [deepcopy(a, memo) for a in x]
2024-09-12T12:45:03.043499567Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043508267Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in
2024-09-12T12:45:03.043513367Z y = [deepcopy(a, memo) for a in x]
2024-09-12T12:45:03.043517867Z ^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043522368Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:03.043526968Z y = copier(x, memo)
2024-09-12T12:45:03.043531468Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043535868Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:03.043540468Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:03.043545068Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043549568Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy
2024-09-12T12:45:03.043554168Z y = _reconstruct(x, memo, *rv)
2024-09-12T12:45:03.043558668Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043563168Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct
2024-09-12T12:45:03.043567668Z state = deepcopy(state, memo)
2024-09-12T12:45:03.043572168Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043576668Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy
2024-09-12T12:45:03.043581368Z y = copier(x, memo)
2024-09-12T12:45:03.043585768Z ^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043590068Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict
2024-09-12T12:45:03.043594668Z y[deepcopy(key, memo)] = deepcopy(value, memo)
2024-09-12T12:45:03.043599168Z ^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:03.043603668Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 161, in deepcopy
2024-09-12T12:45:03.043608268Z rv = reductor(4)
2024-09-12T12:45:03.043612668Z ^^^^^^^^^^^
2024-09-12T12:45:03.043616969Z TypeError: cannot pickle '_thread.RLock' object
2024-09-12T12:45:03.043621369Z [2024-09-12 12:45:03 +0000] [88] [INFO] Worker exiting (pid: 88)
2024-09-12T12:45:04.843246335Z [2024-09-12 12:45:04 +0000] [81] [ERROR] Worker (pid:92) exited with code 3
2024-09-12T12:45:04.843288836Z Traceback (most recent call last):
2024-09-12T12:45:04.843295436Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 209, in run
2024-09-12T12:45:04.843301136Z self.sleep()
2024-09-12T12:45:04.843306036Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 360, in sleep
2024-09-12T12:45:04.843311136Z ready = select.select([self.PIPE[0]], [], [], 1.0)
2024-09-12T12:45:04.843315936Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-12T12:45:04.843320836Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
2024-09-12T12:45:04.843338936Z self.reap_workers()
2024-09-12T12:45:04.843343637Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
2024-09-12T12:45:04.843348837Z raise HaltServer(reason, self.WORKER_BOOT_ERROR)
2024-09-12T12:45:04.843353437Z gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
2024-09-12T12:45:04.843358437Z
2024-09-12T12:45:04.843362737Z During handling of the above exception, another exception occurred:
2024-09-12T12:45:04.843367237Z
2024-09-12T12:45:04.843371637Z Traceback (most recent call last):
2024-09-12T12:45:04.843376037Z File "/opt/python/3.11.8/bin/gunicorn", line 8, in
2024-09-12T12:45:04.843380837Z sys.exit(run())
2024-09-12T12:45:04.843385237Z ^^^^^
2024-09-12T12:45:04.843389637Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
2024-09-12T12:45:04.843394337Z WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]", prog=prog).run()
2024-09-12T12:45:04.843398837Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 236, in run
2024-09-12T12:45:04.843403437Z super().run()
2024-09-12T12:45:04.843407837Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 72, in run
2024-09-12T12:45:04.843412337Z Arbiter(self).run()
2024-09-12T12:45:04.843416737Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 229, in run
2024-09-12T12:45:04.843421237Z self.halt(reason=inst.reason, exit_status=inst.exit_status)
2024-09-12T12:45:04.843425837Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 342, in halt
2024-09-12T12:45:04.843430537Z self.stop()
2024-09-12T12:45:04.860256615Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 396, in stop
2024-09-12T12:45:04.860795520Z time.sleep(0.1)
2024-09-12T12:45:04.861067323Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
2024-09-12T12:45:04.861422527Z self.reap_workers()
2024-09-12T12:45:04.861657630Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
2024-09-12T12:45:04.862053834Z raise HaltServer(reason, self.WORKER_BOOT_ERROR)
2024-09-12T12:45:04.862302136Z gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Tasks
To be filled in by the engineer picking up the issue