search

Azure-Samples / chat-with-your-data-solution-accelerator

A Solution Accelerator for the RAG pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences. This includes most common requirements and best practices.
https://azure.microsoft.com/products/search
MIT License
815 stars 416 forks source link

Main App Failed to Boot #1304

Closed DSOTM-RSA closed 1 month ago

DSOTM-RSA commented 1 month ago

Describe the bug

I have initiated (seperately) both "code" and "container" based deployments from the Local Development/Deployment approach, on Windows WSL using Dev Container. They succesfully deployed. However the main app is consisently failing to start in the App Service. I've redployed multiple times, in new resource groups, without any success.

Expected behavior

Both the admin and main chat app are running based on a standard, uncustomized local deployment

How does this bug make you feel?

Share a gif from giphy to tells us how you'd feel

image

Debugging information

Steps to reproduce

Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Screenshots

If applicable, add screenshots to help explain your problem.

Logs

Ok 2024-09-12T12:38:50.128 INFO - Pulling image: mcr.microsoft.com/appsvc/middleware:stage5 Ok 2024-09-12T12:38:50.408 INFO - stage5 Pulling from appsvc/middleware Ok 2024-09-12T12:38:50.428 INFO - Digest: sha256:6b3c135340f42bbf62290b3ff21556271c6e979753e7cafacae3e74191535b7e Ok 2024-09-12T12:38:50.429 INFO - Status: Image is up to date for mcr.microsoft.com/appsvc/middleware:stage5 Ok 2024-09-12T12:38:50.444 INFO - Pull Image successful, Time taken: 0 Seconds Ok 2024-09-12T12:38:50.544 INFO - Starting container for site Ok 2024-09-12T12:38:50.545 INFO - docker run -d --expose=8181 --name web-**_e59fbcc5_middleware -e WEBSITE_CORS_ALLOWED_ORIGINS=https://portal.azure.com,https://ms.portal.azure.com -e WEBSITE_CORS_SUPPORT_CREDENTIALS=False -e WEBSITE_SITE_NAME=web-c -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=web-****.azurewebsites.net -e WEBSITE_INSTANCE_ID=32a926d1a80c14b7a13d64095168e8c96cae058331d08d583124b88b7de0a309 -e HTTP_LOGGING_ENABLED=12024-09-12T12:38:51.606Z INFO - Initiating warmup request to container web-_0_e59fbcc5_msiProxy for site web-cgz7qvevojqp6 Ok 2024-09-12T12:38:51.635 INFO - Container web-cgz7*0_e59fbcc5_msiProxy for site web-**** initialized successfully and is ready to serve requests. Ok 2024-09-12T12:38:51.636 INFO - Initiating warmup request to container web-cg**6_0_e59fbcc5 for site web-cgz7qvevojqp6 Error 2024-09-12T12:39:49.558 ERROR - Container *ojqp6_0_e59fbcc5 for site **has exited, failing site start Ok 2024-09-12T12:39:49.558 INFO - Initiating warmup request to container *_middleware for site web-cgz7qvevojqp6 Ok 2024-09-12T12:39:50.289 INFO - Container web-cgz7qvevojqp6_0_e59fbcc5_middleware for site ** initialized successfully and is ready to serve requests. Ok 2024-09-12T12:39:50.331 ERROR - Container ***5 didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging. Ok 2024-09-12T12:39:50.351 INFO - Stopping site ***** because it failed during startup. Ok 2024-09-12T12:39:50.351

APPLICATION LOGS

2024-09-12T12:44:05.232332694Z
2024-09-12T12:44:05.232365395Z / \ __ ___ ___
2024-09-12T12:44:05.232371095Z / /\ \_ / | _ _/ \ 2024-09-12T12:44:05.232375595Z / | \/ /| | /| | \/\ / 2024-09-12T12:44:05.232379795Z __| /__ __/ |_| \ > 2024-09-12T12:44:05.232384195Z \/ \/ \/ 2024-09-12T12:44:05.232388195Z A P P S E R V I C E O N L I N U X 2024-09-12T12:44:05.232392195Z 2024-09-12T12:44:05.232396095Z Documentation: http://aka.ms/webapp-linux 2024-09-12T12:44:05.232400095Z Python 3.11.8 2024-09-12T12:44:05.232403995Z Note: Any data outside '/home' is not persisted 2024-09-12T12:44:07.247320554Z Starting OpenBSD Secure Shell server: sshd. 2024-09-12T12:44:07.259004280Z WEBSITES_INCLUDE_CLOUD_CERTS is not set to true. 2024-09-12T12:44:07.365299328Z App Command Line not configured, will attempt auto-detect 2024-09-12T12:44:07.365330028Z Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite 2024-09-12T12:44:07.405556963Z Found build manifest file at '/home/site/wwwroot/oryx-manifest.toml'. Deserializing it... 2024-09-12T12:44:07.409694207Z Build Operation ID: 7f022b7e419fd428 2024-09-12T12:44:07.413181045Z Oryx Version: 0.2.20240501.1, Commit: f83f88d3cfb8bb6d3e2765e1dcd218eb0814a095, ReleaseTagName: 20240501.1 2024-09-12T12:44:07.413211045Z Output is compressed. Extracting it... 2024-09-12T12:44:07.416175877Z Extracting '/home/site/wwwroot/output.tar.gz' to directory '/tmp/8dcd26f27168ecd'... 2024-09-12T12:44:31.114636292Z App path is set to '/tmp/8dcd26f27168ecd' 2024-09-12T12:44:31.260690066Z Detected an app based on Flask 2024-09-12T12:44:31.260886468Z Generating gunicorn command for 'app:app' 2024-09-12T12:44:31.357977913Z Writing output script to '/opt/startup/startup.sh' 2024-09-12T12:44:31.425621742Z Using packages from virtual environment antenv located at /tmp/8dcd26f27168ecd/antenv. 2024-09-12T12:44:31.425660242Z Updated PYTHONPATH to '/opt/startup/app_logs:/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages' 2024-09-12T12:44:32.573330103Z [2024-09-12 12:44:32 +0000] [81] [INFO] Starting gunicorn 22.0.0 2024-09-12T12:44:32.588917671Z [2024-09-12 12:44:32 +0000] [81] [INFO] Listening at: http://0.0.0.0:8000 (81) 2024-09-12T12:44:32.588951671Z [2024-09-12 12:44:32 +0000] [81] [INFO] Using worker: sync 2024-09-12T12:44:32.595079137Z [2024-09-12 12:44:32 +0000] [84] [INFO] Booting worker with pid: 84 2024-09-12T12:44:32.672615072Z [2024-09-12 12:44:32 +0000] [85] [INFO] Booting worker with pid: 85 2024-09-12T12:44:32.786003293Z [2024-09-12 12:44:32 +0000] [86] [INFO] Booting worker with pid: 86 2024-09-12T12:44:32.840630881Z [2024-09-12 12:44:32 +0000] [87] [INFO] Booting worker with pid: 87 2024-09-12T12:44:32.927612517Z [2024-09-12 12:44:32 +0000] [88] [INFO] Booting worker with pid: 88 2024-09-12T12:44:33.001185209Z [2024-09-12 12:44:32 +0000] [89] [INFO] Booting worker with pid: 89 2024-09-12T12:44:33.062721871Z [2024-09-12 12:44:33 +0000] [90] [INFO] Booting worker with pid: 90 2024-09-12T12:44:33.154343657Z [2024-09-12 12:44:33 +0000] [91] [INFO] Booting worker with pid: 91 2024-09-12T12:44:33.248754473Z [2024-09-12 12:44:33 +0000] [92] [INFO] Booting worker with pid: 92 2024-09-12T12:44:59.898021010Z [2024-09-12 12:44:59 +0000] [91] [ERROR] Exception in worker process 2024-09-12T12:44:59.898078411Z Traceback (most recent call last): 2024-09-12T12:44:59.898090311Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker 2024-09-12T12:44:59.898100011Z worker.init_process() 2024-09-12T12:44:59.898108511Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process 2024-09-12T12:44:59.898117311Z self.load_wsgi() 2024-09-12T12:44:59.898125411Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi 2024-09-12T12:44:59.898134012Z self.wsgi = self.app.wsgi() 2024-09-12T12:44:59.898142012Z ^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898150112Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi 2024-09-12T12:44:59.898158612Z self.callable = self.load() 2024-09-12T12:44:59.898166612Z ^^^^^^^^^^^ 2024-09-12T12:44:59.898174612Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load 2024-09-12T12:44:59.898183212Z return self.load_wsgiapp() 2024-09-12T12:44:59.898191212Z ^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898199112Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp 2024-09-12T12:44:59.898207912Z return util.import_app(self.app_uri) 2024-09-12T12:44:59.898215912Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898224112Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/util.py", line 371, in import_app 2024-09-12T12:44:59.898232613Z mod = importlib.import_module(module) 2024-09-12T12:44:59.898240613Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898248613Z File "/opt/python/3.11.8/lib/python3.11/importlib/init.py", line 126, in import_module 2024-09-12T12:44:59.898256913Z return _bootstrap._gcd_import(name[level:], package, level) 2024-09-12T12:44:59.898265713Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898273913Z File "", line 1204, in _gcd_import 2024-09-12T12:44:59.898283213Z File "", line 1176, in _find_and_load 2024-09-12T12:44:59.898306613Z File "", line 1147, in _find_and_load_unlocked 2024-09-12T12:44:59.898315613Z File "", line 690, in _load_unlocked 2024-09-12T12:44:59.898324014Z File "", line 940, in exec_module 2024-09-12T12:44:59.898332414Z File "", line 241, in _call_with_frames_removed 2024-09-12T12:44:59.898340814Z File "/tmp/8dcd26f27168ecd/app.py", line 22, in 2024-09-12T12:44:59.898349414Z from create_app import create_app # noqa: E402 2024-09-12T12:44:59.898357514Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898367114Z File "/tmp/8dcd26f27168ecd/create_app.py", line 17, in 2024-09-12T12:44:59.898375814Z from backend.batch.utilities.helpers.orchestrator_helper import Orchestrator 2024-09-12T12:44:59.898384114Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/orchestrator_helper.py", line 5, in 2024-09-12T12:44:59.898392714Z from ..orchestrator.strategies import get_orchestrator 2024-09-12T12:44:59.898400814Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/strategies.py", line 2, in 2024-09-12T12:44:59.898409514Z from .open_ai_functions import OpenAIFunctionsOrchestrator 2024-09-12T12:44:59.898417614Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/open_ai_functions.py", line 5, in 2024-09-12T12:44:59.898426215Z from .orchestrator_base import OrchestratorBase 2024-09-12T12:44:59.898434215Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/orchestrator_base.py", line 5, in 2024-09-12T12:44:59.898442815Z from ..loggers.conversation_logger import ConversationLogger 2024-09-12T12:44:59.898450915Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/loggers/conversation_logger.py", line 1, in 2024-09-12T12:44:59.898459415Z from ..helpers.azure_search_helper import AzureSearchHelper 2024-09-12T12:44:59.898467515Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/azure_search_helper.py", line 29, in 2024-09-12T12:44:59.898476015Z from .llm_helper import LLMHelper 2024-09-12T12:44:59.898484015Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/llm_helper.py", line 5, in 2024-09-12T12:44:59.898492415Z from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion 2024-09-12T12:44:59.898500915Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/init.py", line 3, in 2024-09-12T12:44:59.898509315Z from semantic_kernel.kernel import Kernel 2024-09-12T12:44:59.898517316Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/kernel.py", line 31, in 2024-09-12T12:44:59.898533016Z from semantic_kernel.functions.function_result import FunctionResult 2024-09-12T12:44:59.898541416Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/init.py", line 5, in 2024-09-12T12:44:59.898550016Z from semantic_kernel.functions.kernel_function import KernelFunction 2024-09-12T12:44:59.898558216Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in 2024-09-12T12:44:59.898566716Z class KernelFunction(KernelBaseModel): 2024-09-12T12:44:59.898575516Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 202, in new 2024-09-12T12:44:59.898584016Z complete_model_class( 2024-09-12T12:44:59.898591916Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 572, in complete_model_class 2024-09-12T12:44:59.898600416Z generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper), 2024-09-12T12:44:59.898608717Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898616917Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature 2024-09-12T12:44:59.898625417Z merged_params = _generate_signature_parameters(init, fields, config_wrapper) 2024-09-12T12:44:59.898633517Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898641717Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters 2024-09-12T12:44:59.898650217Z kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)} 2024-09-12T12:44:59.898658417Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898666717Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/fields.py", line 546, in get_default 2024-09-12T12:44:59.898675017Z return _utils.smart_deepcopy(self.default) 2024-09-12T12:44:59.898682917Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898690917Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy 2024-09-12T12:44:59.898699417Z return deepcopy(obj) # slowest way when we actually might need a deepcopy 2024-09-12T12:44:59.898707618Z ^^^^^^^^^^^^^ 2024-09-12T12:44:59.898715618Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy 2024-09-12T12:44:59.898723918Z y = _reconstruct(x, memo, *rv) 2024-09-12T12:44:59.898731818Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898739818Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct 2024-09-12T12:44:59.898754818Z state = deepcopy(state, memo) 2024-09-12T12:44:59.898763018Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898771018Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy 2024-09-12T12:44:59.898779418Z y = copier(x, memo) 2024-09-12T12:44:59.898787318Z ^^^^^^^^^^^^^^^ 2024-09-12T12:44:59.898795318Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict

......................

..................................... 2024-09-12T12:45:02.925627225Z from ..helpers.azure_search_helper import AzureSearchHelper 2024-09-12T12:45:02.925631725Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/azure_search_helper.py", line 29, in 2024-09-12T12:45:02.925640625Z from .llm_helper import LLMHelper 2024-09-12T12:45:02.925645125Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/llm_helper.py", line 5, in 2024-09-12T12:45:02.925649825Z from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion 2024-09-12T12:45:02.925654225Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/init.py", line 3, in 2024-09-12T12:45:02.925658925Z from semantic_kernel.kernel import Kernel 2024-09-12T12:45:02.925663225Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/kernel.py", line 31, in 2024-09-12T12:45:02.925667925Z from semantic_kernel.functions.function_result import FunctionResult 2024-09-12T12:45:02.925672425Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/init.py", line 5, in 2024-09-12T12:45:02.925677125Z from semantic_kernel.functions.kernel_function import KernelFunction 2024-09-12T12:45:02.925681625Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in 2024-09-12T12:45:02.925686326Z class KernelFunction(KernelBaseModel): 2024-09-12T12:45:02.925691126Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 202, in new 2024-09-12T12:45:02.925695926Z complete_model_class( 2024-09-12T12:45:02.925700226Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 572, in complete_model_class 2024-09-12T12:45:02.925704926Z generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper), 2024-09-12T12:45:02.925709426Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925714126Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature 2024-09-12T12:45:02.925718726Z merged_params = _generate_signature_parameters(init, fields, config_wrapper) 2024-09-12T12:45:02.925723326Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925727826Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters 2024-09-12T12:45:02.925732526Z kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)} 2024-09-12T12:45:02.925737026Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925741526Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/fields.py", line 546, in get_default 2024-09-12T12:45:02.925746126Z return _utils.smart_deepcopy(self.default) 2024-09-12T12:45:02.925754526Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925759126Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy 2024-09-12T12:45:02.925763726Z return deepcopy(obj) # slowest way when we actually might need a deepcopy 2024-09-12T12:45:02.925768226Z ^^^^^^^^^^^^^ 2024-09-12T12:45:02.925772526Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy 2024-09-12T12:45:02.925777126Z y = _reconstruct(x, memo, rv) 2024-09-12T12:45:02.925781427Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925785827Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct 2024-09-12T12:45:02.925790327Z state = deepcopy(state, memo) 2024-09-12T12:45:02.925794727Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925799027Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy 2024-09-12T12:45:02.925803527Z y = copier(x, memo) 2024-09-12T12:45:02.925807827Z ^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925812127Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict 2024-09-12T12:45:02.925817227Z y[deepcopy(key, memo)] = deepcopy(value, memo) 2024-09-12T12:45:02.925821827Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925826327Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy 2024-09-12T12:45:02.925830827Z y = _reconstruct(x, memo, rv) 2024-09-12T12:45:02.925835227Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925839627Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct 2024-09-12T12:45:02.925844127Z state = deepcopy(state, memo) 2024-09-12T12:45:02.925848427Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925852927Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy 2024-09-12T12:45:02.925857427Z y = copier(x, memo) 2024-09-12T12:45:02.925861827Z ^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925866127Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in _deepcopy_tuple 2024-09-12T12:45:02.925873628Z y = [deepcopy(a, memo) for a in x] 2024-09-12T12:45:02.925878128Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925882428Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in 2024-09-12T12:45:02.925887028Z y = [deepcopy(a, memo) for a in x] 2024-09-12T12:45:02.925891428Z ^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925895828Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy 2024-09-12T12:45:02.925900328Z y = copier(x, memo) 2024-09-12T12:45:02.925904728Z ^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925913228Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict 2024-09-12T12:45:02.925917828Z y[deepcopy(key, memo)] = deepcopy(value, memo) 2024-09-12T12:45:02.925922228Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925926728Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy 2024-09-12T12:45:02.925931228Z y = _reconstruct(x, memo, *rv) 2024-09-12T12:45:02.925935528Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925940028Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct 2024-09-12T12:45:02.925944528Z state = deepcopy(state, memo) 2024-09-12T12:45:02.925948828Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925953228Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy 2024-09-12T12:45:02.925957728Z y = copier(x, memo) 2024-09-12T12:45:02.925962028Z ^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925966428Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict 2024-09-12T12:45:02.925970929Z y[deepcopy(key, memo)] = deepcopy(value, memo) 2024-09-12T12:45:02.925975329Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:02.925979729Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 161, in deepcopy 2024-09-12T12:45:02.925984229Z rv = reductor(4) 2024-09-12T12:45:02.925988529Z ^^^^^^^^^^^ 2024-09-12T12:45:02.925992929Z TypeError: cannot pickle '_thread.RLock' object 2024-09-12T12:45:02.927983150Z [2024-09-12 12:45:02 +0000] [89] [INFO] Worker exiting (pid: 89) 2024-09-12T12:45:03.042975662Z [2024-09-12 12:45:03 +0000] [88] [ERROR] Exception in worker process 2024-09-12T12:45:03.043013162Z Traceback (most recent call last): 2024-09-12T12:45:03.043020262Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker 2024-09-12T12:45:03.043025962Z worker.init_process() 2024-09-12T12:45:03.043031162Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process 2024-09-12T12:45:03.043036362Z self.load_wsgi() 2024-09-12T12:45:03.043041362Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi 2024-09-12T12:45:03.043046463Z self.wsgi = self.app.wsgi() 2024-09-12T12:45:03.043051363Z ^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043056163Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi 2024-09-12T12:45:03.043061163Z self.callable = self.load() 2024-09-12T12:45:03.043066063Z ^^^^^^^^^^^ 2024-09-12T12:45:03.043070963Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load 2024-09-12T12:45:03.043088963Z return self.load_wsgiapp() 2024-09-12T12:45:03.043093863Z ^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043098563Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp 2024-09-12T12:45:03.043103563Z return util.import_app(self.app_uri) 2024-09-12T12:45:03.043108363Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043112863Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/util.py", line 371, in import_app 2024-09-12T12:45:03.043117763Z mod = importlib.import_module(module) 2024-09-12T12:45:03.043122563Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043127163Z File "/opt/python/3.11.8/lib/python3.11/importlib/init.py", line 126, in import_module 2024-09-12T12:45:03.043132163Z return _bootstrap._gcd_import(name[level:], package, level) 2024-09-12T12:45:03.043137063Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043141964Z File "", line 1204, in _gcd_import 2024-09-12T12:45:03.043147264Z File "", line 1176, in _find_and_load 2024-09-12T12:45:03.043152264Z File "", line 1147, in _find_and_load_unlocked 2024-09-12T12:45:03.043157264Z File "", line 690, in _load_unlocked 2024-09-12T12:45:03.043162164Z File "", line 940, in exec_module 2024-09-12T12:45:03.043167264Z File "", line 241, in _call_with_frames_removed 2024-09-12T12:45:03.043172264Z File "/tmp/8dcd26f27168ecd/app.py", line 22, in 2024-09-12T12:45:03.043177264Z from create_app import create_app # noqa: E402 2024-09-12T12:45:03.043182164Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043188264Z File "/tmp/8dcd26f27168ecd/create_app.py", line 17, in 2024-09-12T12:45:03.043193364Z from backend.batch.utilities.helpers.orchestrator_helper import Orchestrator 2024-09-12T12:45:03.043197964Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/orchestrator_helper.py", line 5, in 2024-09-12T12:45:03.043202764Z from ..orchestrator.strategies import get_orchestrator 2024-09-12T12:45:03.043207264Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/strategies.py", line 2, in 2024-09-12T12:45:03.043212164Z from .open_ai_functions import OpenAIFunctionsOrchestrator 2024-09-12T12:45:03.043216564Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/open_ai_functions.py", line 5, in 2024-09-12T12:45:03.043221364Z from .orchestrator_base import OrchestratorBase 2024-09-12T12:45:03.043225864Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/orchestrator/orchestrator_base.py", line 5, in 2024-09-12T12:45:03.043235565Z from ..loggers.conversation_logger import ConversationLogger 2024-09-12T12:45:03.043244365Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/loggers/conversation_logger.py", line 1, in 2024-09-12T12:45:03.043249165Z from ..helpers.azure_search_helper import AzureSearchHelper 2024-09-12T12:45:03.043253865Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/azure_search_helper.py", line 29, in 2024-09-12T12:45:03.043258765Z from .llm_helper import LLMHelper 2024-09-12T12:45:03.043263665Z File "/tmp/8dcd26f27168ecd/backend/batch/utilities/helpers/llm_helper.py", line 5, in 2024-09-12T12:45:03.043268565Z from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion 2024-09-12T12:45:03.043273165Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/init.py", line 3, in 2024-09-12T12:45:03.043278065Z from semantic_kernel.kernel import Kernel 2024-09-12T12:45:03.043282665Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/kernel.py", line 31, in 2024-09-12T12:45:03.043287465Z from semantic_kernel.functions.function_result import FunctionResult 2024-09-12T12:45:03.043292065Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/init.py", line 5, in 2024-09-12T12:45:03.043296965Z from semantic_kernel.functions.kernel_function import KernelFunction 2024-09-12T12:45:03.043301665Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in 2024-09-12T12:45:03.043306565Z class KernelFunction(KernelBaseModel): 2024-09-12T12:45:03.043311665Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 202, in new 2024-09-12T12:45:03.043316565Z complete_model_class( 2024-09-12T12:45:03.043321165Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 572, in complete_model_class 2024-09-12T12:45:03.043325965Z generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper), 2024-09-12T12:45:03.043330666Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043335366Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature 2024-09-12T12:45:03.043340266Z merged_params = _generate_signature_parameters(init, fields, config_wrapper) 2024-09-12T12:45:03.043344866Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043349566Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters 2024-09-12T12:45:03.043358666Z kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)} 2024-09-12T12:45:03.043363466Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043367966Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/fields.py", line 546, in get_default 2024-09-12T12:45:03.043372666Z return _utils.smart_deepcopy(self.default) 2024-09-12T12:45:03.043377166Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043381666Z File "/tmp/8dcd26f27168ecd/antenv/lib/python3.11/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy 2024-09-12T12:45:03.043386566Z return deepcopy(obj) # slowest way when we actually might need a deepcopy 2024-09-12T12:45:03.043391466Z ^^^^^^^^^^^^^ 2024-09-12T12:45:03.043395866Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy 2024-09-12T12:45:03.043400366Z y = _reconstruct(x, memo, rv) 2024-09-12T12:45:03.043404866Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043409266Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct 2024-09-12T12:45:03.043413866Z state = deepcopy(state, memo) 2024-09-12T12:45:03.043418266Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043422766Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy 2024-09-12T12:45:03.043427467Z y = copier(x, memo) 2024-09-12T12:45:03.043432067Z ^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043436367Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict 2024-09-12T12:45:03.043441367Z y[deepcopy(key, memo)] = deepcopy(value, memo) 2024-09-12T12:45:03.043445767Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043450267Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy 2024-09-12T12:45:03.043454867Z y = _reconstruct(x, memo, rv) 2024-09-12T12:45:03.043459367Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043463867Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct 2024-09-12T12:45:03.043468367Z state = deepcopy(state, memo) 2024-09-12T12:45:03.043472767Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043477067Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy 2024-09-12T12:45:03.043481667Z y = copier(x, memo) 2024-09-12T12:45:03.043486067Z ^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043490467Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in _deepcopy_tuple 2024-09-12T12:45:03.043495167Z y = [deepcopy(a, memo) for a in x] 2024-09-12T12:45:03.043499567Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043508267Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 211, in 2024-09-12T12:45:03.043513367Z y = [deepcopy(a, memo) for a in x] 2024-09-12T12:45:03.043517867Z ^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043522368Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy 2024-09-12T12:45:03.043526968Z y = copier(x, memo) 2024-09-12T12:45:03.043531468Z ^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043535868Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict 2024-09-12T12:45:03.043540468Z y[deepcopy(key, memo)] = deepcopy(value, memo) 2024-09-12T12:45:03.043545068Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043549568Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 172, in deepcopy 2024-09-12T12:45:03.043554168Z y = _reconstruct(x, memo, *rv) 2024-09-12T12:45:03.043558668Z ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043563168Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 271, in _reconstruct 2024-09-12T12:45:03.043567668Z state = deepcopy(state, memo) 2024-09-12T12:45:03.043572168Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043576668Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 146, in deepcopy 2024-09-12T12:45:03.043581368Z y = copier(x, memo) 2024-09-12T12:45:03.043585768Z ^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043590068Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 231, in _deepcopy_dict 2024-09-12T12:45:03.043594668Z y[deepcopy(key, memo)] = deepcopy(value, memo) 2024-09-12T12:45:03.043599168Z ^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:03.043603668Z File "/opt/python/3.11.8/lib/python3.11/copy.py", line 161, in deepcopy 2024-09-12T12:45:03.043608268Z rv = reductor(4) 2024-09-12T12:45:03.043612668Z ^^^^^^^^^^^ 2024-09-12T12:45:03.043616969Z TypeError: cannot pickle '_thread.RLock' object 2024-09-12T12:45:03.043621369Z [2024-09-12 12:45:03 +0000] [88] [INFO] Worker exiting (pid: 88) 2024-09-12T12:45:04.843246335Z [2024-09-12 12:45:04 +0000] [81] [ERROR] Worker (pid:92) exited with code 3 2024-09-12T12:45:04.843288836Z Traceback (most recent call last): 2024-09-12T12:45:04.843295436Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 209, in run 2024-09-12T12:45:04.843301136Z self.sleep() 2024-09-12T12:45:04.843306036Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 360, in sleep 2024-09-12T12:45:04.843311136Z ready = select.select([self.PIPE[0]], [], [], 1.0) 2024-09-12T12:45:04.843315936Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-12T12:45:04.843320836Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 242, in handle_chld 2024-09-12T12:45:04.843338936Z self.reap_workers() 2024-09-12T12:45:04.843343637Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 530, in reap_workers 2024-09-12T12:45:04.843348837Z raise HaltServer(reason, self.WORKER_BOOT_ERROR) 2024-09-12T12:45:04.843353437Z gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3> 2024-09-12T12:45:04.843358437Z 2024-09-12T12:45:04.843362737Z During handling of the above exception, another exception occurred: 2024-09-12T12:45:04.843367237Z 2024-09-12T12:45:04.843371637Z Traceback (most recent call last): 2024-09-12T12:45:04.843376037Z File "/opt/python/3.11.8/bin/gunicorn", line 8, in 2024-09-12T12:45:04.843380837Z sys.exit(run()) 2024-09-12T12:45:04.843385237Z ^^^^^ 2024-09-12T12:45:04.843389637Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 67, in run 2024-09-12T12:45:04.843394337Z WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]", prog=prog).run() 2024-09-12T12:45:04.843398837Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 236, in run 2024-09-12T12:45:04.843403437Z super().run() 2024-09-12T12:45:04.843407837Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/app/base.py", line 72, in run 2024-09-12T12:45:04.843412337Z Arbiter(self).run() 2024-09-12T12:45:04.843416737Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 229, in run 2024-09-12T12:45:04.843421237Z self.halt(reason=inst.reason, exit_status=inst.exit_status) 2024-09-12T12:45:04.843425837Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 342, in halt 2024-09-12T12:45:04.843430537Z self.stop() 2024-09-12T12:45:04.860256615Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 396, in stop 2024-09-12T12:45:04.860795520Z time.sleep(0.1) 2024-09-12T12:45:04.861067323Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 242, in handle_chld 2024-09-12T12:45:04.861422527Z self.reap_workers() 2024-09-12T12:45:04.861657630Z File "/opt/python/3.11.8/lib/python3.11/site-packages/gunicorn/arbiter.py", line 530, in reap_workers 2024-09-12T12:45:04.862053834Z raise HaltServer(reason, self.WORKER_BOOT_ERROR) 2024-09-12T12:45:04.862302136Z gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

Tasks

To be filled in by the engineer picking up the issue

adamdougal commented 1 month ago

Heya, apologies for this error. It should be fixed by this PR https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/1305

DSOTM-RSA commented 1 month ago

Amazing! I'll repull everything and try again.