Open geraldstanje1 opened 6 months ago
hi,
i tried to torch.compile the model using: model.model_body[0].auto_model = torch.compile(model.model_body[0].auto_model, mode="reduce-overhead")and get the following error:
model.model_body[0].auto_model = torch.compile(model.model_body[0].auto_model, mode="reduce-overhead")
error:
W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised: | 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised: W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
used image: 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference:2.1.0-transformers4.37.0-gpu-py310-cu1 torch compile: https://pytorch.org/docs/stable/generated/torch.compile.html
full error log:
2024-05-06T22:59:03.955-04:00 Collecting scikit-learn==1.2.2 (from -r /opt/ml/model/code/requirements.txt (line 1)) Downloading scikit_learn-1.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB) 2024-05-06T22:59:03.955-04:00 Collecting setfit==1.0.3 (from -r /opt/ml/model/code/requirements.txt (line 2)) Downloading setfit-1.0.3-py3-none-any.whl.metadata (11 kB) 2024-05-06T22:59:03.955-04:00 Collecting pandas==1.5.3 (from -r /opt/ml/model/code/requirements.txt (line 3)) Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: torch==2.1.0 in /opt/conda/lib/python3.10/site-packages (from -r /opt/ml/model/code/requirements.txt (line 4)) (2.1.0+cu118) 2024-05-06T22:59:03.955-04:00 Collecting optimum (from -r /opt/ml/model/code/requirements.txt (line 5)) Downloading optimum-1.19.1-py3-none-any.whl.metadata (19 kB) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: numpy>=1.17.3 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (1.22.4) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: scipy>=1.3.2 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (1.12.0) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: joblib>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (1.3.2) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (3.2.0) 2024-05-06T22:59:03.955-04:00 Collecting datasets>=2.3.0 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading datasets-2.19.1-py3-none-any.whl.metadata (19 kB) 2024-05-06T22:59:03.955-04:00 Collecting sentence-transformers>=2.2.1 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading sentence_transformers-2.7.0-py3-none-any.whl.metadata (11 kB) 2024-05-06T22:59:03.955-04:00 Collecting evaluate>=0.3.0 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading evaluate-0.4.2-py3-none-any.whl.metadata (9.3 kB) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: huggingface-hub>=0.13.0 in /opt/conda/lib/python3.10/site-packages (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (0.20.3) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: packaging in /opt/conda/lib/python3.10/site-packages (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (23.2) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/lib/python3.10/site-packages (from pandas==1.5.3->-r /opt/ml/model/code/requirements.txt (line 3)) (2.8.2) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas==1.5.3->-r /opt/ml/model/code/requirements.txt (line 3)) (2023.4) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (3.13.1) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (4.9.0) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (1.12) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (3.2.1) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (3.1.3) 2024-05-06T22:59:03.955-04:00 Requirement already satisfied: fsspec in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (2023.12.2) 2024-05-06T22:59:03.955-04:00 Collecting coloredlogs (from optimum->-r /opt/ml/model/code/requirements.txt (line 5)) Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB) 2024-05-06T22:59:04.348-04:00 Requirement already satisfied: transformers<4.41.0,>=4.26.0 in /opt/conda/lib/python3.10/site-packages (from transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (4.37.0) 2024-05-06T22:59:04.348-04:00 Collecting pyarrow>=12.0.0 (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading pyarrow-16.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.0 kB) 2024-05-06T22:59:04.348-04:00 Collecting pyarrow-hotfix (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB) 2024-05-06T22:59:04.348-04:00 Collecting dill<0.3.9,>=0.3.0 (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB) 2024-05-06T22:59:04.348-04:00 Requirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.10/site-packages (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (2.31.0) 2024-05-06T22:59:04.348-04:00 Requirement already satisfied: tqdm>=4.62.1 in /opt/conda/lib/python3.10/site-packages (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (4.64.1) 2024-05-06T22:59:04.599-04:00 Collecting xxhash (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB) 2024-05-06T22:59:04.849-04:00 Collecting multiprocess (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB) 2024-05-06T22:59:04.849-04:00 Collecting aiohttp (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.5 kB) 2024-05-06T22:59:04.849-04:00 Collecting huggingface-hub>=0.13.0 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading huggingface_hub-0.23.0-py3-none-any.whl.metadata (12 kB) 2024-05-06T22:59:05.100-04:00 Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (6.0.1) 2024-05-06T22:59:05.100-04:00 Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.1->pandas==1.5.3->-r /opt/ml/model/code/requirements.txt (line 3)) (1.16.0) 2024-05-06T22:59:05.601-04:00 Requirement already satisfied: Pillow in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.1->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (10.2.0) 2024-05-06T22:59:05.601-04:00 Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.10/site-packages (from transformers<4.41.0,>=4.26.0->transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (2023.12.25) 2024-05-06T22:59:05.601-04:00 Requirement already satisfied: tokenizers<0.19,>=0.14 in /opt/conda/lib/python3.10/site-packages (from transformers<4.41.0,>=4.26.0->transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (0.15.1) 2024-05-06T22:59:05.601-04:00 Requirement already satisfied: safetensors>=0.3.1 in /opt/conda/lib/python3.10/site-packages (from transformers<4.41.0,>=4.26.0->transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (0.4.2) 2024-05-06T22:59:05.601-04:00 Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /opt/conda/lib/python3.10/site-packages (from transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (0.1.99) 2024-05-06T22:59:05.601-04:00 Requirement already satisfied: protobuf in /opt/conda/lib/python3.10/site-packages (from transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (3.20.2) 2024-05-06T22:59:05.601-04:00 Collecting humanfriendly>=9.1 (from coloredlogs->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB) 2024-05-06T22:59:05.601-04:00 Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2->torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (2.1.4) 2024-05-06T22:59:05.601-04:00 Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (1.3.0) 2024-05-06T22:59:05.601-04:00 Collecting aiosignal>=1.1.2 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB) 2024-05-06T22:59:05.852-04:00 Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (23.2.0) 2024-05-06T22:59:06.102-04:00 Collecting frozenlist>=1.1.1 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB) 2024-05-06T22:59:06.102-04:00 Collecting multidict<7.0,>=4.5 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB) 2024-05-06T22:59:06.352-04:00 Collecting yarl<2.0,>=1.0 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB) 2024-05-06T22:59:06.352-04:00 Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB) 2024-05-06T22:59:06.352-04:00 Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (3.3.2) 2024-05-06T22:59:06.352-04:00 Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (3.4) 2024-05-06T22:59:06.352-04:00 Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (1.26.18) 2024-05-06T22:59:06.607-04:00 Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (2023.11.17) 2024-05-06T22:59:06.608-04:00 Downloading scikit_learn-1.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.6/9.6 MB 87.0 MB/s eta 0:00:00 2024-05-06T22:59:06.609-04:00 Downloading setfit-1.0.3-py3-none-any.whl (75 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.9/75.9 kB 38.4 MB/s eta 0:00:00 2024-05-06T22:59:06.855-04:00 Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.1/12.1 MB 130.5 MB/s eta 0:00:00 2024-05-06T22:59:06.855-04:00 Downloading optimum-1.19.1-py3-none-any.whl (417 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 417.0/417.0 kB 122.1 MB/s eta 0:00:00 2024-05-06T22:59:06.855-04:00 Downloading datasets-2.19.1-py3-none-any.whl (542 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 542.0/542.0 kB 138.6 MB/s eta 0:00:00 2024-05-06T22:59:06.855-04:00 Downloading evaluate-0.4.2-py3-none-any.whl (84 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.1/84.1 kB 43.6 MB/s eta 0:00:00 2024-05-06T22:59:06.855-04:00 Downloading huggingface_hub-0.23.0-py3-none-any.whl (401 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 401.2/401.2 kB 120.6 MB/s eta 0:00:00 2024-05-06T22:59:06.855-04:00 Downloading sentence_transformers-2.7.0-py3-none-any.whl (171 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 171.5/171.5 kB 78.0 MB/s eta 0:00:00 2024-05-06T22:59:06.855-04:00 Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 kB 24.8 MB/s eta 0:00:00 2024-05-06T22:59:06.855-04:00 Downloading dill-0.3.8-py3-none-any.whl (116 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 59.0 MB/s eta 0:00:00 2024-05-06T22:59:06.855-04:00 Downloading aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 155.8 MB/s eta 0:00:00 2024-05-06T22:59:06.856-04:00 Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 45.1 MB/s eta 0:00:00 2024-05-06T22:59:07.130-04:00 Downloading pyarrow-16.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (40.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 MB 88.9 MB/s eta 0:00:00 2024-05-06T22:59:07.133-04:00 Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 60.0 MB/s eta 0:00:00 2024-05-06T22:59:07.363-04:00 Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB) 2024-05-06T22:59:07.364-04:00 Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 82.8 MB/s eta 0:00:00 2024-05-06T22:59:07.364-04:00 Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB) 2024-05-06T22:59:07.364-04:00 Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB) 2024-05-06T22:59:07.364-04:00 Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.5/239.5 kB 71.2 MB/s eta 0:00:00 2024-05-06T22:59:07.364-04:00 Downloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.3/124.3 kB 49.2 MB/s eta 0:00:00 2024-05-06T22:59:08.868-04:00 Downloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 301.6/301.6 kB 103.5 MB/s eta 0:00:00 2024-05-06T22:59:12.959-04:00 Installing collected packages: xxhash, pyarrow-hotfix, pyarrow, multidict, humanfriendly, frozenlist, dill, async-timeout, yarl, scikit-learn, pandas, multiprocess, huggingface-hub, coloredlogs, aiosignal, aiohttp, sentence-transformers, datasets, optimum, evaluate, setfit Attempting uninstall: scikit-learn Found existing installation: scikit-learn 1.4.0 Uninstalling scikit-learn-1.4.0: Successfully uninstalled scikit-learn-1.4.0 Attempting uninstall: pandas Found existing installation: pandas 2.2.0 Uninstalling pandas-2.2.0: Successfully uninstalled pandas-2.2.0 2024-05-06T22:59:17.155-04:00 Attempting uninstall: huggingface-hub Found existing installation: huggingface-hub 0.20.3 Uninstalling huggingface-hub-0.20.3: Successfully uninstalled huggingface-hub-0.20.3 2024-05-06T22:59:17.155-04:00 Successfully installed aiohttp-3.9.5 aiosignal-1.3.1 async-timeout-4.0.3 coloredlogs-15.0.1 datasets-2.19.1 dill-0.3.8 evaluate-0.4.2 frozenlist-1.4.1 huggingface-hub-0.23.0 humanfriendly-10.0 multidict-6.0.5 multiprocess-0.70.16 optimum-1.19.1 pandas-1.5.3 pyarrow-16.0.0 pyarrow-hotfix-0.6 scikit-learn-1.2.2 sentence-transformers-2.7.0 setfit-1.0.3 xxhash-3.4.1 yarl-1.9.4 2024-05-06T22:59:17.155-04:00 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv 2024-05-06T22:59:17.155-04:00 [notice] A new release of pip is available: 23.3.2 -> 24.0 2024-05-06T22:59:17.406-04:00 [notice] To update, run: pip install --upgrade pip 2024-05-06T22:59:17.656-04:00 Warning: MMS is using non-default JVM parameters: -XX:-UseContainerSupport 2024-05-06T22:59:18.157-04:00 WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2024-05-06T22:59:18.157-04:00 2024-05-07T02:59:18,138 [INFO ] main com.amazonaws.ml.mms.ModelServer - 2024-05-06T22:59:18.157-04:00 MMS Home: /opt/conda/lib/python3.10/site-packages 2024-05-06T22:59:18.157-04:00 Current directory: / 2024-05-06T22:59:18.157-04:00 Temp directory: /home/model-server/tmp 2024-05-06T22:59:18.157-04:00 Number of GPUs: 1 2024-05-06T22:59:18.157-04:00 Number of CPUs: 4 2024-05-06T22:59:18.157-04:00 Max heap size: 3934 M 2024-05-06T22:59:18.157-04:00 Python executable: /opt/conda/bin/python3.10 2024-05-06T22:59:18.157-04:00 Config file: /etc/sagemaker-mms.properties 2024-05-06T22:59:18.157-04:00 Inference address: http://0.0.0.0:8080 2024-05-06T22:59:18.157-04:00 Management address: http://0.0.0.0:8080 2024-05-06T22:59:18.157-04:00 Model Store: / 2024-05-06T22:59:18.157-04:00 Initial Models: model=/opt/ml/model 2024-05-06T22:59:18.157-04:00 Log dir: null 2024-05-06T22:59:18.157-04:00 Metrics dir: null 2024-05-06T22:59:18.157-04:00 Netty threads: 0 2024-05-06T22:59:18.157-04:00 Netty client threads: 0 2024-05-06T22:59:18.157-04:00 Default workers per model: 1 2024-05-06T22:59:18.157-04:00 Blacklist Regex: N/A 2024-05-06T22:59:18.157-04:00 Maximum Response Size: 6553500 2024-05-06T22:59:18.157-04:00 Maximum Request Size: 6553500 2024-05-06T22:59:18.157-04:00 Preload model: false 2024-05-06T22:59:18.157-04:00 Prefer direct buffer: false 2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,145 [INFO ] main com.amazonaws.ml.mms.ModelServer - Loading initial models: /opt/ml/model preload_model: false 2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,190 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-9000-model 2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,261 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model_service_worker started with args: --sock-type unix --sock-name /home/model-server/tmp/.mms.sock.9000 --handler sagemaker_huggingface_inference_toolkit.handler_service --model-path /opt/ml/model --model-name model --preload-model false --tmp-dir /home/model-server/tmp 2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,262 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Listening on port: /home/model-server/tmp/.mms.sock.9000 2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,263 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [PID] 67 2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,263 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - MMS worker started. 2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,263 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Python runtime: 3.10.9 2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,263 [INFO ] main com.amazonaws.ml.mms.wlm.ModelManager - Model model loaded. 2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,267 [INFO ] main com.amazonaws.ml.mms.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,273 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.mms.sock.9000 2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,324 [INFO ] main com.amazonaws.ml.mms.ModelServer - Inference API bind to: http://0.0.0.0:8080 2024-05-06T22:59:18.409-04:00 Model server started. 2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,328 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Connection accepted: /home/model-server/tmp/.mms.sock.9000. 2024-05-06T22:59:21.932-04:00 2024-05-07T02:59:18,329 [WARN ] pool-3-thread-1 com.amazonaws.ml.mms.metrics.MetricCollector - worker pid is not available yet. 2024-05-06T22:59:22.179-04:00 2024-05-07T02:59:21,694 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - PyTorch version 2.1.0+cu118 available. 2024-05-06T22:59:22.179-04:00 2024-05-07T02:59:21,955 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Load pretrained SentenceTransformer: /opt/ml/model 2024-05-06T22:59:23.697-04:00 2024-05-07T02:59:22,156 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Use pytorch device_name: cuda 2024-05-06T22:59:23.947-04:00 2024-05-07T02:59:23,557 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 18 2024-05-06T22:59:23.947-04:00 2024-05-07T02:59:23,818 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - /opt/conda/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator LogisticRegression from version 1.3.2 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to: 2024-05-06T22:59:23.947-04:00 2024-05-07T02:59:23,819 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations 2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,819 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - warnings.warn( 2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,953 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - device=cuda 2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,954 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model loaded!!! 2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,955 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Current device - cuda 2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,976 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch.compile for model SetFitModel(model_body=SentenceTransformer( 2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,977 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,978 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) 2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,978 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - (2): Normalize() 2024-05-06T22:59:24.458-04:00 2024-05-07T02:59:23,980 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ), model_head=LogisticRegression(), multi_target_strategy=None, normalize_embeddings=False, labels=['DOCUMENT', 'POLICY'], model_card_data=SetFitModelCardData(language=None, license=None, tags=['setfit', 'sentence-transformers', 'text-classification', 'generated_from_setfit_trainer'], model_name='SetFit with sentence-transformers/all-MiniLM-L6-v2', model_id=None, dataset_name=None, dataset_id=None, dataset_revision=None, task_name=None, st_id='sentence-transformers/all-MiniLM-L6-v2', hyperparameters={}, eval_results_dict={}, eval_lines_list=[], metric_lines=[], widget=[], predict_example=None, label_example_list=[], tokenizer_warning=False, train_set_metrics_list=[], train_set_sentences_per_label_list=[], code_carbon_callback=None, num_classes=None, best_model_step=None, metrics=['accuracy'], pipeline_tag='text-classification', library_name='setfit', version={'python': '3.10.9', 'setfit': '1.0.3', 'sentence_transformers': '2.7.0', 'transformers': '4.37.0', 'torch': '2.1.0+cu118', 'datasets': '2.19.1', 'tokenizers': '0.15.1'})) 2024-05-06T22:59:24.460-04:00 2024-05-07T02:59:24,347 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model compiled successfully... 2024-05-06T22:59:24.460-04:00 2024-05-07T02:59:24,348 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Model model loaded io_fd=de239efffefa3af1-00000029-00000000-1db2d9548cfdc087-9fd56c90 2024-05-06T22:59:24.460-04:00 2024-05-07T02:59:24,349 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 5989 2024-05-06T22:59:28.210-04:00 2024-05-07T02:59:24,350 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-model-1 2024-05-06T22:59:32.953-04:00 2024-05-07T02:59:28,186 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 1 2024-05-06T22:59:38.015-04:00 2024-05-07T02:59:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0 2024-05-06T22:59:42.953-04:00 2024-05-07T02:59:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0 2024-05-06T22:59:47.953-04:00 2024-05-07T02:59:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0 2024-05-06T22:59:52.953-04:00 2024-05-07T02:59:48,173 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 1 2024-05-06T22:59:57.953-04:00 2024-05-07T02:59:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:02.992-04:00 2024-05-07T02:59:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:07.952-04:00 2024-05-07T03:00:03,176 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:12.953-04:00 2024-05-07T03:00:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:13.092-04:00 2024-05-07T03:00:13,009 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input_data: {"inputs": ["How to create policy?"]}, content_type: application/json 2024-05-06T23:00:13.092-04:00 2024-05-07T03:00:13,012 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - decoded input: {'inputs': ['How to create policy?']}, content_type: application/json 2024-05-06T23:00:13.092-04:00 2024-05-07T03:00:13,013 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - data: {'inputs': ['How to create policy?']}, data_type: <class 'dict'> 2024-05-06T23:00:13.342-04:00 2024-05-07T03:00:13,020 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:17.953-04:00 2024-05-07T03:00:13,178 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:22.953-04:00 2024-05-07T03:00:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:25.975-04:00 2024-05-07T03:00:23,176 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,734 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:00<?, ?it/s] 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,734 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - predict_fn error: backend='inductor' raised: 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,735 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,735 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:12<?, ?it/s] 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,735 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,736 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,736 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,737 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,737 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting: 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,737 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,738 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,738 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,739 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,739 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,739 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 258, in handle 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,740 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 12726 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,740 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - response = self.transform_fn(*([self.model, input_data, content_type, accept] + self.transform_extra_arg)) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,740 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 214, in transform_fn 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - predictions = self.predict(*([processed_data, model] + self.predict_extra_arg)) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/ml/model/code/inference.py", line 111, in predict_fn 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise e 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/ml/model/code/inference.py", line 108, in predict_fn 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-9000-model ACCESS_LOG - /169.254.178.2:42770 "POST /invocations HTTP/1.1" 400 12737 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,742 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return model.predict(inputs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,742 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/setfit/modeling.py", line 560, in predict 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,742 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - embeddings = self.encode(inputs, batch_size=batch_size, show_progress_bar=show_progress_bar) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,743 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/setfit/modeling.py", line 452, in encode 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,743 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.model_body.encode( 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,743 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 371, in encode 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,744 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - out_features = self.forward(features) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,744 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,744 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input = module(input) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,744 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,745 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,745 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,745 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,746 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 98, in forward 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,746 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - output_states = self.auto_model(**trans_features, return_dict=False) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,746 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,747 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,747 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,747 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,747 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,748 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,748 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,749 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,749 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,749 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,750 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 490, in catch_errors 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,750 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return callback(frame, cache_entry, hooks, frame_state) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,750 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 641, in _convert_frame 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,751 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - result = inner_convert(frame, cache_size, hooks, frame_state) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,751 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 133, in _fn 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,751 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,752 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 389, in _convert_frame_assert 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,752 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return _compile( 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,752 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 569, in _compile 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,752 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,753 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,753 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,753 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 491, in compile_inner 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,753 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - out_code = transform_code_object(code, transform) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,754 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,754 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - transformations(instructions, code_options) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,754 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 458, in transform 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,755 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - tracer.run() 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,755 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2074, in run 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,755 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - super().run() 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,755 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,756 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - and self.step() 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,756 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,756 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - getattr(self, inst.opname)(inst) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,756 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2162, in RETURN_VALUE 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,757 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.output.compile_subgraph( 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,757 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 857, in compile_subgraph 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,757 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,757 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 957, in compile_and_call_fx_graph 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = self.call_user_compiler(gm) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,759 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1024, in call_user_compiler 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,759 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,759 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1009, in call_user_compiler 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,759 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = compiler_fn(gm, self.example_inputs()) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_gm = compiler_fn(gm, example_inputs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/__init__.py", line 1568, in __call__ 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compile_fx(model_, inputs_, config_patches=self.config) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 961, in compile_fx 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compile_fx( 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1150, in compile_fx 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return aot_autograd( 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - cg = aot_module_simplified(gm, example_inputs, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,762 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,762 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = create_aot_dispatcher_function( 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,762 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,763 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,763 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,763 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,763 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fw = compiler(fw_module, flat_args) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,765 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,765 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,765 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1092, in fw_compiler_base 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,765 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return inner_compile( 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - inner_compiled_fn = compiler_fn(gm, example_inputs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 54, in newFunction 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return old_func(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,768 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 341, in compile_fx_inner 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,768 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_graph: CompiledFxGraph = fx_codegen_and_compile( 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,768 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 565, in fx_codegen_and_compile 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,768 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = graph.compile_to_fn() 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/graph.py", line 970, in compile_to_fn 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.compile_to_module().call 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/graph.py", line 941, in compile_to_module 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1139, in load_by_key_path 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - exec(code, mod.__dict__, mod.__dict__) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/home/model-server/tmp/torchinductor_root/av/cavozcyohmg3erjtmk3oyovim5verzsympqf2nkp4bzglt2x6rwi.py", line 424, in <module> 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - async_compile.wait(globals()) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1418, in wait 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - scope[key] = result.result() 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1278, in result 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel = self.kernel = _load_kernel(self.kernel_name, self.source_code) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1261, in _load_kernel 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel.precompile() 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 168, in precompile 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.launchers = [ 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 169, in <listcomp> 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self._precompile_config(c, warm_cache_only_with_cc) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 205, in _precompile_config 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - binary._init_handles() 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/triton/compiler/compiler.py", line 683, in _init_handles 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting: 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred: 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/mms/service.py", line 108, in predict 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self._entry_point(input_batch, self.context) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 267, in handle 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400) 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised: 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting: 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo 2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True 2024-05-06T23:00:28.232-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - : 400 2024-05-06T23:00:32.953-04:00 2024-05-07T03:00:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:37.953-04:00 2024-05-07T03:00:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:42.953-04:00 2024-05-07T03:00:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:47.953-04:00 2024-05-07T03:00:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:53.000-04:00 2024-05-07T03:00:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:00:57.953-04:00 2024-05-07T03:00:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:02.953-04:00 2024-05-07T03:00:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:07.986-04:00 2024-05-07T03:01:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:12.953-04:00 2024-05-07T03:01:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:17.953-04:00 2024-05-07T03:01:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:22.952-04:00 2024-05-07T03:01:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:27.952-04:00 2024-05-07T03:01:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:32.987-04:00 2024-05-07T03:01:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:37.953-04:00 2024-05-07T03:01:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:42.953-04:00 2024-05-07T03:01:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:47.990-04:00 2024-05-07T03:01:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:52.953-04:00 2024-05-07T03:01:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:01:57.952-04:00 2024-05-07T03:01:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:02.985-04:00 2024-05-07T03:01:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:07.953-04:00 2024-05-07T03:02:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:12.953-04:00 2024-05-07T03:02:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:17.953-04:00 2024-05-07T03:02:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:22.953-04:00 2024-05-07T03:02:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:27.990-04:00 2024-05-07T03:02:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:32.953-04:00 2024-05-07T03:02:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:37.953-04:00 2024-05-07T03:02:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:42.990-04:00 2024-05-07T03:02:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:47.953-04:00 2024-05-07T03:02:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:02:52.953-04:00 2024-05-07T03:02:48,176 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:02:57.952-04:00 2024-05-07T03:02:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:02.953-04:00 2024-05-07T03:02:58,173 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:07.953-04:00 2024-05-07T03:03:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:12.952-04:00 2024-05-07T03:03:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:17.952-04:00 2024-05-07T03:03:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:22.953-04:00 2024-05-07T03:03:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:27.990-04:00 2024-05-07T03:03:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:32.953-04:00 2024-05-07T03:03:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:37.952-04:00 2024-05-07T03:03:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:42.989-04:00 2024-05-07T03:03:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:47.952-04:00 2024-05-07T03:03:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:52.952-04:00 2024-05-07T03:03:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:03:57.979-04:00 2024-05-07T03:03:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:02.953-04:00 2024-05-07T03:03:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:07.953-04:00 2024-05-07T03:04:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:12.953-04:00 2024-05-07T03:04:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:17.953-04:00 2024-05-07T03:04:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:22.990-04:00 2024-05-07T03:04:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:27.952-04:00 2024-05-07T03:04:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:32.953-04:00 2024-05-07T03:04:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:37.953-04:00 2024-05-07T03:04:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:42.952-04:00 2024-05-07T03:04:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:47.952-04:00 2024-05-07T03:04:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:52.952-04:00 2024-05-07T03:04:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:04:57.988-04:00 2024-05-07T03:04:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:02.952-04:00 2024-05-07T03:04:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:07.953-04:00 2024-05-07T03:05:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:13.052-04:00 2024-05-07T03:05:08,174 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:17.952-04:00 2024-05-07T03:05:13,174 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:05:22.952-04:00 2024-05-07T03:05:18,173 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:27.985-04:00 2024-05-07T03:05:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:05:32.953-04:00 2024-05-07T03:05:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:37.953-04:00 2024-05-07T03:05:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:42.953-04:00 2024-05-07T03:05:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:47.953-04:00 2024-05-07T03:05:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:52.953-04:00 2024-05-07T03:05:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:05:57.953-04:00 2024-05-07T03:05:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:02.952-04:00 2024-05-07T03:05:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:07.952-04:00 2024-05-07T03:06:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:12.984-04:00 2024-05-07T03:06:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:17.952-04:00 2024-05-07T03:06:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:22.952-04:00 2024-05-07T03:06:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:27.989-04:00 2024-05-07T03:06:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:32.953-04:00 2024-05-07T03:06:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:37.953-04:00 2024-05-07T03:06:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:06:42.953-04:00 2024-05-07T03:06:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:06:47.953-04:00 2024-05-07T03:06:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:52.991-04:00 2024-05-07T03:06:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:06:57.953-04:00 2024-05-07T03:06:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:07:02.953-04:00 2024-05-07T03:06:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:07:07.953-04:00 2024-05-07T03:07:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:07:12.952-04:00 2024-05-07T03:07:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:07:17.952-04:00 2024-05-07T03:07:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:07:22.953-04:00 2024-05-07T03:07:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:07:27.953-04:00 2024-05-07T03:07:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:07:32.953-04:00 2024-05-07T03:07:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:07:38.016-04:00 2024-05-07T03:07:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:07:42.953-04:00 2024-05-07T03:07:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:07:47.952-04:00 2024-05-07T03:07:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:07:52.994-04:00 2024-05-07T03:07:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:07:57.953-04:00 2024-05-07T03:07:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:08:02.953-04:00 2024-05-07T03:07:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:08:07.987-04:00 2024-05-07T03:08:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:08:12.953-04:00 2024-05-07T03:08:08,175 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:08:17.953-04:00 2024-05-07T03:08:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:08:22.953-04:00 2024-05-07T03:08:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:08:27.953-04:00 2024-05-07T03:08:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:08:32.987-04:00 2024-05-07T03:08:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:08:37.953-04:00 2024-05-07T03:08:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:08:42.953-04:00 2024-05-07T03:08:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:08:47.952-04:00 2024-05-07T03:08:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:08:52.953-04:00 2024-05-07T03:08:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:08:57.953-04:00 2024-05-07T03:08:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:09:02.953-04:00 2024-05-07T03:08:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:09:07.953-04:00 2024-05-07T03:09:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:09:12.953-04:00 2024-05-07T03:09:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:09:17.953-04:00 2024-05-07T03:09:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:09:22.952-04:00 2024-05-07T03:09:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:09:27.986-04:00 2024-05-07T03:09:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:09:32.953-04:00 2024-05-07T03:09:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:09:37.953-04:00 2024-05-07T03:09:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:09:42.988-04:00 2024-05-07T03:09:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:09:47.952-04:00 2024-05-07T03:09:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:09:52.952-04:00 2024-05-07T03:09:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:09:57.952-04:00 2024-05-07T03:09:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:10:02.953-04:00 2024-05-07T03:09:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:10:07.953-04:00 2024-05-07T03:10:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:10:12.953-04:00 2024-05-07T03:10:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:10:17.953-04:00 2024-05-07T03:10:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:10:22.953-04:00 2024-05-07T03:10:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:10:27.953-04:00 2024-05-07T03:10:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:10:32.953-04:00 2024-05-07T03:10:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:10:37.984-04:00 2024-05-07T03:10:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:10:42.952-04:00 2024-05-07T03:10:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:10:47.953-04:00 2024-05-07T03:10:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:10:52.952-04:00 2024-05-07T03:10:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:10:57.953-04:00 2024-05-07T03:10:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:02.993-04:00 2024-05-07T03:10:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:07.952-04:00 2024-05-07T03:11:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:12.953-04:00 2024-05-07T03:11:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:17.953-04:00 2024-05-07T03:11:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:22.953-04:00 2024-05-07T03:11:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:27.953-04:00 2024-05-07T03:11:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:32.953-04:00 2024-05-07T03:11:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:38.019-04:00 2024-05-07T03:11:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:11:42.952-04:00 2024-05-07T03:11:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:47.953-04:00 2024-05-07T03:11:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:11:52.983-04:00 2024-05-07T03:11:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:11:57.953-04:00 2024-05-07T03:11:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:02.952-04:00 2024-05-07T03:11:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:07.987-04:00 2024-05-07T03:12:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:12.953-04:00 2024-05-07T03:12:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:17.952-04:00 2024-05-07T03:12:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:22.952-04:00 2024-05-07T03:12:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:27.953-04:00 2024-05-07T03:12:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:32.988-04:00 2024-05-07T03:12:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:37.953-04:00 2024-05-07T03:12:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:42.953-04:00 2024-05-07T03:12:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:47.953-04:00 2024-05-07T03:12:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:52.953-04:00 2024-05-07T03:12:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:12:57.988-04:00 2024-05-07T03:12:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:02.953-04:00 2024-05-07T03:12:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:07.953-04:00 2024-05-07T03:13:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:12.988-04:00 2024-05-07T03:13:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:17.953-04:00 2024-05-07T03:13:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:22.952-04:00 2024-05-07T03:13:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:27.952-04:00 2024-05-07T03:13:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:32.952-04:00 2024-05-07T03:13:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:13:37.988-04:00 2024-05-07T03:13:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:13:42.952-04:00 2024-05-07T03:13:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:47.953-04:00 2024-05-07T03:13:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:52.991-04:00 2024-05-07T03:13:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:13:57.953-04:00 2024-05-07T03:13:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:02.953-04:00 2024-05-07T03:13:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:07.953-04:00 2024-05-07T03:14:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:12.953-04:00 2024-05-07T03:14:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:17.953-04:00 2024-05-07T03:14:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:22.953-04:00 2024-05-07T03:14:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:27.983-04:00 2024-05-07T03:14:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:32.953-04:00 2024-05-07T03:14:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:37.953-04:00 2024-05-07T03:14:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:42.952-04:00 2024-05-07T03:14:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:47.952-04:00 2024-05-07T03:14:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:52.989-04:00 2024-05-07T03:14:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:14:57.953-04:00 2024-05-07T03:14:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:15:02.952-04:00 2024-05-07T03:14:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1 2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:06,343 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input_data: {"inputs": ["How to create policy?"]}, content_type: application/json 2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:06,343 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - decoded input: {'inputs': ['How to create policy?']}, content_type: application/json 2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:06,344 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - data: {'inputs': ['How to create policy?']}, data_type: <class 'dict'> 2024-05-06T23:15:08.404-04:00 2024-05-07T03:15:06,346 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:08,177 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:48522 "GET /ping HTTP/1.1" 200 0 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,253 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:00<?, ?it/s] 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,253 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - predict_fn error: backend='inductor' raised: 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:05<?, ?it/s] 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting: 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 258, in handle 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - response = self.transform_fn(*([self.model, input_data, content_type, accept] + self.transform_extra_arg)) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 214, in transform_fn 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - predictions = self.predict(*([processed_data, model] + self.predict_extra_arg)) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/ml/model/code/inference.py", line 111, in predict_fn 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise e 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/ml/model/code/inference.py", line 108, in predict_fn 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return model.predict(inputs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/setfit/modeling.py", line 560, in predict 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - embeddings = self.encode(inputs, batch_size=batch_size, show_progress_bar=show_progress_bar) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/setfit/modeling.py", line 452, in encode 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.model_body.encode( 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 371, in encode 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - out_features = self.forward(features) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input = module(input) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 5914 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,257 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,257 [INFO ] W-9000-model ACCESS_LOG - /169.254.178.2:58184 "POST /invocations HTTP/1.1" 400 5915 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,257 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 98, in forward 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - output_states = self.auto_model(**trans_features, return_dict=False) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 490, in catch_errors 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return callback(frame, cache_entry, hooks, frame_state) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 641, in _convert_frame 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - result = inner_convert(frame, cache_size, hooks, frame_state) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 133, in _fn 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 389, in _convert_frame_assert 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return _compile( 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 569, in _compile 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 491, in compile_inner 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - out_code = transform_code_object(code, transform) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - transformations(instructions, code_options) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 458, in transform 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - tracer.run() 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2074, in run 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - super().run() 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - and self.step() 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - getattr(self, inst.opname)(inst) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2162, in RETURN_VALUE 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.output.compile_subgraph( 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 857, in compile_subgraph 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 957, in compile_and_call_fx_graph 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = self.call_user_compiler(gm) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1024, in call_user_compiler 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1009, in call_user_compiler 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = compiler_fn(gm, self.example_inputs()) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_gm = compiler_fn(gm, example_inputs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/__init__.py", line 1568, in __call__ 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compile_fx(model_, inputs_, config_patches=self.config) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 961, in compile_fx 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compile_fx( 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1150, in compile_fx 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return aot_autograd( 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - cg = aot_module_simplified(gm, example_inputs, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = create_aot_dispatcher_function( 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fw = compiler(fw_module, flat_args) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1092, in fw_compiler_base 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return inner_compile( 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - inner_compiled_fn = compiler_fn(gm, example_inputs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 54, in newFunction 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return old_func(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 341, in compile_fx_inner 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_graph: CompiledFxGraph = fx_codegen_and_compile( 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 565, in fx_codegen_and_compile 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = graph.compile_to_fn() 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/graph.py", line 970, in compile_to_fn 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.compile_to_module().call 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/graph.py", line 941, in compile_to_module 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1139, in load_by_key_path 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - exec(code, mod.__dict__, mod.__dict__) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/home/model-server/tmp/torchinductor_root/av/cavozcyohmg3erjtmk3oyovim5verzsympqf2nkp4bzglt2x6rwi.py", line 424, in <module> 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - async_compile.wait(globals()) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1418, in wait 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - scope[key] = result.result() 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1278, in result 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel = self.kernel = _load_kernel(self.kernel_name, self.source_code) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1261, in _load_kernel 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel.precompile() 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 168, in precompile 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.launchers = [ 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 169, in <listcomp> 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self._precompile_config(c, warm_cache_only_with_cc) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 205, in _precompile_config 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - binary._init_handles() 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/triton/compiler/compiler.py", line 683, in _init_handles 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting: 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred: 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/mms/service.py", line 108, in predict 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self._entry_point(input_batch, self.context) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 267, in handle 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400) 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised: 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting: 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo 2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True 2024-05-06T23:15:13.420-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - : 400
any idea how this can be fixed? cc @chen3933
hi,
i tried to torch.compile the model using:
model.model_body[0].auto_model = torch.compile(model.model_body[0].auto_model, mode="reduce-overhead")and get the following error:error:
used image: 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference:2.1.0-transformers4.37.0-gpu-py310-cu1 torch compile: https://pytorch.org/docs/stable/generated/torch.compile.html
full error log:
any idea how this can be fixed? cc @chen3933