feat: making torchserve usable without torch

[racinmat]

Description

Please read our CONTRIBUTING.md prior to creating your first pull request.

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #3350

Type of change

Please delete options that are not relevant.

• [x] New feature (non-breaking change which adds functionality)
• [x] This change requires a documentation update

This feature request is basically just a refactoring.

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced. Please also list any relevant details for your test configuration.

I ran python torchserve_sanity.py but it got stuck at 82% for some reason. Logs are below.

Checklist:

• [x] Did you have fun? Partly, it was a bit tedious to make the suite run.
• [x] Have you added tests that prove your fix is effective or that this feature works? I refactored the tests, but did not add new ones.
• [x] Has code been commented, particularly in hard-to-understand areas?
• [x] Have you made corresponding changes to the documentation?

[racinmat]

I tried running the tests, but they got stuck:

    2024-10-21T15:55:32,097 [WARN ] W-9026-mnist_traced_
1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: mnist_traced, error: Worker died.
    2024-10-21T15:55:32,098 [DEBUG] W-9026-mnist_traced_
1.0 org.pytorch.serve.wlm.ModelVersionedRefs - Removed model: mnist_traced version: 1.0
    2024-10-21T15:55:32,098 [DEBUG] W-9026-mnist_traced_
1.0 org.pytorch.serve.wlm.WorkerThread - W-9026-mnist_tr
aced_1.0 State change WORKER_STARTED -> WORKER_SCALED_DOWN
    2024-10-21T15:55:32,099 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG - Traceback (most recent call last):
    2024-10-21T15:55:32,100 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/mnt/c/Projects/others/pytorch_serve/ts/model_loader.py", line 108, in load      
    2024-10-21T15:55:32,100 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -     module, function_name = self._load_handler_file(handler)
    2024-10-21T15:55:32,101 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    2024-10-21T15:55:32,101 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/mnt/c/Projects/others/py
torch_serve/ts/model_loader.py", line 153, in _load_handler_file
    2024-10-21T15:55:32,102 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -     module = importlib.import_module(module_name)
    2024-10-21T15:55:32,102 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    2024-10-21T15:55:32,103 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/home/racinsky/miniforge3
/lib/python3.12/importlib/__init__.py", line 90, in import_module
    2024-10-21T15:55:32,103 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -     return _bootstrap._gcd_import(name[level:], package, level)
    2024-10-21T15:55:32,104 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    2024-10-21T15:55:32,104 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    2024-10-21T15:55:32,105 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    2024-10-21T15:55:32,105 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
    2024-10-21T15:55:32,105 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
    2024-10-21T15:55:32,107 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
    2024-10-21T15:55:32,107 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    2024-10-21T15:55:32,108 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/tmp/models/8a6821d9383c4
d44b2e6de0e24e1efc5/mnist_handler.py", line 8, in <module>
    2024-10-21T15:55:32,108 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -     from torchvision import transforms
    2024-10-21T15:55:32,108 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG - ModuleNotFoundError: No module named 'torchvision'
    2024-10-21T15:55:32,109 [INFO ] W-9026-mnist_traced_1.0-stdout MODEL_LOG -
    2024-10-21T15:55:32,109 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG - During handling of the above exception, another exception occurred:
    2024-10-21T15:55:32,109 [INFO ] W-9026-mnist_traced_1.0-stdout MODEL_LOG -
    2024-10-21T15:55:32,110 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG - Traceback (most recent call last):
    2024-10-21T15:55:32,110 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/mnt/c/Projects/others/py
torch_serve/ts/model_service_worker.py", line 302, in <module>
    2024-10-21T15:55:32,110 [INFO ] W-9026-mnist_traced_1.0-stdout MODEL_LOG -     worker.run_server()
    2024-10-21T15:55:32,110 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/mnt/c/Projects/others/py
torch_serve/ts/model_service_worker.py", line 269, in run_server
    2024-10-21T15:55:32,110 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -     self.handle_connection(cl_socket)
    2024-10-21T15:55:32,111 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/mnt/c/Projects/others/py
torch_serve/ts/model_service_worker.py", line 197, in handle_connection
    2024-10-21T15:55:32,111 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -     service, result, code = self.load_model(msg)
    2024-10-21T15:55:32,112 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -                             ^^^^^^^^^^^^^^^^^^^^
    2024-10-21T15:55:32,112 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/mnt/c/Projects/others/py
torch_serve/ts/model_service_worker.py", line 134, in load_model
    2024-10-21T15:55:32,112 [INFO ] W-9026-mnist_traced_1.0-stdout MODEL_LOG -     service = model_loader.load( 
    2024-10-21T15:55:32,112 [INFO ] W-9026-mnist_traced_1.0-stdout MODEL_LOG -               ^^^^^^^^^^^^^^^^^^ 
    2024-10-21T15:55:32,113 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/mnt/c/Projects/others/pytorch_serve/ts/model_loader.py", line 110, in load      
    2024-10-21T15:55:32,113 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -     module = self._load_default_handler(handler)
    2024-10-21T15:55:32,114 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    2024-10-21T15:55:32,114 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/mnt/c/Projects/others/py
torch_serve/ts/model_loader.py", line 159, in _load_default_handler
    2024-10-21T15:55:32,115 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -     module = importlib.import_module(module_name, "ts.torch_handler")
    2024-10-21T15:55:32,115 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    2024-10-21T15:55:32,116 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "/home/racinsky/miniforge3
/lib/python3.12/importlib/__init__.py", line 90, in import_module
    2024-10-21T15:55:32,116 [INFO ] W-9026-mnist_traced_
1.0 org.pytorch.serve.wlm.ModelManager - Model mnist_traced unregistered.
    2024-10-21T15:55:32,116 [DEBUG] W-9026-mnist_traced_
1.0 org.pytorch.serve.wlm.WorkerThread - W-9026-mnist_tr
aced_1.0 State change WORKER_SCALED_DOWN -> WORKER_STOPPED
    2024-10-21T15:55:32,117 [INFO ] epollEventLoopGroup-
3-2 ACCESS_LOG - /127.0.0.1:54568 "POST /models?url=mnis
t_traced.mar&model_name=mnist_traced&runtime=python&initial_workers=1&synchronous=true HTTP/1.1" 500 3342       
    2024-10-21T15:55:32,117 [DEBUG] W-9026-mnist_traced_
1.0 org.pytorch.serve.wlm.WorkerThread - Worker terminated due to scale-down call.
    2024-10-21T15:55:32,117 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -     return _bootstrap._gcd_import(name[level:], package, level)
    2024-10-21T15:55:32,118 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

TorchServeSuite > TorchServe > org.pytorch.serve.ModelServerTest > testLoadMNISTTracedModel FAILED
    java.lang.AssertionError at ModelServerTest.java:2225

TorchServeSuite > TorchServe > org.pytorch.serve.ModelServerTest > testPredictionMNISTTracedModel STANDARD_OUT  
    2024-10-21T15:55:32,119 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    2024-10-21T15:55:32,119 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    2024-10-21T15:55:32,119 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
    2024-10-21T15:55:32,120 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    2024-10-21T15:55:32,120 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    2024-10-21T15:55:32,120 [INFO ] W-9026-mnist_traced_
1.0-stdout MODEL_LOG -   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    2024-10-21T15:55:32,121 [INFO ] W-9026-mnist_traced_
3-1 ACCESS_LOG - /127.0.0.1:41728 "POST /v2/models/mnist_traced/infer HTTP/1.1" 404 0

TorchServeSuite > TorchServe > org.pytorch.serve.ModelServerTest > testKFV2PredictionMNISTTracedModel FAILED
    java.lang.AssertionError at ModelServerTest.java:2341

TorchServeSuite > TorchServe > org.pytorch.serve.ModelServerTest > testKFV1ExplanationMNISTTracedModel STANDARD_OUT
    2024-10-21T15:55:32,818 [INFO ] epollEventLoopGroup-3-2 ACCESS_LOG - /127.0.0.1:41732 "POST /v1/models/mnist_traced:explain HTTP/1.1" 404 0

TorchServeSuite > TorchServe > org.pytorch.serve.ModelServerTest > testKFV1ExplanationMNISTTracedModel FAILED
    java.lang.AssertionError at ModelServerTest.java:2320
<==========---> 82% EXECUTING [1h 19m 13s]
> :server:test > Executing test org.pytorch.serve.ModelServerTest
> :server:test > 152 tests completed, 22 failed

no idea what I'm doing wrong, I'm on 85% for an hour.