search

pytorch / serve

Serve, optimize and scale PyTorch models in production
https://pytorch.org/serve/
Apache License 2.0
4.23k stars 863 forks source link

load_workflows Feature #2615

Open lauesa opened 1 year ago

lauesa commented 1 year ago

🚀 The feature

Add a load_workflows parameter to the config file to mirror the functionality of the load_models feature.

Motivation, pitch

Currently, one must register a workflow through the API in order to have it run. There is a load_models feature that allows listed models to be automatically loaded on torchserve start. I would propose a similiar functionality for workflow archives so that systems that support workflows can be automatically launched from the workflow_store.

Alternatives

I have a rather hacky script that searches the workflow_store and launches them with the REST API. I would prefer this be supported natively.

Additional context

If you add a .war file to the load_models parameter, you'll get the following error:

2023-09-22T15:32:56,033 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...
2023-09-22T15:32:56,052 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: sample.war
2023-09-22T15:32:56,063 [WARN ] main org.pytorch.serve.ModelServer - Failed to load model: sample.war
java.io.IOException: Input is not in the .gz format
    at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.init(GzipCompressorInputStream.java:240) ~[model-server.jar:?]
    at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.<init>(GzipCompressorInputStream.java:189) ~[model-server.jar:?]
    at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.<init>(GzipCompressorInputStream.java:153) ~[model-server.jar:?]
    at org.pytorch.serve.archive.utils.ZipUtils.decompressTarGzipFile(ZipUtils.java:105) ~[model-server.jar:?]
    at org.pytorch.serve.archive.utils.ZipUtils.unzip(ZipUtils.java:91) ~[model-server.jar:?]
    at org.pytorch.serve.archive.model.ModelArchive.downloadModel(ModelArchive.java:76) ~[model-server.jar:?]
    at org.pytorch.serve.wlm.ModelManager.createModelArchive(ModelManager.java:169) ~[model-server.jar:?]
    at org.pytorch.serve.wlm.ModelManager.registerModel(ModelManager.java:135) ~[model-server.jar:?]
    at org.pytorch.serve.ModelServer.initModelStore(ModelServer.java:264) [model-server.jar:?]
    at org.pytorch.serve.ModelServer.startRESTserver(ModelServer.java:396) [model-server.jar:?]
    at org.pytorch.serve.ModelServer.startAndWait(ModelServer.java:118) [model-server.jar:?]
    at org.pytorch.serve.ModelServer.main(ModelServer.java:99) [model-server.jar:?]
2023-09-22T15:32:56,074 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2023-09-22T15:32:56,122 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080
2023-09-22T15:32:56,122 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2023-09-22T15:32:56,123 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081
2023-09-22T15:32:56,123 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2023-09-22T15:32:56,124 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://0.0.0.0:8082
Model server started.
samils7 commented 1 year ago

This would be a nice feature to improve user experience working with workflows.