Example for using TorcheServe for pytorch trained models

[danielduckworth]

With the migration to PyTorch, can saved models be served with TorchServe?

It would be useful to add a guide to the examples showing how to serve a model with TorchServe. The reason I'm suggesting this is that to use ludwig serve, the whole library is required which is about a 17GB docker image.

It would be nice to have a lightweight docker image for model serving as a REST API.

[w4nderlust]

@danielduckworth In 0.5 models will be PyTorch models, but without data preprocessing and postprocessing you would not have much use of serving them directly. That's why on one hand there's the Ludwig serve command, and on the other hand we support neuropod and will support torchscript exports. The torchscript (which includes compiled version of preprocessing and postprocessing inside the torchscript output) will initially work only for simple data types and later will include also more complex ones, and will likely be the highest throughput solution that requires the smallest fottprint on disk and memory.

Anyway, I think there could be several optimziations to bring down the current image to something smaller than 16gb

[fire]

Is there design docs for how torchscript export will work?

[w4nderlust]

@fire no there's no specific design doc, but you can look at the ludwig-docs (ludwig05 branch) https://github.com/ludwig-ai/ludwig-docs/tree/ludwig05 . In summary though, there will be a command to export a Ludwig model (that is a combination of preprocessing instructions and PyTorch saved module(s)) to torchscript. If you look in master some features already have torchscript compatible processing functions, those are used in the compiled torchscript instead of the Python dataframe-based ones.

By the way, I'm a fan of Godot! Do you plan to use Ludwig models extrated as torchscript inside it? That would be super cool.

[fire]

My dream is to use the approach outlined in https://github.com/V-Sekai-fire/ML_fire_juan_mosaic where I take an ONNX model converted from torchscript and then use directml, cuda or cpu to execute inference.

A current task is inference bone names from a character scene.

I would also like to do other tasks that may not fit ludwig:

1. auto rigging https://github.com/PeizhuoLi/neural-blend-shapes
2. database search of 3d scenes
3. inputs + voice based plausible facial animations
4. motion matching 3d animation sets https://montreal.ubisoft.com/en/introducing-learned-motion-matching/

[fire]

So I converted the torchscript to a inference_module. https://github.com/V-Sekai-fire/ML_avatar_wellness

What are the next steps?

[geoffreyangus]

Hi @fire! I worked on some of the torchscript stuff so can help out here a bit.

Next step is probably checking that the exported model works as expected. You can verify this by making a Python dict of inputs, where each key is the CSV column name used for model training and each value is the corresponding inputs.

I took a look at your repository link– for text features, the model will expect a list of strings of length batch_size. For numerical features, the model will expect a tensor.

Once you have this, pass it in to the model and inspect the outputs. Let me know how this goes– super happy to answer any questions you have along the way.

[fire]

@geoffreyangus Can you write a blog post picking a standard problem for example number digits matching, and showing how it can be converted from torchscript served and also to be ran on an onnx runtime?

I don't need image support, but I could use image support, numerical, categorical and text.

Edited. I'll separate this to a new issue.