Open SkalskiP opened 1 year ago
@SkalskiP I checked the implemented code. In general, the idea of using torch server is not good. For example, torch server can be replaced by REST api server(Flask/ fastapi/tornado/etc), model serving(onnx/opencv dnn/Tensorflow/etc) with the usage of opencv. In this way, the user is not bound to use only torch server. If I am missing anything specific advantage then let me know. I can create such small server within a day. I already built something like that.
@hardikdava - basically it is done - obviously, you can implement it in another way - but this method supports ONNX, TRT, TF, and whatever you want - TorchServe is "Torch"-oriented only by the name. We now have YOLOv5 and YOLOv7, it is trivial to deploy TorchHub, other models should also work. Obviously, that is only an example server - everyone can provide their implementation just interface matters
It is better now to focus on integration from labeling app side
Hi @PawelPeczek and @hardikdava 👋!
I apologize to you for engaging so little here so far. The new job is weighing me down a bit. I promise to improve. :)
Guys, remember that this server is only an example that we will use for development and as a guideline for others to build an API that is compatible with make-sense. Others may use it but don't have to :)
@PawelPeczek could you describe in simple words how other non-torch models than be deployed?
up to details described in readme - custom dependency can be installed in environment and the inside model handler - at first model needs to be loaded (it will yield some object of specific type - and torch.device object should only be used there to conclude which device to use), then at inference time - properly constructed handler function is going to be given reference to model and input image - this should be enough to infer. If I were not so lazy I could add onnx model for instance :joy:
As per readme:
ModelObject = Any # this can be anything even, tensorflow if someone wanted
InferenceFunction = Callable[
[ModelObject, List[np.ndarray], torch.device], List[InferenceResult]
]
and you are supposed to construct two functions in your module - first is:
def load_model(
context: Context, device: torch.device
) -> Tuple[ModelObject, InferenceFunction]:
and the second is InferenceFunction
as per signature
overview
The reason for the repository is to give makesense.ai users even more opportunities to support manual annotation with pre-trained models. So far, we've been using tensorflow.js for this purpose, but quite recently there was an issue https://github.com/SkalskiP/make-sense/issues/293 where users are asking to add support for inference via http. I've been thinking about this for a while now, so we're doing it!
scope
scope (nice to have)