search

triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
BSD 3-Clause "New" or "Revised" License
8.18k stars 1.46k forks source link

[Metrics] | When model namespacing is enabled, Triton does not distinguish Metrics between two models with same name belonging to different ensemble/namespace #6138

Closed nikhil-sk closed 5 months ago

nikhil-sk commented 1 year ago

Description

  1. In SageMaker, we enable --model-namespacing=true in order to treat every model repository that is registered as its own namespace.
  2. Enabling model namespacing allows us to re-use the 'model-name' of a dependent model in ensemble across another model. E.g. if model ensemble_1 depends on dali_1 and pytorch_1, and if ensemble_2 depends on dali_1 and pytorch_2, with model namespacing dali_1 for first ensemble and dali_1 for second ensemble need to be in different model repositories, however, the name 'dali_1' is allowed to repeat.
  3. Now what happens is - this dali_1 model is not distinguished when reporting metrics

Triton Information What version of Triton are you using? 23.05

Are you using the Triton container or did you build it yourself? We build ourselves by enabling --enable-sagemaker flag in build.py (but is essentially same as NGC Triton container)

To Reproduce Steps to reproduce the behavior.

  1. When running Triton server, specify --allow-metrics=true and --model-namespacing=true.
  2. Register repository_1 and repository_2 using TRITONSERVER_ServerRegisterModelRepository API. Within each repo respectively, load ensemble_1 and ensemble_2 as described above with one of the dependent model names repeated e.g., dali_1.
  3. Run some inference requests for ensemble_1 and ensemble_2.
  4. Query metrics as localhost:8002/metrics.
  5. You will notice that the metrics for ensemble_1/dali_1 and ensemble_2/dali_1 have 'added up'/aggregated for dali_1 as there is no namspace-level distinguishing.

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

Expected behavior dali_1 from ensemble_1 and dali_1 from ensemble_2 should be treated as separate models when reporting in metrics. It can perhaps be done by adding a prefix for the namespace e.g., ensemble_1-dali_1, and ensemble_2-dali_1

dyastremsky commented 1 year ago

CC: @GuanLuo

dyastremsky commented 1 year ago

@GuanLuo, have you seen this issue with the model namespaces? Is this something we should create a ticket to work on?

dyastremsky commented 11 months ago

Thank you for submitting this bug with detailed reproduction information! We have filed a ticket to investigate.

Ticket reference: DLIS-5722.

kthui commented 5 months ago

Hi @nskool, we have added the "namespace" label when model namespacing is enabled. This will distinguish models with the same name and version but on different namespaces. Please let us know if there are any issues.