Multiple FIL models in a Triton ensemble fails due to `output__0` output name requirement

[michaelhagel]

I have a fairly robust Triton config generator for complex ensembles. In attempting to ensemble two XGBoost FIL models and two preprocessing steps, I run into the issue of the generator defaulting to using the output name output__0 for both FIL models, as the docs indicate is necessary. However, looking at this generated ensemble pbtxt:

name: "ensemble"
platform: "ensemble"
input {
  name: "preprocessing_ltcp2__input"
  data_type: TYPE_STRING
  dims: -1
  dims: -1
}
output {
  name: "output__0"
  data_type: TYPE_FP32
  dims: 1
}
output {
  name: "output__0"
  data_type: TYPE_FP32
  dims: 1
}
ensemble_scheduling {
  step {
    model_name: "preprocessing_ltcp2"
    model_version: -1
    input_map {
      key: "preprocessing_ltcp2__input"
      value: "preprocessing_ltcp2__input"
    }
    output_map {
      key: "preprocessing_ltcp2__output__ID:0dfcb__0"
      value: "preprocessing_ltcp2__output__ID:0dfcb__0"
    }
  }
  step {
    model_name: "LTCP2"
    model_version: -1
    input_map {
      key: "input__0"
      value: "preprocessing_ltcp2__output__ID:0dfcb__0"
    }
    output_map {
      key: "output__0"
      value: "output__0"
    }
  }
  step {
    model_name: "preprocessing_mpe2"
    model_version: -1
    input_map {
      key: "output__0"
      value: "output__0"
    }
    input_map {
      key: "preprocessing_ltcp2__output__ID:0dfcb__0"
      value: "preprocessing_ltcp2__output__ID:0dfcb__0"
    }
    output_map {
      key: "preprocessing_mpe2__output__ID:a7f6b__0"
      value: "preprocessing_mpe2__output__ID:a7f6b__0"
    }
  }
  step {
    model_name: "MPE2"
    model_version: -1
    input_map {
      key: "input__0"
      value: "preprocessing_mpe2__output__ID:a7f6b__0"
    }
    output_map {
      key: "output__0"
      value: "output__0"
    }
  }
}

we understandably receive the error: ensemble tensor 'output__0' can appear in an output map only once for ensemble 'ensemble'.

How do we handle ensemble output mapping in the case of an ensemble having two FIL models, then?

I know BLS is an option, but I would generally prefer to reserve that for actual business logic applications and not what should be a linear flow of preprocessing -> model -> preprocessing -> model.

[michaelhagel]

I should note that LTCP2 and MPE2 are the FIL models.

[wphicks]

Thanks for raising this! The team that maintains the FIL backend is separate from the team that works on ensembling, so I've asked them to weigh in here as well to see whether this is better addressed in the FIL backend or the ensemble implementation. More soon!

[Tabrizian]

Hi @michaelhagel, you can use ensemble with two models that use the same output name. However, the value field in the output_map should be unique across all ensemble steps. Also, I noticed that you have two output tensors with the same name which is incorrect. Based on the pipeline description, I assume the ensemble model only has one output. In that case, the model configuration should look like below:

name: "ensemble"
platform: "ensemble"
input {
  name: "preprocessing_ltcp2__input"
  data_type: TYPE_STRING
  dims: -1
  dims: -1
}
output {
  name: "output__0"
  data_type: TYPE_FP32
  dims: 1
}
ensemble_scheduling {
  step {
    model_name: "preprocessing_ltcp2"
    model_version: -1
    input_map {
      key: "preprocessing_ltcp2__input"
      value: "preprocessing_ltcp2__input"
    }
    output_map {
      key: "preprocessing_ltcp2__output__ID:0dfcb__0"
      value: "preprocessing_ltcp2__output__ID:0dfcb__0"
    }
  }
  step {
    model_name: "LTCP2"
    model_version: -1
    input_map {
      key: "input__0"
      value: "preprocessing_ltcp2__output__ID:0dfcb__0"
    }
    output_map {
      key: "output__0"
      value: "ltcp2__output" # Note the change here
    }
  }
  step {
    model_name: "preprocessing_mpe2"
    model_version: -1
    input_map {
      key: "output__0" 
      value: "ltcp2__output" # Note the change here
    }
    input_map {
      key: "preprocessing_ltcp2__output__ID:0dfcb__0"
      value: "preprocessing_ltcp2__output__ID:0dfcb__0"
    }
    output_map {
      key: "preprocessing_mpe2__output__ID:a7f6b__0"
      value: "preprocessing_mpe2__output__ID:a7f6b__0"
    }
  }
  step {
    model_name: "MPE2"
    model_version: -1
    input_map {
      key: "input__0"
      value: "preprocessing_mpe2__output__ID:a7f6b__0"
    }
    output_map {
      key: "output__0"
      value: "output__0"
    }
  }
}

[michaelhagel]

Thank you for the reply, and good call out on the value portion of the mapping. For the outputs in this case I was checkpointing both FIL model scores for output, hence the two outputs from the ensemble. But with changes to the value name in the mapping, I think that should be remedied as well?

Will make appropriate changes to my generator, thanks again.

[Tabrizian]

But with changes to the value name in the mapping, I think that should be remedied as well?

Yes, that can be resolved too. You just need to use unique output tensor names for ensemble.

[wphicks]

Thanks so much, @Tabrizian! @michaelhagel I'm going to mark this as resolved, but please don't hesitate to re-open if your question isn't fully answered or you run into other problems.