Description of changes:
The structure for the model spec is different across model IDs. Some of them have "default_payloads" as a top-level attribute of the spec (which is what we have been assuming up til now), while others have "default_payloads" as a nested attribute, which is the case for models that have alt configs.
Note that sagemaker version 2.225.0 introduces a utility function, verify_model_region_and_return_specs which returns a JumpStartModelSpecs object that can be used in place of the raw model spec json object. So instead of manually parsing out the default payloads via model_spec[INFERENCE_CONFIG_COMPONENTS][TGI][DEFAULT_PAYLOADS], one would do model_spec.inference_configs.get_top_config_from_ranking().resolved_metadata_config["default_payloads"].
However, in the case where the model doesn't have alt configs (i.e. the default payloads are a top-level attribute of the model spec) model_spec.default_payloads doesn't actually give us what we want, as certain fields like generated_text will not be present.
Since when dealing with the case where "default_payloads" is a top-level attribute, we have to manually parse the model spec json object, I've decided to keep the code simple and not use the utility method introduced in sagemaker version 2.225.0.
This PR updates the model spec-parsing logic to account for the new, second case.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Issue #, if available:
Description of changes: The structure for the model spec is different across model IDs. Some of them have
"default_payloads"as a top-level attribute of the spec (which is what we have been assuming up til now), while others have"default_payloads"as a nested attribute, which is the case for models that have alt configs.Note that
sagemakerversion2.225.0introduces a utility function,verify_model_region_and_return_specswhich returns aJumpStartModelSpecsobject that can be used in place of the raw model spec json object. So instead of manually parsing out the default payloads viamodel_spec[INFERENCE_CONFIG_COMPONENTS][TGI][DEFAULT_PAYLOADS], one would domodel_spec.inference_configs.get_top_config_from_ranking().resolved_metadata_config["default_payloads"]. However, in the case where the model doesn't have alt configs (i.e. the default payloads are a top-level attribute of the model spec)model_spec.default_payloadsdoesn't actually give us what we want, as certain fields likegenerated_textwill not be present.Since when dealing with the case where
"default_payloads"is a top-level attribute, we have to manually parse the model spec json object, I've decided to keep the code simple and not use the utility method introduced insagemakerversion2.225.0.This PR updates the model spec-parsing logic to account for the new, second case.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.