kedro-sagemaker plugin fails to execute if a kedro project has more than 100 parameters

[njvijay-mck]

I'm currently using the Kedro-SageMaker plugin to run a Kedro pipeline on SageMaker. My Kedro project has multiple parameters (using several parameters YAML files).

While executing kedro sagemaker run -i <image name>, I encountered this error:

ClientError: ValidationError for UpdatePipeline operation: Unable to parse pipeline definition. Model validation failed - container ProcessingEnvironmentMap length (132) exceeds maximum limit (100).

Here are the steps to reproduce the issue.

• Add a dummy_parameter.yml file under conf/base/parameters/ in the kedro space flight example code. (check dummy_pamaramter.yml file content below)
• Simply run kedro sagemaker run --auto-build

dummy_parameter.yml file content:

dummy_params:
  parameter1: 1
  parameter2: 2
  parameter3: 3
  parameter4: 4
  parameter5: 5
  .
  .
  .
  .
  .
 parameter100: 100

[marrrcin]

Thanks for creating the issue @njvijay-mck !

Do you use the feature of parameter overriding on the SageMaker UI side? If not, then maybe we can introduce a flag to skip the translation of Kedro params to SageMaker params and just rely on the config files stored in the docker image 🤔

[njvijay-mck]

I am not using sagemaker UI for parameter override since I rely entirely on kedro parameters yaml file (mainly for training). It is a good idea to introduce a flag to skip. But in some cases, I depend on Kedro command parameters like --env, --pipeline

[marrrcin]

Are you willing to contribute that feature in a form of a PR?

[njvijay-mck]

I'd be happy to contribute, but it may take some time due to my current workload. I'll do my best to prioritize it as it is an important feature.

[marrrcin]

Awesome, let's stay in touch!

[marrrcin]

@njvijay-mck are you still willing to contribute a fix for this issue?