Open jmahlik opened 2 years ago
It is not clear why there is such a limit.
Yeah this limit des not work very well with their other services.
These pipelines are regularly added to part of step functions.
I am trying to pass in the Step Function Task Token (which goes up to 1024 length) to our post-processing step of our Sagemaker transform pipeline and hitting this wall.
Was anyone able to get past this problem?
Was anyone able to get past this problem?
No, it's enforced by the api. Writing files to s3 or using secrets manager are about the only solutions.
Describe the bug Currently, the value length constraints for environment variables are quite limited in
CreateProcessingJob
andCreateTrainingJob
.This causes issues when passing urls or other long strings via the environment parameter, as they often exceed the limit. I am not aware of an alternative to pass items that shouldn't be written to files to a job at runtime. Storing some info on s3 and treating it as an input cannot be used as a workaround, as some variables should not be shared with everyone who has access to the s3 bucket nor stored long term. It's also inconvenient to write a notebook/user's env vars to s3, read them in, set them as env vars etc.
It looks like it would be possible to have an api call accept keys that are over 256 characters in length. Example https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html#sagemaker-CreateTrainingJob-request-HyperParameters accepts 2500 characters.
To reproduce Pass a string over the char limit similar to below.
Expected behavior Ability to pass longer environment variables to jobs.
Screenshots or logs
System information A description of your system. Please provide:
Additional context I suspect others may be facing the same issue as there is mention of passing custom urls for pip configurations.
https://github.com/aws/sagemaker-python-sdk/issues/2207#issuecomment-810096332