passing configuration for spark processing job

[ajaiswalgit]

Describe the feature you'd like Feature to create spark.processing configuration.json file from kwargs to an S3 path. Many organizations do not have permission to write a file at bucket level.

How would this feature be used? Please describe. We need to pass few Spark configurations in configurations.json to override default behavior. Configs pass through kwargs.configuration creates a file configuration.json at default bucket level. But user do not have write access at bucket level.

Describe alternatives you've considered If we can pass default s3 path along with default S3 bucket then it will not fail.

Additional context User should be able to pass user_defined_s3_folder so that configuration.json is created at S3bucket/user_defined_s3_folder where they have got write permission.

s3_uri = ( f"s3://{self.sagemaker_session.default_bucket()}/{user_defined_s3_folder}/{self._current_job_name}/" f"input/{self._conf_container_input_name}/{self._conf_file_name}" )

[jmahlik]

Looks like this might be related to https://github.com/aws/sagemaker-python-sdk/issues/3200 and possibly fixed.

[martinRenou]

Closing as fixed by #3486