Forbidden Access to S3 bucket after domain initialization

[salemn]

Describe the bug I'm not able to deploy a HuggingFaceModel from local development environment due to forbidden access error while calling HeadBucket operation

To reproduce

• Set up single user domain on sagemaker
• Created new user with policy iam:PassRole on execution role generated during domain creation
• Create Access token for the aforementioned user
• Run AWS configure with the access token newly created
• Run the following code

• 
  import sagemaker
  import boto3
  from sagemaker.huggingface.model import HuggingFaceModel

iam_client = boto3.client('iam') role = iam_client.get_role(RoleName='AmazonSageMaker-ExecutionRole-XXXX')['Role']['Arn'] sess = sagemaker.Session(boto_session=boto3.session.Session(region_name="eu-north-1")) model_name = 'google/flan-t5-base'

Hub model configuration https://huggingface.co/models

hub = { 'HF_MODEL_ID': model_name, # model_id from hf.co/models 'HF_TASK': 'summarization' # NLP task you want to use for predictions }

huggingface_model = HuggingFaceModel( env=hub, role=role,
transformers_version="4.26", pytorch_version="1.13.1", py_version='py39', )

predictor = huggingface_model.deploy( initial_instance_count=1, instance_type="ml.m5.xlarge" )

**Expected behavior**
Should deploy model 

**Screenshots or logs**

Bucket sagemaker-eu-north-1-XXXXXXXXXXXX exists, but access is forbidden. Please try again after adding appropriate access. Traceback (most recent call last): File "/Users/naoufel/Work/Pers/model/Sample/main/SM_summary.py", line 22, in predictor = huggingface_model.deploy( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/huggingface/model.py", line 319, in deploy return super(HuggingFaceModel, self).deploy( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/model.py", line 1695, in deploy self._create_sagemaker_model( File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/model.py", line 930, in _create_sagemaker_model container_def = self.prepare_container_def( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/huggingface/model.py", line 524, in prepare_container_def self._upload_code(deploy_key_prefix, repack=True) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/model.py", line 756, in _upload_code bucket, key_prefix = s3.determine_bucket_and_prefix( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/s3_utils.py", line 147, in determine_bucket_and_prefix final_bucket = sagemaker_session.default_bucket() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/session.py", line 602, in default_bucket self._create_s3_bucket_if_it_does_not_exist( File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/session.py", line 634, in _create_s3_bucket_if_it_does_not_exist self.general_bucket_check_if_user_has_permission(bucket_name, s3, bucket, region, True) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/session.py", line 685, in general_bucket_check_if_user_has_permission s3.meta.client.head_bucket(Bucket=bucket_name) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/botocore/client.py", line 565, in _api_call return self._make_api_call(operation_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/botocore/client.py", line 1017, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.227.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: 1.13.1
- **Framework version**:
- **Python version**: Py39
- **CPU or GPU**:
- **Custom Docker image (Y/N)**:

**Additional context**
Add any other context about the problem here.