Open 0ut0fcontrol opened 7 months ago
DVC should work properly with the virtual-host style endpointurl addressing. The error indicates that you don't have the right permissions to access that bucket. Are you able to use the AWS CLI to access that byte-plus bucket?
It looks like you have to pass addressing_style: 'virtual'
to botocore to enable this?
In botocore it defaults to auto
, where if you are setting an endpointurl it uses virtual style and then falls back to path
https://github.com/boto/botocore/blob/e7c5b6ab22174797db551f44053a0b2245430649/botocore/utils.py#L2604-L2614
Thank you, @skshetry and @pmrowla.
When forcing boto3 to use {'addressing_style': 'virtual'}
, I can access byte-plus bucket:
when not, it raises error:
What should I do to make dvc work?
The error given with s3fs/DVC is a permission error when accessing a specific file. It does not look like s3fs is not failing to create the client session (InvalidPathAccess is not the raised exception when you use DVC). This would normally mean that the issue is specifically with the credentials you are setting for your DVC remote (and not that the issue is due to an incorrect addressing style)
@0ut0fcontrol can you verify that you are able to access your bucket via the AWS CLI?
@pmrowla Yes, I can access my bucket via aws cli:
And, if I insert config_kwargs['s3'] = {'addressing_style': 'virtual'}
in here, dvc will work fine.
However, I don't know how to pass this kwarg from dvc cli into s3fs.
add {'addressing_style': 'virtual'}
Comment out {'addressing_style': 'virtual'}
, dvc not work again.
This looks to me like it may be an aiobotocore or botocore bug since their behavior to default to virtual host addressing and fall back to path addressing may not be working correctly?
But we can add support for setting this explicitly in dvc-s3 dvc-s3 (we set config timeouts in the same way). It will also need to get added to the DVC remote config schema https://github.com/iterative/dvc-s3/blob/43f70226160f1a5c9ffdad4092d41a8bab7ec19b/dvc_s3/__init__.py#L176-L179
Does dvc support virtual-hosted–style S3-compatible remote?
From doc, dvc seem dvc only support path-style S3 remote.
When I try to use dvc with Tinder Object Storage (TOS) which only support virtual-hosted–style S3, it report error.
Error info: