search

pulumi / pulumi-azure-native

Azure Native Provider
Apache License 2.0
127 stars 34 forks source link

Support chunked uploads for blobs #2150

Open sanmai-NL opened 1 year ago

sanmai-NL commented 1 year ago

Hello!

Issue details

I can't find documentation on how to upload a large file (of about 1.2 GiB) using Pulumi's FileAsset class. It's possible using the Azure's own Python SDK.

  azure-native:storage:Blob (vhd-linux):
    error: blobs.Client#PutBlockBlob: Failure responding to request: StatusCode=413 -- Original Error: autorest/azure: Service returned an error. Status=413 Code="RequestBodyTooLarge" Message="The request body is too large and exceeds the maximum permissible limit.\nRequestId:39dda59b-701e-0061-083a-155045000000\nTime:2022-12-21T12:51:17.7652469Z"

I am using Pulumi v3.50.0.

Affected area/feature

danielrbradley commented 1 year ago

Thanks for writing this up @sanmai-NL

Workarounds

As mentioned, this is possible via the Azure SDK. You can also wrap the SDK in a dynamic provider to use it within the lifecycle of a Pulumi program: https://github.com/pulumi/examples/tree/master/aws-py-dynamicresource

Another approach could be to use the Pulumi Command Provider to do this operation via the Azure CLI.

Notes

Blob support is currently manually implemented because it's based on the the azure data plane rather than the resource provider API. I believe this is implemented here: https://github.com/pulumi/pulumi-azure-native/blob/81cd3f7af267f2cd451c32f5b1441d29bfb46c2a/provider/pkg/resources/custom_storage.go

sanmai-NL commented 1 year ago

Understood. Thanks for taking interest in this. From a user perspective, though, the whole idea of infrastructure-as-code with Pulumi falls apart when you have to shell out to some executable in the middle of a deployment process, with the auth and reproducibility/predictability problems that come with it.

Side note, Microsoft seems to advise azcopy for these operations. Perhaps Pulumi can wrap that.

danielrbradley commented 1 year ago

Yup, would be good to be able to solve this properly in the provider. The workaround is just the temporary stop-gap until we can address the issue at its root.