[BUG] Google Drive upload & replace file actions missing `uploadType` prop

[ctrlaltdylan]

Describe the bug

The Google Drive API requires the uploadType to equal resumable or multi-part if uploading files over 5mb in size.

However, we're currently not exposing this option in the Replace File or Upload File pre-built actions.

Specific Google Drive uploadType parameter docs:

https://developers.google.com/drive/api/guides/manage-uploads

To Reproduce Steps to reproduce the behavior:

1. Upload a large file over 5mbs to your workflow
2. Attempt to upload the file to your Google Drive account
3. Receive "File too large" error

Expected behavior I expect to be able to specify a uploadType prop on the pre-built action to allow me to upload larger files to Google Drive.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Original user report: https://pipedream.com/community/t/google-drive-replace-file-uploadtype-param/2551/3

[noahlocke-uw]

I believe the first statement in the issue description is in error:

The Google Drive API requires the uploadType to equal media if uploading files over 5mb in size.

If uploadType is set to media then the file must be less than 5MB. Files over 5MB must have an uploadType of resumable per Google's documentation:

[ctrlaltdylan]

Thanks for the correction @noahlocke-uw !

[vellames]

Humm, I was able to upload a 13mb image https://upload.wikimedia.org/wikipedia/commons/3/3d/LARGE_elevation.jpg

[ctrlaltdylan]

@vellames-turing can you share the Node.js code you used to do the upload? Or did you use a pre-built Google Drive action?

I believe @noahlocke-uw was hitting this issue with the Google Drive - Replace File pre-built action.

[vellames]

@ctrlaltdylan Pre-built action, anyways I am doing somethings to choose the upload type automatically based on the buffer size, also I am adding the prop :)

[ctrlaltdylan]

@vellames-turing oh very smart! Makes it just a bit more convenient for the customers to just use the pre-built action and not have to worry about setting the uploadType. Especially if the same workflow processes < 5mb files or > 5mb files.

Maybe just for debugging purposes we could leave a console.log on the uploadType during execution? Might be overkill.

[vellames]

Sure, np

[dannyroosevelt]

This is ready for release!

[ctrlaltdylan]

User is reporting that v0.0.5 of the Replace File action is still not working properly with large files, can we have this retested?

https://pipedream.com/community/t/google-drive-replace-file-uploadtype-param/2551/10

[vellames]

@ctrlaltdylan Is it possible to the user to share the specific file for analysis?

[ctrlaltdylan]

@vellames-turing hoping they will! I've asked them in the community forum for an example, hopefully we can get one or at least they can describe the exact size so we can reproduce.

[ctrlaltdylan]

@vellames-turing , @noahlocke-uw describes the file as ~7.5 megabytes and it's a CSV file.

https://pipedream.com/community/t/google-drive-replace-file-uploadtype-param/2551/12

Sounds like it's private data, but I think any file over 5 megabytes should reproduce the bug.

[vellames]

Humm, I was able to upload a 13mb image file, probably the problem is with the file or with csv

[noahlocke-uw]

0.5MB-raw.csv 7.5MB-raw.csv

@ctrlaltdylan @vellames-turing I've attached two files. The one that is failing is the 7.5MB csv file. If I do the exact same workflow with the 0.5MB csv file it succeeds. When I attempt to use the 7.5MB csv file using the uploadType set to resumable I get "Request Too Large" error.

[vellames]

Great, thank you!! I will take a look ASAP

[vellames]

Pretty strange, it works for me. @noahlocke-uw How exactly are you trying to replace the file? Are you uploading it for you /tmp? Could you also please check your action version? ![Uploading Screen Shot 2022-06-21 at 11.35.52 AM.png…]()

[noahlocke-uw]

Here are the steps:

I'm sending a POST to https://eob4e2hxcwt26fg.m.pipedream.net/?pipedream_upload_body=1 Pipedream uploads the payload (csv) to AWS S3 and exports the raw URL, which I'm passing to Google Drive Replace File v0.0.5

Again, this works if the file is less than 5MB, every time. It worked with v0.0.4 as well, so long as the file was under 5MB.

[vellames]

Could you please share the raw URL generated by S3 with me?

[noahlocke-uw]

I can, they are temporary and expire quite rapidly, so depending on when you get this it may or not be expired

https://pipedream-catcher-bodies.s3.amazonaws.com/3256dc4f-6ffc-4ed8-a653-97790aba2d47?AWSAccessKeyId=ASIA5F5AGIEAUNXFSF4E&Expires=1655836999&Signature=qkoaV8ry5GrGZDknjH3Vr0Z0ZsU%3D&x-amz-security-token=FwoGZXIvYXdzEEMaDKj1WP5d9LIp1oTMCSKNBPY836B8ghueS1gsBSbM%2FxIusO0jXJCf5O57NQazqjOvv8tcNMpNoCIg9sQLcO2M9JEwGRT%2F4ifRd4LJ9kj0Z7srao7X20fAxyXiVQvp2Ko1vjJlkujn5O0T5weEppKO6aH6wSynJfc8kR9QGYxvm%2FoiPsXuNgoPuNDSnp4brV%2FiRd9sUgkq686mcRizatVYeF5mrvxmoXBr7KBjIsjaXI2TQmyvdW3iuz8H9ISFvV8rG3H8FlASRTf4pE7PAfNMlFYWY2vezb3h4AemqSWuOGi4kRVssfa1UGVz8ap7wOFbaVpC42UiE90fzAs63clYsXAWjDSN5Kbhmpzy%2FLxSDEsWtJDMjuA%2Ft2S%2BRvX%2B76SIaroxSVT6ww2SQBfxj0f83n26d1L9L4XmU8It9Gg83NGT9N7tefMV0BACKhAJvVCBPckZc4KV9dQDmnM9hR4aKsfFVH0bCzz0tb15uKjmJ%2F2oJ5KdhoGWs1eldv43aoCxVzK%2B0G%2FKN2qEPMi4mZlM5%2FYe2Eyh3gLv7%2BXvoVB9MLO4wDuW89%2F85LqIJ1TiEm6Lmnb36XWe907nd5esSIKD%2FjjSLuUebKTx5%2FCc03Uvk22tsneYdqhNSRePodq29%2FcM8Xsamzj6MckAf7%2FWFjQbhFD6wiSNRWOGMhyMD%2F0cAMU2KdIqUx2ysBgo4OfeYWMIWd8JVc6MkFJc7IED1ijiiciVBjIq52TPq03epE77a%2FmYmWCBmRLec%2FSgm2eGzqDvabDVAKTKWnPxaJbQaYd%2B

[vellames]

Huummm, it still working for me, even using you URL @Hassan-Abbas-SQA could you please try to test these files as well? Maybe there is some specific condition that I cant see

0.5MB-raw.csv 7.5MB-raw.csv

@ctrlaltdylan @vellames-turing I've attached two files. The one that is failing is the 7.5MB csv file. If I do the exact same workflow with the 0.5MB csv file it succeeds. When I attempt to use the 7.5MB csv file using the uploadType set to resumable I get "Request Too Large" error.

[vellames]

@dylburger @dannyroosevelt @js07 . Do you have any idea what may be happening here?

[noahlocke-uw]

I'll try an entirely fresh workflow for good measure

[noahlocke-uw]

Brand new workflow, same steps, same error:

[noahlocke-uw]

But if I take the raw csv data and shave it down to below 5MB:

[noahlocke-uw]

Fixed! I changed the CURL to --data-binary and it works now

[noahlocke-uw]

FYI before I was using:

curl --data @data.csv https://pipedream-endpoint.com

now I'm using

curl --data-binary @data.csv https://pipedream-endpoint.com

[vellames]

Fixed! I changed the CURL to --data-binary and it works now

Oh, so it is working as you expected now?

[noahlocke-uw]

yes