unclear how to create a folder with google_drive2

[huin]

I've been assuming that the correct call for this is Drive::files().insert(). However, when using this to "upload" an empty file whose mime type is application/vnd.google-apps.folder, I get a 500 error in response.

This is an excerpt of the code that I'm using unsuccessfully:

            let fm = Mime(TopLevel::Application,
                SubLevel::Ext("vnd.google-apps.folder".to_string()),
                vec![]);

            let emptybuf: [u8; 0] = []; 
            let empty_stream = Cursor::new(&emptybuf);
            let mut folder_info: drive2::File = Default::default();
            folder_info.title = Some(drive_folder_name);
            folder_info.mime_type = Some(fm.to_string());
            hub.files()
                .insert(folder_info)
                .add_scope(drive2::Scope::File)
                .param("fields", "id")
                .upload(empty_stream, fm) 

Other Google Drive APIs seem to have an execute method to perform the file insert operation, which feels more correct in the case of creating a folder, rather than uploading an empty file. However, I'm not clear how else to perform this using this Rust API, which only provides upload methods to perform the files.insert request.

https://developers.google.com/drive/v2/web/folder

[Byron]

Even though I have not used it in a while, it was working back in the days when I was using it for YouTube. Can you post more information about the 500 you are getting? Maybe it reveals information useful to solving this issue. Thank you

[huin]

Source code: https://github.com/huin/test-gdrive-create-folder

Results in the following: (credentials and multipart markers replaced with Xs)

Connector #1: Successfully connected www.googleapis.com:443 (https)
Stream #1/1: Wrote 722 bytes: POST //upload/drive/v2/files?fields=id&alt=json&uploadType=multipart HTTP/1.1\r\nHost: www.googleapis.com\r\nUser-Agent: google-api-rust-client/1.0.5\r\nAuthorization: Bearer XXXXXXX\r\nContent-Type: multipart/Related; boundary=XXXXXX\r\nTransfer-Encoding: chunked\r\n\r\n130\r\n\r\n--XXXXXX\r\nContent-Type: application/json\r\nContent-Length: 71\r\n\r\n{\"mimeType\":\"application/vnd.google-apps.folder\",\"title\":\"test-folder\"}\r\n--XXXXXX\r\nContent-Type: application/vnd.google-apps.folder\r\nContent-Length: 0\r\n\r\n\r\n--XXXXXX--\r\n0\r\n\r\n
Stream #1/1: Flush succeeded
Stream #1/1: Read 548 bytes: HTTP/1.1 500 Internal Server Error\r\nX-GUploader-UploadID: XXXXXXXXXXXXXXXXXXXX\r\nVary: Origin\r\nVary: X-Origin\r\nContent-Type: application/json; charset=UTF-8\r\nContent-Length: 180\r\nDate: Mon, 12 Jun 2017 17:03:22 GMT\r\nServer: UploadServer\r\nAlt-Svc: quic=\":443\"; ma=2592000; v=\"38,37,36,35\"\r\n\r\n{\n \"error\": {\n  \"errors\": [\n   {\n    \"domain\": \"global\",\n    \"reason\": \"internalError\",\n    \"message\": \"Internal Error\"\n   }\n  ],\n  \"code\": 500,\n  \"message\": \"Internal Error\"\n }\n}\n
Stream #1/1: Dropped
thread 'main' panicked at 'failed to create folder: BadRequest(ErrorResponse { error: ServerError { errors: [ServerMessage { domain: "global", reason: "internalError", message: "Internal Error", location_type: None, location: None }], code: 500, message: "Internal Error" } })', /checkout/src/libcore/result.rs:859
note: Run with `RUST_BACKTRACE=1` for a backtrace.

[Byron]

Thanks so much for the test-program, it helps tremendously!

I have a hunch that the problem may be that the API forces you to upload zero bytes as multipart body, which it just doesn't expect to see. A solution could be to offer a plain doIt() method in addition to the ones that want to upload data. As a side-note, I also tried the provided example program with the drive3 API, which yielded exactly the same result.

[Byron]

Just for testing I have pushed to a fork of the excellent test program you provided - thanks again for that by the way!

It turns out that uploading anything is causing the error you see. However, I really believe that the endpoint should allow that as technically, Folders are just a special form of file. Maybe this can be suggested to the respective team at google?

Please see the details for the error I now get - it seems that none of the scopes drive or drive.file are sufficient to create a folder when tokens are obtained through Flow::InstalledInteractive or Flow::InstalledRedirect(_). I know that some operations can only be performed if the a certain Oauth2 flow is used, but I wouldn't know what else there is that could cause a permission issue.

Maybe you have ideas about that too.

With scope `drive.file` ``` Connector #1: Successfully connected www.googleapis.com:443 (https) Stream #1/1: Wrote 400 bytes: POST /drive/v3/teamdrives?fields=id&alt=json HTTP/1.1\r\nHost: www.googleapis.com\r\nUser-Agent: google-api-rust-client/1.0.5\r\nAuthorization: Bearer XXXX\r\nContent-Type: application/json\r\nContent-Length: 70\r\n\r\n{\"mimeType\":\"application/vnd.google-apps.folder\",\"name\":\"test-folder\"} Stream #1/1: Flush succeeded Stream #1/1: Read 794 bytes: HTTP/1.1 403 Forbidden\r\nVary: X-Origin\r\nWWW-Authenticate: Bearer realm=\"https://accounts.google.com/\", error=insufficient_scope, scope=\"https://www.googleapis.com/auth/drive\"\r\nContent-Type: application/json; charset=UTF-8\r\nDate: Sat, 17 Jun 2017 08:44:36 GMT\r\nExpires: Sat, 17 Jun 2017 08:44:36 GMT\r\nCache-Control: private, max-age=0\r\nX-Content-Type-Options: nosniff\r\nX-Frame-Options: SAMEORIGIN\r\nX-XSS-Protection: 1; mode=block\r\nServer: GSE\r\nAlt-Svc: quic=\":443\"; ma=2592000; v=\"38,37,36,35\"\r\nAccept-Ranges: none\r\nVary: Origin,Accept-Encoding\r\nTransfer-Encoding: chunked\r\n\r\nd0\r\n{\n \"error\": {\n \"errors\": [\n {\n \"domain\": \"global\",\n \"reason\": \"insufficientPermissions\",\n \"message\": \"Insufficient Permission\"\n }\n ],\n \"code\": 403,\n \"message\": \"Insufficient Permission\"\n }\n}\n\r\n0\r\n\r\n Stream #1/1: Dropped ``` With scope `drive` ``` Connector #1: Successfully connected www.googleapis.com:443 (https) Stream #1/1: Wrote 426 bytes: POST /drive/v3/teamdrives?requestId=some-request-id&fields=id&alt=json HTTP/1.1\r\nHost: www.googleapis.com\r\nUser-Agent: google-api-rust-client/1.0.5\r\nAuthorization: Bearer XXXXXX\r\nContent-Type: application/json\r\nContent-Length: 70\r\n\r\n{\"mimeType\":\"application/vnd.google-apps.folder\",\"name\":\"test-folder\"} Stream #1/1: Flush succeeded Stream #1/1: Read 737 bytes: HTTP/1.1 403 Forbidden\r\nVary: X-Origin\r\nContent-Type: application/json; charset=UTF-8\r\nDate: Sat, 17 Jun 2017 08:57:49 GMT\r\nExpires: Sat, 17 Jun 2017 08:57:49 GMT\r\nCache-Control: private, max-age=0\r\nX-Content-Type-Options: nosniff\r\nX-Frame-Options: SAMEORIGIN\r\nX-XSS-Protection: 1; mode=block\r\nServer: GSE\r\nAlt-Svc: quic=\":443\"; ma=2592000; v=\"38,37,36,35\"\r\nAccept-Ranges: none\r\nVary: Origin,Accept-Encoding\r\nTransfer-Encoding: chunked\r\n\r\n11e\r\n{\n \"error\": {\n \"errors\": [\n {\n \"domain\": \"global\",\n \"reason\": \"insufficientFilePermissions\",\n \"message\": \"The user does not have sufficient permissions for this file.\"\n }\n ],\n \"code\": 403,\n \"message\": \"The user does not have sufficient permissions for this file.\"\n }\n}\n\r\n0\r\n\r\n Stream #1/1: Dropped ```

[huin]

Side note: I'm not sure how sensitive things are in what you previously pasted - I was a bit paranoid and redacted things in my own previous comment, including the "Bearer" thing that might have been related to the auth token. My Oauth knowledge is pretty shaky, however.

Sorry for the slow reply. I've made another (slightly hacky) test program using Google's Go drive v2 client, based on their example code. This appears to successfully create a folder - and I can confirm that by using the ID it printed on drive.google.com.

https://github.com/huin/test-gdrive-create-folder-golang

Output (with redaction Xs where irrelevant/potentially sensitive), on a client that had already got an Oauth token (with no change in scopes):

2017/06/21 09:59:28 Request: "POST /drive/v2/files?alt=json HTTP/1.1\r\nHost: www.googleapis.com\r\nUser-Agent: google-api-go-client/0.5\r\nContent-Length: 72\r\nAuthorization: Bearer XXXXXXXXX\r\nContent-Type: application/json\r\nAccept-Encoding: gzip\r\n\r\n{\"mimeType\":\"application/vnd.google-apps.folder\",\"title\":\"test-folder\"}\n"
2017/06/21 09:59:30 Response: "HTTP/2.0 200 OK\r\nAlt-Svc: quic=\":443\"; ma=2592000; v=\"38,37,36,35\"\r\nCache-Control: no-cache, no-store, max-age=0, must-revalidate\r\nContent-Type: application/json; charset=UTF-8\r\nDate: Wed, 21 Jun 2017 09:03:12 GMT\r\nEtag: \"XXXXXXXXXXXXXX\"\r\nExpires: Mon, 01 Jan 1990 00:00:00 GMT\r\nPragma: no-cache\r\nServer: GSE\r\nVary: Origin\r\nVary: X-Origin\r\nX-Content-Type-Options: nosniff\r\nX-Frame-Options: SAMEORIGIN\r\nX-Xss-Protection: 1; mode=block\r\n\r\n"
Folder inserted:  XXXXXXXXX

Note that the equivalent API for Go doesn't seem to require the file content to be uploaded:

https://godoc.org/google.golang.org/api/drive/v2#FilesInsertCall

Specifically the Media and ResumableMedia methods are not required for the final execution of the call. I think this is a key difference between how the Rust API is implemented versus the official Go one. The logged HTTP call from the Go client doesn't attempt an upload, but goes straight to POSTing the folder's metadata.

[Byron]

Thanks @huin for letting me know! Indeed the token snuck into the output, and could have been used to access my private google drive for a few hours at least. There is no sensitive data on it fortunately, but it's somewhat worrisome anyway.

That said, I have come to the same conclusion that you arrived at in my rust-based tests (see the commits linked in this thread), and believe that the drive API would be more consistent if folders could indeed be treated like empty files, including the upload of 0 bytes.

Right now the generator has no way of knowing about this special case, and even though everything is possible (I can provide custom flags on a per-API basis for example), I would find it wrong to fix it on this side.

So you think we should reach out to the google team doing the drive API? It's a bug on their side, clearly, as the http status 500 indicates.

[huin]

I'm not sure that I'm of the opinion that folders should be treated as files with zero bytes -- the other language APIs and existence of the "insert" (rather than upload) endpoint support not uploading file content at all. The reason I wrote code to do a zero-byte upload was because that's the closest to what seemed like a correct way with the Rust API - but it gave me pause at the time, and I wasn't entirely surprised to see it fail.

Regarding the 500 - it does sound like there is a server-side bug in handling an upload in the case of a folder mime type (i.e maybe it should either create the folder, or return a 400 bad request error). I don't think this implies one way or the other that the upload endpoint was intended for creating folders. Maybe the documentation could/should be clearer on this front.

[Byron]

From a logical perspective I follow your train of thought without hesitance.Thus, maybe one way to fix it on the end of the API generator is to add a third case which doesn't allow to upload any bytes at all, implying a zero-byte media upload just for the sake of creating it.

That would certainly be favourable, after all even if the 500 in the google-drive API would be fixed, there might be others that do it similarly.

[huin]

I think that approach makes sense to me.

[cristicbz]

So we're hitting what actually looks like the same issue with Google Cloud Storage. I think the reason for this error is much more mundane: if you look at the trace, the URL you're posting to is

POST //upload/drive/v2/files?fields=id&alt=json&uploadType=multipart HTTP/1.1\r\nHost: www.googleapis.com\r\nUser-Agent: google-api-rust-client/1.0.5\r\nAuthorization: Bearer XXXXXXX\r\nContent-Type: multipart/Related; boundary=XXXXXX\r\nTransfer-Encoding: chunked\r\n\r\n130\r\n\r\n--XXXXXX

Notice the double slash at the beginning; I think this error was introduced by #168 which went from python path.join to rust string concatenation. I think we should strip the path of leading slashes.