search

google / skicka

Command-line utility for working with Google Drive. Join the mailing list at https://groups.google.com/forum/#!forum/skicka-users.
Apache License 2.0
1.3k stars 118 forks source link

Optionally forgo the setting of extended metadata #87

Closed bobobo1618 closed 9 years ago

bobobo1618 commented 9 years ago

I'm mainly using Skicka as a faster way to update my stuff on Drive. The metadata cache make this far faster than Rclone which seems to insist on re-hashing all my local data every time I sync.

However I don't really need the additional metadata Skicka provides like UNIX style file permissions and with the amount of data I have (~50000 files), those extra requests can significantly slow things, particularly with Drive API rate limiting.

It'd be great if we had a switch to disable the extra metadata stuff.

mmp commented 9 years ago

I don't think that the metadata is causing any extra HTTP requests (but am open to being convinced that I'm wrong). To confirm, I looked at some upload traces using -debug -dump-http. The results were:

  1. When the local metadata cache is updated, it does ask for the extra properties that encode the permissions metadata, but that's one of multiple fields asked for, so it costs some bandwidth and such, but I assume that is fairly limited.
  2. When doing an upload, skicka currently does 3 HTTP transactions for a new file (and 2 for an already-existing one). For a new file, it does a POST to create the file; it includes the metadata during this one. It then does a PUT to fill in its contents. Only after a successful upload does it then do a PATCH to update the modification time. (It is careful to leave the modification time in the year 1970 until a successful upload is done, just so that if there's an upload error, there's no confusion that a partial file is actually legitimate.) For an existing file, it does a PUT and a PATCH.

(It'd be better to create new files with just a PUT and a PATCH and skip creating the zero-length file, but that's a separate issue.)

So, I don't think there are any separate HTTP transactions related to the permissions; that's just all wrapped up in the existing calls (as far as I can see).

bobobo1618 commented 9 years ago

Ah, sorry, you might be right. I just saw the logging messages all over the place and assumed that must be what was happening.

I ended up 'fixing' the issue by using my own API key and raising the limits anyway so it's all fine on my end :)