Google Drive 5 million file count limit - Githubissues

function61 / varasto

All your files in one simple, replicated, encrypted place - with built-in backups and configurable ransomware protection.

https://function61.com/varasto

Apache License 2.0

53 stars 5 forks source link

Google Drive 5 million file count limit #221

Open joonas-fi opened 1 year ago

joonas-fi commented 1 year ago

Problem

https://www.theverge.com/2023/4/3/23667971/google-drive-5-million-files-limit-storage

My own stats 2023-04-11

7.5 million files
26.85 TB in total
3.76 MB / average file

Possible solution

Pack multiple files inside a single archive file:
- Blob A: byte range 0-1234
- Blob B: byte range 1235-9999
- ...
Partial download is supported by their API: https://developers.google.com/drive/api/guides/manage-downloads#partial_download
Does G drive allow enough metadata for us to keep the "blob ID => archive filename" look-up-table in G drive only?

Worries

If Google reintroduces a limit without a warning, we can't start writing these archive files before we get our state under the limit.
What if Google gives a migration timeline to the new limit, but the timeline is not enough to rewrite our storage to be below this limit? F.ex. right now I'm moving 3.2 million files to another directory and it takes something like 18 days because we're rate limited to 2 moves a second.

joonas-fi commented 3 months ago

Google has gone to shit while backpedaling on unlimited storage and constantly worsening the use policies.

I will no longer spend any effort in supporting any of their garbage.