search

sagemathinc / cocalc

CoCalc: Collaborative Calculation in the Cloud
https://CoCalc.com
Other
1.17k stars 216 forks source link

fully support a user being a collaborator on an unbounded number of projects #4123

Open williamstein opened 5 years ago

williamstein commented 5 years ago

Right now if one user creates $N$ projects, where N is huge, and each project has about 100 collaborators (say), then things "go to hell", and they can't even sign in. Instead, there should be no limit at all on the number of projects that a user can create or be a collaborator on. To implement this, we'll need to restructure some of our data, queries, and assumptions to scale properly.

  1. Put an arbitrary limit (100? 500?) on the initial number of results from the projects and collaborators queries.

  2. Changefeeds still work as always, but with a way to handle changes to records that aren't in the current query, as seen by the client.

  3. Make it possible for the client to explicitly request more data about either the projects or collaborators queries: either "another page of records", or "a specific record or records".

  4. Restructure the information about "upgrades I have applied" so it is NOT in the projects table at all. The problem is that right now it is in the users JSONB column in the projects table, making it require a costly scan of all projects the user is a collab on to know. This is why you might see "You may have unallocated Disk Space, Idle Timeout, Internet Access and Member Hosting upgrades..." in the Projects page before you click "Load all older projects". Because without loading everything, currently there is no way to safely know the state of your global upgrads allocation. This needs to change. Probably there should be a completely new table with upgrade allocations in it, with a column for the account-id and a column for the project-id. That would very nicely solve this problem...

With the above changes, things should scale to any global number of projects and collaborators (though there would still be a limit on the number of collabs per project).

REQUESTED BY: Minerva

williamstein commented 5 years ago

See #4124 for scaling the number of users on one project at once.

williamstein commented 5 years ago

NOTE: There is a university with an admin account that owns well over 10,000 projects, and it works fine, at least when tested with a fast internet connection. So the N that makes cocalc break and grind to a halt is probably somewhere between 20,000 and 100,000 projects. With 10,000 things continue to work pretty well, even with all projects loaded.