Quota Contribution/Status Page

[ellestad]

It would be handy to have the group storage quota contributions and current usage on the website as we do for the "Contributing Member Shares" page.

[HenrikBengtsson]

Roadmap to implement this:

1. Figure out how to query the system for this information
2. Write Bash script that pulls this information and outputs a tab-delimited file
3. Set up a cron job that write this file docs/assets/data/ and git commit ... and git push it, cf. https://github.com/ucsf-wynton/wynton-website-hpc/tree/master/cron-scripts
4. Using JavaScript to read this data file into a D3 object and then render it as a table in HTML, cf. https://raw.githubusercontent.com/ucsf-wynton/wynton-website-hpc/master/docs/about/shares.md

Things to consider:

• Is there a privacy issue here? Are we talking group-storage only? There's nothing that prevents us from providing the same info on a per-user basis, but then you're revealing to the world how much each user consumes, and overtime also when they use the cluster.

[jlbl]

Roadmap to implement this:

1. Figure out how to query the system for this information

See this page on the website.

[HenrikBengtsson]

Roadmap to implement this:

1. Figure out how to query the system for this information

See this page on the website.

And how do figure out which groups to be queried and reported on?

[HenrikBengtsson]

Turns out, from beegfs-ctl --getquota --help, that one can do:

beegfs-ctl --getquota --storagepoolid=12 --gid --range "$GID_FROM" "$GID_TO"

to get more than one at the time. So, what's the GID range to scan?

[ellestad]

GIDnumber range won't work, they could be in any part of the range.

Purchased group storage only.

If they have a non-zero quota for group storage, they have purchased storage. Ignore any group with a quota value of zero/unlimited (or 1).

[HenrikBengtsson]

GIDnumber range won't work, they could be in any part of the range.

Purchased group storage only.

If they have a non-zero quota for group storage, they have purchased storage. Ignore any group with a quota value of zero/unlimited (or 1).

So,

beegfs-ctl --getquota --storagepoolid=12 --gid --all

?

[HenrikBengtsson]

Hrmm, why do so many HPC tools output non-machine non-parsable output. You'd imagine it could generate tsv/csv output if it already produced tabular output. Oh well.

FYI, I've protyped a parser for the beegfs-ctl output, cf. https://ucsf-wynton.github.io/wyntonquery/reference/beegfs_quota.html. It's the first step to automate this.

[HenrikBengtsson]

I think we should be able to use Greg's beegfs-quota tool for this, because it reports on the hard limit too.

[Nicki-Martin]

@HenrikBengtsson @murashka-ucsf any update?

[Nicki-Martin]

@HenrikBengtsson and @murashka-ucsf any update?

[HenrikBengtsson]

I think we're waiting for @gregcouch to update his beegfs-quota tool, so it's possible to list the quota for all available groups;

$ beegfs-quota --group
Group Name           Used     Hard Limit Use% Filesystem      Storage Pool
cbi        14759580453008 43980465111040  34% beegfs_nodev#12 group

If someone can tell me how to list all available groups, one could write a for-loop around this tool (but it would more efficient if the tool does that internally).

[Nicki-Martin]

@HenrikBengtsson I don't have rights to add labels or anything to this issue - can you check my permissions please?

[HenrikBengtsson]

@HenrikBengtsson I don't have rights to add labels or anything to this issue - can you check my permissions please?

Oh. I've now added this repo to the 'wynton-ops' team - try again

[gregcouch]

Now that we have groups and subgroups, revise this to:

1. write a script that generates a list of groups, their subgroups, and associated allocation/quota (using LDAP information)
2. get the current quota use for each of those groups
3. show that information in a web page

[ellestad]

@gregcouch Note, you cannot determine subgroups from the parent group. You can only locate the parent group from the "parentGroup" entry in the subgroup.

[Nicki-Martin]

@Nicki-Martin raise in admin meeting