OSC / ondemand

Supercomputing. Seamlessly. Open, Interactive HPC Via the Web
https://openondemand.org/
MIT License
289 stars 106 forks source link

When home directories are full the error message is not verbose enough to the end user #3846

Open rgbryant opened 1 month ago

rgbryant commented 1 month ago

When a users home directory is full it make OOD unusable. I can check the logs and see the full home directory message, but it is not presented to the user on the webpage.

If this was presented on the webpage to the end user it would prevent the user from sending a ticket to their institiution's support team and then the team telling them that their home directory is full. This stops the researcher from working while the ticket is answered for something that should just be presented on the webpage.

Asked about here as well: https://discourse.openondemand.org/t/custom-error-page-when-home-directories-are-full/3788

Jeff says this is a big problem at OSC as well. This seems like a simple thing that could reduce a lot of support back and forth and keep researcher working instead of waiting on a response from a ticket.

stdweird commented 1 month ago

i guess the big problem here is what "full" means and how it can be tested.

@johrstrom ideally this is implemented in a way a site can refine/change/update it (even a bash script that can be reconfigured would already be good.)

without quota, i would implement it in bash by trying to write a 10MB files and also touch 99 files (need free inodes as well) to the HOME/ondemand dir. with quota, i would have to come up with reliable calculation (eg consider the gpfs in_doubt data).

ideally ondemand can handle some caching and only actually runs this eg once an hour on background (even with quota, this can be slow). maybe with some randomisation offset. i could also implement this caching/offset in the bash script itself ofcourse. maybe use a systemd user timer and write some small result to json container boolean ok value and an error_msg to display (and log)?