search

berkeley-dsep-infra / datahub

JupyterHubs for use by Berkeley enrolled students
https://docs.datahub.berkeley.edu
BSD 3-Clause "New" or "Revised" License
63 stars 39 forks source link

Student Terminal Not Loading #5886

Closed paulamarm closed 2 months ago

paulamarm commented 2 months ago

Bug description

The following is returned when my student opens the terminal in datahub:

Error: Status code 503 returned by RStudio Server when executing 'process_get_buffer_chunk'

I tried to sweep it which returned a 403 error code, and then I tried to open a new terminal tab which also returned 403.

Attempting to open the terminal options results in a 403 crode which makes datahub refresh itself.

I'm not sure about the student but on my admin end, i'm using google chrome with no browser extension blockers.

It's been giving them trouble and preventing them from submitting their assignments to gradescope all summer session long I think.

As of right now, I've identified two students with this issue, and they're both in the publichealth.datahub.berkeley server.

Environment & setup

How to reproduce

  1. Go to '(https://publichealth.datahub.berkeley.edu/user)'
  2. Change 'user' to please email me paulamarquez@berkeley.edu for the specific users to reproduce!
  3. Run 'Terminal'
  4. See error
ryanlovett commented 2 months ago

Hi @paulamarm , I've asked for some information the the UC Tech slack, ucb-datahubs channel.

shaneknapp commented 2 months ago

@paulamarm i'm looking in to this now. i can reproduce on their servers using both chrome and firefox, so it's not a browser issue. once i figure out what going on, i'll update this issue.

paulamarm commented 2 months ago

Thank you!

shaneknapp commented 2 months ago

hey @paulamarm, we spent a bunch of time looking in to this yesterday and it appears that it was something done by the users to their Rstudio settings. we found a bunch of garbage in their $HOME/.local/share/rstudio/projects/none/console07 directories, and after we cleaned that up their terminals should now properly work.

thanks @felder for figuring this out!

felder commented 2 months ago

@paulamarm specifically, the directory shane mentioned contains the terminal output history/buffer. In the particular cases investigated, one person had about 1GB of data in the output history and the other about 2GB of data. On server startup, rstudio attempted to load this info which caused timeouts as well as resource exhaustion (kernel ran out of free memory).

Of particular interest was that in both cases, the vast majority of the data consisted of lines with nothing but the letter 'y'. When I filtered those out and looked at the rest of the output (only a couple of kb with the 'y's removed), it was them submitting their assignment via the turn_in.py script.

So I'm not sure if the script itself has an issue that manifests in certain situations, or if their program had an issue where it simply outputted 'y' indefinitely until their server crashed.

Either way, removing this terminal history/buffers resulted in normal server startup for both cases investigated.

ryanlovett commented 2 months ago

@felder Fwiw, lines containing nothing but y can be generated by yes.

felder commented 2 months ago

@ryanlovett That's very interesting, because the terminal output suggests they may have inadvertently ran yes while trying to answer the questions turn_in.py asks. It forces you to type "yes" or "no" as opposed to accepting "y"...

Yes! :P If I run "yes" it outputs lines consisting of 'y' until it's stopped. Based off that, that seems exactly what happened here. The students accidentally ran "yes" and it continued to output 'y', with that output being stored in their output history, until their server crashed.

felder commented 2 months ago 
YES(1)                                          General Commands Manual                                         YES(1)

NAME
     yes – be repetitively affirmative

SYNOPSIS
     yes [expletive]

DESCRIPTION
     The yes utility outputs expletive, or, by default, “y”, forever.

SEE ALSO
     jot(1), seq(1)

HISTORY
     The yes command appeared in Version 7 AT&T UNIX.

macOS 14.5                                           June 4, 2014                                           macOS 14.5
felder commented 2 months ago 
rstudio@jupyter-REMOVED:~/ph142-su24/lab/lab06$ cd; cd ph142-su24/lab/lab06; pyth
cd; cd ph142-su24/lab/lab06; python3 turn_in.py

------------------
No lab06.pdf found; please make sure that you knitted your Rmd!
------------------

rstudio@jupyter-REMOVED:~/ph142-su24/lab/lab06$ yes

Many many many lines consisting of "y" removed...

rstudio@jupyter-REMOVED:~/ph142-su24/lab/lab06$ rstu
dio@jupyter-REMOVED:~/ph142-su24/lab/lab06$ rstudio@jupyter-REMOVED:~/ph142-su24
/lab/lab06$
ryanlovett commented 2 months ago

@felder Makes sense! If this problem were to become widespread, we could alias yes in a new file /etc/profile.d/ph142-yes-no.sh so that it runs true or echo something harmless instead. Alternatively the turn_in.py script could be changed to prompt for something besides yes, like Y.

felder commented 2 months ago 
rstudio@jupyter-REMOVED:~$ cd; cd ph142-su24/lab/lab01; python3 turn_in.py
Have you knitted your file after completing the lab? (Yes/No): Y
Invalid input; please type 'Yes' or 'No'
rstudio@jupyter-REMOVED:~/ph142-su24/lab/lab01$ yes

Many many many lines of "y" removed...

rstudio@jupyter-REMOVED:~/ph142-su24/lab/lab01$

@paulamarm if turn_in.py quits after indicating that "yes" or "no" needs to be supplied as a opposed to "y" or "n" I can see how this could happen accidentally pretty often. If turn_in.py does indeed behave that way, I'd suggest it not quit, but instead reprompt.

paulamarm commented 2 months ago

Thank you for the help! I'll definitely be changing the turnitin.py file to reprompt in the future to prevent this from happening again- it's so weird to me that this is the first semester it's an issue for students, though!