search

MSD-LIVE / issues

0 stars 0 forks source link

figure out why and fix notebook launch failures from workshop on 10/4 #140

Closed zguillen closed 2 months ago

zguillen commented 1 year ago

New error in the log that I haven't seen before: [W 2023-10-04 23:25:26.258 JupyterHub web:1852] 404 GET /hub/user/a0574bace99140eb97e7202c41817869/api/contents?content=1&1696461925814 (::ffff:10.15.137.188): No access to resources or resources not found

This is the notebook everyone was trying to run: https://uc-ebook.org/docs/html/A2_Jupyter_Notebooks.html#sobol-sa-tutorial

Chris said they did this a few months ago with about 30 people and didn't have any trouble. There's about 100 people in this meeting now.

when everyone stopped trying and only the presenter tried to open the notebook it worked seamlessly.

the logs had all network related errors. Maybe the libraries on the hub are throttling the connections for some reason.

Might be PNNL's proxy: https://github.com/jupyterhub/jupyterhub/issues/3017

we might have experienced Fargate throttling quotas. To spawn the notebooks, we are using the ECS API, and I found this on the AWS docs: In addition, Fargate limits the request rate when launching tasks using the Amazon ECS RunTask API using a separate quota. Fargate limits Amazon ECS RunTask API requests for each AWS account on a per-Region basis. Each request that you make removes one token from the bucket. We do this to help the performance of the service, and to ensure fair usage for all Fargate customers. API calls are subject to the request quotas whether they originate from the Amazon Elastic Container Service console, a command line tool, or a third-party application. The rate quota for calls to the Amazon ECS RunTask API is 20 calls per second (burst and sustained).

[10:15 AM] Lansing, Carina S

If 100 people all tried to start a notebook at the same time, maybe. There weren't a lot of notebook's spawned when I checked, which meant that they didn't start at all. Which is different then them starting and getting orphaned because the connection was broken. We'll have to do some testing to be sure.

These notebooks are being used in 100+ person college classes for what it’s worth. We should figure out how to test at scale without waiting for it to break again.

clansing commented 2 months ago

This was caused by running out of IP addresses