Closed rbarreto closed 2 years ago
aculich
commented
2 years ago Hi @balajialg this is a joint D-Lab / Graduate Division workshop, so let me know how you want to configure it, whether we should use datahub.berkeley.edu or dlab.datahub.berkeley.edu ?
balajialg
commented
2 years ago @rbarreto Thanks for raising the request! Based on our RAM/CPU allocation policy, this should be good to go from our end. @aculich IMHO, Using dlab.datahub.berkeley.edu seems like a better option as it is contextualized to your needs. There are a couple of options that can be pursued here,
@felder Do you have any suggestions on the way forward here?
aculich
commented
2 years ago @balajialg thanks! We'd like to make sure we can finalize this request fully early this week so we can test soon.
@felder @balajialg I agree that we should probably start moving to dlab.datahub.berkeley.edu as our primary environment for our workshops. How do we make sure that:
1) we first get our image to be sync'd with the standard datahub image? 2) then make a request for additional packages added by default? cc: @emgrabow9
For the Fall semester, I'd like all of our D-Lab workshops to switch over to using the dlab.datahub with a custom set of R & Python packages, however we otherwise probably want to keep pace with the semesterly or yearly releases on the main datahub so there are the fewest differences possible. We also want to track your upstream LTR and then just add extra packages.
felder
commented
2 years ago @aculich dlab already uses datahub's image.
New package requests should be ticketed using https://github.com/berkeley-dsep-infra/datahub/issues/new?assignees=felder%2Cbalajialg&labels=package-request&template=package_request.yml&title=Request+python+package+X+for+class+Y
It may be wise to review the existing packages deployed to datahub to see if there will be conflicts with your requests.
https://github.com/berkeley-dsep-infra/datahub/blob/prod/deployments/datahub/images/default/environment.yml https://github.com/berkeley-dsep-infra/datahub/blob/prod/deployments/datahub/images/default/requirements.txt https://github.com/berkeley-dsep-infra/datahub/tree/prod/deployments/datahub/images/default/r-packages
balajialg
commented
2 years ago @rbarreto @aculich Looping back to my earlier comment regarding the increase in RAM - https://github.com/berkeley-dsep-infra/datahub/issues/3463#issuecomment-1159305322. Do you foresee instructors/students using the D-Lab hub during the summer who are not affiliated with this course AND do not need the 8 GB RAM increase? This input will guide our implementation decision with regards to increasing RAM for all users vs only for the users affiliated with this course.
rbarreto
commented
2 years ago @rbarreto @aculich Looping back to my earlier comment regarding the increase in RAM - #3463 (comment). Do you foresee instructors/students using the D-Lab hub during the summer who are not affiliated with this course AND do not need the 8 GB RAM increase? This input will guide our implementation decision with regards to increasing RAM for all users vs only for the users part of this course.
That's a good question! I think Aaron @aculich may have more background to provide there. This is a new, longer, and more in depth workshop relative to other workshops in the past, so at min we'd need the 8 GB RAM increase for the students in the course. I'll leave it up to Aaron to decide if the increase is needed for all users taking workshops at the D-Lab
balajialg
commented
2 years ago That sounds good @rbarreto! Please do note - We might need all the enrolled students' Calnet IDs if we decide (based on @aculich's recommendation) to not upgrade RAM for all users in the hub. @aculich - Please advise on the way forward.
balajialg
commented
2 years ago @rbarreto Just to avoid any further delay at our end and provide you with ample time to test the notebooks, Is it possible to share the calnet id/email id of all the students who enrolled in your course? You can share it via email for privacy reasons.
rbarreto
commented
2 years ago @rbarreto Just to avoid any further delay at our end and provide you with ample time to test the notebooks, Is it possible to share the calnet id/email id of all the students who enrolled in your course? You can share it via email for privacy reasons.
Hi - thanks for following up. I actually don't have the calnet id / email id of all of the students in the course, but I can check in with Kara and Aaron and get back to you by tomorrow morning.
yuvipanda
commented
2 years ago @balajialg let's just give everyone on the dlab hub 8G of RAM, and if it becomes too much we can re-evaluate and restrict that somehow. Otherwise I think it gets too cumbersome for us all.
aculich
commented
2 years ago @yuvipanda @balajialg @rbarreto just sent you email with the names and email addresses of students and instructors/helpers who will need increased limits for: D-Lab's Data Science for Social Justice Workshop, July 5 - August 16, 2022.
However, I prefer @yuvipanda 's approach for now which is to enable it for everyone on the D-Lab datahub, because anyone (grad or undergrad student) may run the notebook over the summer or in the future.
It would be great if we could some how include resource hints in a notebook that would tell you how many resources it needs, so that when anyone starts up a notebook they could be given a soft quota bump without asking for it just by virtue of logging into the dlab-datahub and starting a notebook that needs more resources.
felder
commented
2 years ago @aculich I would think such an approach would be difficult because resources are allocated when the pod is created. By the time the pod loads a notebook, it's too late. Additionally there would be no guarantee the node the pod was running on could provide the additional resources requested. I imagine a scenario where a pod could ask for a memory upgrade (not even sure if it's possible to modify the RAM allocation after the fact) and requested say 4GB of additional ram but only 1GB of additional RAM was available on the node the pod was running on.
So I feel it's likely any sort of idea like this would require something to examine the proposed notebook before the pod is launched.
balajialg
commented
2 years ago Thanks all! Closing this GitHub issue as the actual request was satisfied (including providing admin access). @aculich If you would like to continue the conversation about resource hints then please reopen this issue or open a new enhancement issue!
aculich
commented
2 years ago Doesn't seem like admin access is enabled for me: https://dlab.datahub.berkeley.edu/hub/admin
403 : Forbidden Action is not authorized with current scopes; requires any of [admin:users]
On Tue, Jun 28, 2022 at 3:50 PM Balaji Alwar @.***> wrote:
Thanks all! Closing this GitHub issue https://github.com/berkeley-dsep-infra/datahub/issues/new?assignees=balajialg&labels=type%3A+enhancement&template=featurerequest.md as the actual request got satisfied (including providing admin access). @aculich https://github.com/aculich If you would like to continue the conversation about resource hints then please reopen this issue or open a new enhancement issue https://github.com/berkeley-dsep-infra/datahub/issues/new?assignees=balajialg&labels=type%3A+enhancement&template=featurerequest.md !
— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3463#issuecomment-1169364979, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA5BNNXVCPUDPZKZHBH27TVRN6URANCNFSM5ZDKZPIA . You are receiving this because you were mentioned.Message ID: @.***>
aculich
commented
2 years ago 
balajialg
commented
2 years ago @aculich That's weird! Can you try now?
aculich
commented
2 years ago Hi Balaji, great! It works now for me as admin!
On Tue, Jun 28, 2022 at 4:25 PM Balaji Alwar @.***> wrote:
@aculich https://github.com/aculich That's weird! Can you try now?
— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3463#issuecomment-1169382345, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA5BNOT3WFQK3KIV5XI463VROCVVANCNFSM5ZDKZPIA . You are receiving this because you were mentioned.Message ID: @.***>
balajialg
commented
2 years ago @aculich Hope the classes are going great. Please feel free to reopen if there are any issues with the hub.
Which hub do you want more RAM on?
For example: datahub, r hub, eecs hub, etc
DataHub
Which class is this request for?
You must include a link to your class for the correct semester in https://classes.berkeley.edu.
This is the workshop: https://grad.berkeley.edu/graduate-diversity/current-grad-students/data-science-social-justice-workshop/
How many students do you expect in this class?
An approximate number would do.
We expect about 25 students -- final count.
How much RAM does this class need?
The default is 1GB of RAM per student.
8 GB of RAM per student is needed.
Why does this class need this much RAM?
A short justification for this resource request.
Specifically, we are hosting the Data Science and Social Justice workshop this summer. We want to provide students with multiple alternatives for running their Jupyter Notebooks. One option is for them to run it on their local machine -- but for equity reasons we also need to provide the DataHub option, in case their personal computers do not suffice.
By when (MM/DD) do you want this request to be fulfilled?
This will help us with prioritization of this request.
July 5th is the start of the class
Any additional information we should know about?