search

nerc-project / operations

Issues related to the operation of the NERC OpenShift environment
1 stars 0 forks source link

Reporting for Accounting and Invoices: Accounting for resources used by the notebooks from three classes #374

Open DanNiESh opened 5 months ago

DanNiESh commented 5 months ago

to set these 3 issues in perspective

Details

Three classes are operating within the same namespace. This setup presents a challenge in accounting resources usage and billing. We need to figure out how to distinguish and report resource utilization by each class.

this issues follows https://github.com/nerc-project/operations/issues/191 (merging) @dystewart and Harshad from RHOAI team are in disussion: new features for default namespace naming maybe available

schwesig commented 4 months ago

Things to clarify

DanNiESh commented 4 months ago

Things to clarify

  • [ ] How is accounting done today? (Dylan, Kristi, ?)
  • [ ] Why is is in one namespace? (there were discussions and this was the solutions, but ask for details)
  • [ ] RH Cost Management?
  • [ ] Labels? Set by RHOAI, automatic, automagic, script vs list?

Each class has its own namespace which is mapping from coldfront project. Previously, when students launched jupyter workbench via data science project in their designated namespace, they got access to all notebook instances within that namespace. This level of access allowed them to view, login and stop others notebooks which was a significant privacy concern. To address this issue, we enabled the rhods-notebooks namespace, and directed students to use the Jupyter tile that belongs to rhods-notebooks namespace. In this way, each student has access exclusively to their individual notebooks. This solution led to classes sharing resources in rhods-notebooks namespace.

schwesig commented 4 months ago

https://github.com/nerc-project/operations/issues/191

schwesig commented 2 months ago

https://github.com/nerc-project/operations/issues/191

Billing using our current model relies on measuring usage on a per namespace basis. Given that we can only have one rhods instance installed per cluster, all the rhods namespaces must be shared by all the rhods users. This means that billing becomes complicated with say: multiple classes using the same rhods instance, because we can't trivially separate the usage per class.

There are a few things to consider here;

  1. @gagansk is reaching out to the rhods engineers to raise this concern and seek a possible workaround from the devs
  2. Maybe there is a way we can tag resources being deployed based on class membership, that way there is some way to distinguish the workloads

Based on discussion today with @jtriley @Milstein and Wayne, this is a blocker for offering rhods in production

schwesig commented 2 months ago

https://github.com/nerc-project/operations/issues/191#issuecomment-1671241381

While I agree this is an issue, it should not be a blocker for this fall. The classes that will be using RHODS are all in the same department, so a single bill for that department for RHODS use should be an acceptable workaround. A brief look at tagging suggests that we can implement a solution even without RHODS product support, but not before classes start at the beginning of Sept. Based on previous input from Orran, billing by department is an acceptable temporary solution.

joachimweyl commented 3 weeks ago

@schwesig would it be helpful to break this into two issues one where we bill to one department and a new one to track how to deal with it once we don't have just one department?

schwesig commented 3 weeks ago

@joachimweyl That is a good idea. I will do.

msdisme commented 2 weeks ago

@schwesig just responding to an old comment- it is a blocker for this fall because we need to understand the costs from last semester to set them accurately for the fall. Perhaps that is a different issue which I have not found yet.

msdisme commented 2 weeks ago

Also, if this is the right issue for the "what was the cost per class last semester" then it should be prioritized higher

msdisme commented 2 weeks ago

move to current sprint (jun 19,2024)- for discussion in spirnt planning