[BUG]: k8s dashboard doesn't work for resource view across all namespaces at once

[mecampbellsoup]

You can view all namespaces thanks to @mecampbellsoup --namespace-header PR to the k8s-dashboard. However, any non-namespaced request for a resource (across all namespaces) fails:

(you can verify there are plenty of pods in at least one of the namespaces: https://k8s-dashboard.staging.coreweave.com/#/pod?namespace=tenant-sta-43b6c8-vb)

The underlying API request is failing due to k8s RBAC:

GET https://k8s-dashboard.staging.coreweave.com/api/v1/pod/%20?itemsPerPage=10&page=1&sortBy=d,creationTimestamp
{
 "listMeta": {
  "totalItems": 0
 },
 "cumulativeMetrics": [
  {
   "dataPoints": [],
   "metricPoints": [],
   "metricName": "cpu/usage_rate",
   "aggregation": "sum"
  },
  {
   "dataPoints": [],
   "metricPoints": [],
   "metricName": "memory/usage",
   "aggregation": "sum"
  }
 ],
 "status": {
  "running": 0,
  "pending": 0,
  "failed": 0,
  "succeeded": 0,
  "unknown": 0,
  "terminating": 0
 },
 "pods": [],
 "errors": [
  {
   "ErrStatus": {
    "metadata": {},
    "status": "Failure",
    "message": "pods is forbidden: User \"vpatov+cloud-app-staging@coreweave.com\" cannot list resource \"pods\" in API group \"\" at the cluster scope",
    "reason": "Forbidden",
    "details": {
     "kind": "pods"
    },
    "code": 403
   }
  },
  {
   "ErrStatus": {
    "metadata": {},
    "status": "Failure",
    "message": "events is forbidden: User \"vpatov+cloud-app-staging@coreweave.com\" cannot list resource \"events\" in API group \"\" at the cluster scope",
    "reason": "Forbidden",
    "details": {
     "kind": "events"
    },
    "code": 403
   }
  }
 ]
}

slack thread (from locked channel): https://coreweave.slack.com/archives/C04SD385LCF/p1703094653832809?thread_ts=1703086728.890979&cid=C04SD385LCF

If we want to put the dashboard into production we will either have to:

• fix this (preferred)
• remove the "all namespaces" feature from the dashboard

[mecampbellsoup]

You can view all namespaces thanks to @mecampbellsoup --namespace-header PR to the k8s-dashboard. However, any non-namespaced request for a resource (across all namespaces) fails:

@vpatov I intentionally namespaced the link out in the cloud UI since this dashboard doesn't work with any "[foo resource] in all namespaces" requests.

Before I spend any more time on this I would like to know if there's appetite on the team for incorporating this or not.

[vpatov]

Before I spend any more time on this I would like to know if there's appetite on the team for incorporating this or not.

That's fair, agreed.

I intentionally namespaced the link out in the cloud UI since this dashboard doesn't work with any "[foo resource] in all namespaces" requests.

Right, but once you arrive in the UI you are able to select "All namespaces" which has broken/confusing behavior. So my stance is, if we want to incorporate the dashboard into the UI, then this has to be fixed, one way or another (either remove the feature by removing the button, or fix the feature).

[mecampbellsoup]

Right, but once you arrive in the UI you are able to select "All namespaces" which has broken/confusing behavior. So my stance is, if we want to incorporate the dashboard into the UI, then this has to be fixed, one way or another (either remove the feature by removing the button, or fix the feature).

Yep, this is why I want people to get their hands dirty before we commit to any more work - there may be a fair amount of gotchas related to our somewhat custom form of multi-tenancy.