[UI] Unable to see logs in UI

Torkolis commented 2 years ago

Rancher Server Setup

Rancher version: 2.6.5
Installation option (Docker install/Helm Chart): Helm
- If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2
Proxy/Cert Details: Trusted certificate in Ingress

Information about the Cluster

Kubernetes version: v1.23.7+rke2r2
Cluster Type (Local/Downstream): Local

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom): Admin

Describe the bug When clicking "show logs" for any pod, the window pops up but stays empty all the time. Console logs say that there is some TypeError.

To Reproduce

Change preference in the logs window to show e.g. last 1,000 lines.
log out from Rancher
close and reopen browser
try to show logs again.

Screenshots

Additional context Errors in browser console:

vue.runtime.esm.js:1888 TypeError: r.replace is not a function
    at C (opsgenie.vue?578b:153:1)
    at f.rangeOptions (opsgenie.vue?578b:116:9)
    at _n.get (vue.runtime.esm.js:4479:25)
    at _n.evaluate (vue.runtime.esm.js:4584:21)
    at f.rangeOptions (vue.runtime.esm.js:4836:17)
    at fn (rke2.svg:1:2097)
    at Object.r [as title] (vue.runtime.esm.js:2590:62)
    at f.r (rke.svg:1:159)
    at t._render (vue.runtime.esm.js:3548:22)
    at f.r (vue.runtime.esm.js:4066:21)
te @ vue.runtime.esm.js:1888
Zt @ vue.runtime.esm.js:1879
Jt @ vue.runtime.esm.js:1839
t._render @ vue.runtime.esm.js:3550
r @ vue.runtime.esm.js:4066
_n.get @ vue.runtime.esm.js:4479
_n @ vue.runtime.esm.js:4468
t @ vue.runtime.esm.js:4073
$n.$mount @ vue.runtime.esm.js:8415
init @ vue.runtime.esm.js:3118
(anonymous) @ vue.runtime.esm.js:5978
v @ vue.runtime.esm.js:5925
(anonymous) @ vue.runtime.esm.js:6477
t._update @ vue.runtime.esm.js:3945
r @ vue.runtime.esm.js:4066
_n.get @ vue.runtime.esm.js:4479
_n @ vue.runtime.esm.js:4468
t @ vue.runtime.esm.js:4073
$n.$mount @ vue.runtime.esm.js:8415
init @ vue.runtime.esm.js:3118
(anonymous) @ vue.runtime.esm.js:5978
v @ vue.runtime.esm.js:5925
_ @ vue.runtime.esm.js:6053
v @ vue.runtime.esm.js:5954
S @ vue.runtime.esm.js:6106
(anonymous) @ vue.runtime.esm.js:6233
j @ vue.runtime.esm.js:6319
(anonymous) @ vue.runtime.esm.js:6482
t._update @ vue.runtime.esm.js:3948
r @ vue.runtime.esm.js:4066
_n.get @ vue.runtime.esm.js:4479
_n.run @ vue.runtime.esm.js:4554
gn @ vue.runtime.esm.js:4310
(anonymous) @ vue.runtime.esm.js:1980
ie @ vue.runtime.esm.js:1906
Promise.then (async)
ee @ vue.runtime.esm.js:1933
se @ vue.runtime.esm.js:1990
(anonymous) @ vue.runtime.esm.js:4402
_n.update @ vue.runtime.esm.js:4544
_t.notify @ vue.runtime.esm.js:730
(anonymous) @ vue.runtime.esm.js:882
n.addObject @ array.ts:65
addTab @ app.19d73dab09087954b96b.js:2
(anonymous) @ vuex.esm.js:840
(anonymous) @ vuex.esm.js:462
(anonymous) @ vuex.esm.js:461
y._withCommit @ vuex.esm.js:620
y.commit @ vuex.esm.js:460
commit @ vuex.esm.js:405
r.commit @ vuex.esm.js:792
open @ app.19d73dab09087954b96b.js:2
(anonymous) @ vuex.esm.js:847
y.dispatch @ vuex.esm.js:512
dispatch @ vuex.esm.js:402
r.dispatch @ vuex.esm.js:775
value @ RoleDetailEdit.vue?9983:83
(anonymous) @ app.19d73dab09087954b96b.js:2
execute @ app.19d73dab09087954b96b.js:2
(anonymous) @ vuex.esm.js:847
y.dispatch @ vuex.esm.js:512
dispatch @ vuex.esm.js:402
execute @ PromptRemove.vue?ff5e:207
click @ browserMd5.js:3
Qt @ vue.runtime.esm.js:1854
n @ vue.runtime.esm.js:2179
c._wrapper @ vue.runtime.esm.js:6917
Show 5 more frames
socket.js:81 Socket connecting (id=32, url=wss://#####/api/v1/namespaces/#####/pods/#####/log...)
vue.runtime.esm.js:1888 TypeError: Cannot read properties of undefined (reading 'addEventListener')
    at opsgenie.vue?578b:222:1
    at d (runtime.js:45:40)
    at Generator._invoke (runtime.js:274:22)
    at O.forEach.t.<computed> [as next] (runtime.js:97:21)
    at r (asyncToGenerator.js:3:20)
    at l (asyncToGenerator.js:25:9)
te @ vue.runtime.esm.js:1888
Zt @ vue.runtime.esm.js:1879
Jt @ vue.runtime.esm.js:1839
(anonymous) @ vue.runtime.esm.js:1856
Promise.catch (async)
Qt @ vue.runtime.esm.js:1856
cn @ vue.runtime.esm.js:4219
insert @ vue.runtime.esm.js:3139
$ @ vue.runtime.esm.js:6346
(anonymous) @ vue.runtime.esm.js:6565
t._update @ vue.runtime.esm.js:3948
r @ vue.runtime.esm.js:4066
_n.get @ vue.runtime.esm.js:4479
_n.run @ vue.runtime.esm.js:4554
gn @ vue.runtime.esm.js:4310
(anonymous) @ vue.runtime.esm.js:1980
ie @ vue.runtime.esm.js:1906
Promise.then (async)
ee @ vue.runtime.esm.js:1933
se @ vue.runtime.esm.js:1990
(anonymous) @ vue.runtime.esm.js:4402
_n.update @ vue.runtime.esm.js:4544
_t.notify @ vue.runtime.esm.js:730
(anonymous) @ vue.runtime.esm.js:882
n.addObject @ array.ts:65
addTab @ app.19d73dab09087954b96b.js:2
(anonymous) @ vuex.esm.js:840
(anonymous) @ vuex.esm.js:462
(anonymous) @ vuex.esm.js:461
y._withCommit @ vuex.esm.js:620
y.commit @ vuex.esm.js:460
commit @ vuex.esm.js:405
r.commit @ vuex.esm.js:792
open @ app.19d73dab09087954b96b.js:2
(anonymous) @ vuex.esm.js:847
y.dispatch @ vuex.esm.js:512
dispatch @ vuex.esm.js:402
r.dispatch @ vuex.esm.js:775
value @ RoleDetailEdit.vue?9983:83
(anonymous) @ app.19d73dab09087954b96b.js:2
execute @ app.19d73dab09087954b96b.js:2
(anonymous) @ vuex.esm.js:847
y.dispatch @ vuex.esm.js:512
dispatch @ vuex.esm.js:402
execute @ PromptRemove.vue?ff5e:207
click @ browserMd5.js:3
Qt @ vue.runtime.esm.js:1854
n @ vue.runtime.esm.js:2179
c._wrapper @ vue.runtime.esm.js:6917
socket.js:313 Socket opened (state=connecting, id=32)
Taints.vue?e932:408 Ping [management] from dev

martialblog commented 2 years ago

Got a user with the exact same issue. I'm trying to get some details from their system, since I can't recreate the issue.

Rancher Server Setup

Rancher version: 2.6.5
Installation option: Helm
    If Helm Chart, Kubernetes Cluster and version: RKE v1.21.12
Proxy/Cert Details: Trusted certificate in Ingress

Information about the Cluster

Kubernetes version: v1.21.12+rke2r2 
Cluster Type (Local/Downstream): Downstream

User Information

What is the role of the user logged in?: Project Member

Torkolis commented 2 years ago

Oh wow, it is user specific... I just created a brand new user with Administrator Global Permissions and I can see logs on that user. Edit: Only once though, when closing logs and reopening then I get the TypeError: r.replace is not a function again.

Torkolis commented 2 years ago

I think I found a way to reproduce, looks the culprit is changing to show only 1,000 lines in settings:

switch to this, then logout, close and reopen browser, log back in and view logs again.

martialblog commented 2 years ago

Seems it now affects me as well. Same error:

vue.runtime.esm.js:1888 TypeError: r.replace is not a function
    at C (opsgenie.vue?578b:153:1)
    at f.rangeOptions (opsgenie.vue?578b:116:9)
    at _n.get (vue.runtime.esm.js:4479:25)
    at _n.evaluate (vue.runtime.esm.js:4584:21)
    at f.rangeOptions (vue.runtime.esm.js:4836:17)
    at fn (rke2.svg:1:2097)
    at Object.r [as title] (vue.runtime.esm.js:2590:62)
    at f.r (rke.svg:1:159)
    at t._render (vue.runtime.esm.js:3548:22)
    at f.r (vue.runtime.esm.js:4066:21)
te @ vue.runtime.esm.js:1888
Zt @ vue.runtime.esm.js:1879
Jt @ vue.runtime.esm.js:1839
t._render @ vue.runtime.esm.js:3550
r @ vue.runtime.esm.js:4066
_n.get @ vue.runtime.esm.js:4479
_n @ vue.runtime.esm.js:4468
t @ vue.runtime.esm.js:4073
$n.$mount @ vue.runtime.esm.js:8415
init @ vue.runtime.esm.js:3118
(anonymous) @ vue.runtime.esm.js:5978
v @ vue.runtime.esm.js:5925
(anonymous) @ vue.runtime.esm.js:6477
t._update @ vue.runtime.esm.js:3945
r @ vue.runtime.esm.js:4066
_n.get @ vue.runtime.esm.js:4479
_n @ vue.runtime.esm.js:4468
t @ vue.runtime.esm.js:4073
$n.$mount @ vue.runtime.esm.js:8415
init @ vue.runtime.esm.js:3118
(anonymous) @ vue.runtime.esm.js:5978
v @ vue.runtime.esm.js:5925
_ @ vue.runtime.esm.js:6053
v @ vue.runtime.esm.js:5954
S @ vue.runtime.esm.js:6106
(anonymous) @ vue.runtime.esm.js:6233
j @ vue.runtime.esm.js:6319
(anonymous) @ vue.runtime.esm.js:6482
t._update @ vue.runtime.esm.js:3948
r @ vue.runtime.esm.js:4066
_n.get @ vue.runtime.esm.js:4479
_n.run @ vue.runtime.esm.js:4554
gn @ vue.runtime.esm.js:4310
(anonymous) @ vue.runtime.esm.js:1980
ie @ vue.runtime.esm.js:1906
Promise.then (async)
ee @ vue.runtime.esm.js:1933
se @ vue.runtime.esm.js:1990
(anonymous) @ vue.runtime.esm.js:4402
_n.update @ vue.runtime.esm.js:4544
_t.notify @ vue.runtime.esm.js:730
(anonymous) @ vue.runtime.esm.js:882
n.addObject @ array.ts:65
addTab @ app.19d73dab09087954b96b.js:2
(anonymous) @ vuex.esm.js:840
(anonymous) @ vuex.esm.js:462
(anonymous) @ vuex.esm.js:461
y._withCommit @ vuex.esm.js:620
y.commit @ vuex.esm.js:460
commit @ vuex.esm.js:405
r.commit @ vuex.esm.js:792
open @ app.19d73dab09087954b96b.js:2
(anonymous) @ vuex.esm.js:847
y.dispatch @ vuex.esm.js:512
dispatch @ vuex.esm.js:402
r.dispatch @ vuex.esm.js:775
value @ RoleDetailEdit.vue?9983:83
openLogs @ syslog.vue?6ff4:87
(anonymous) @ app.19d73dab09087954b96b.js:2
execute @ app.19d73dab09087954b96b.js:2
(anonymous) @ vuex.esm.js:847
y.dispatch @ vuex.esm.js:512
dispatch @ vuex.esm.js:402
execute @ PromptRemove.vue?ff5e:207
click @ browserMd5.js:3
Qt @ vue.runtime.esm.js:1854
n @ vue.runtime.esm.js:2179
c._wrapper @ vue.runtime.esm.js:6917
Show 37 more frames
socket.js:81 Socket connecting (id=7, url=wss://rancher.***/k8s/clusters/c-m-fbg/api/v1/namespaces/.../log...)
vue.runtime.esm.js:1888 TypeError: Cannot read properties of undefined (reading 'addEventListener')
    at opsgenie.vue?578b:222:1
    at d (runtime.js:45:40)
    at Generator._invoke (runtime.js:274:22)
    at O.forEach.t.<computed> [as next] (runtime.js:97:21)
    at r (asyncToGenerator.js:3:20)
    at l (asyncToGenerator.js:25:9)

@Torkolis So what you are saying is, that once I change the "show the last" setting this error occurs?

Torkolis commented 2 years ago

I didn't test all of those settings, only the show 1,000 last lines and there it occurs. Default for me was show last 30 minutes and there is no problem with that, only when I change to 1,000 lines.

martialblog commented 2 years ago

I'm trying to figure out a workaround, but so far nothing :thinking: Browser cache clear, app restarted...

No logs on any clusters in the UI.

axeal commented 2 years ago

@martialblog @Torkolis Based upon my reproduction you can mitigate this issue by deleting the preferences.management.cattle.io custom resource that stores the "Show the last" log setting for the user within the Rancher local cluster.

Execute kubectl -n <user-id> delete preferences logs-range against the Rancher local cluster where is the Rancher user id of the affected user, e.g. u-nxm8r, user-7m7zc etc.
As the affected user reload Rancher UI in the browser and successfully view Pod logs

th0th commented 2 years ago

@martialblog @Torkolis Based upon my reproduction you can mitigate this issue by deleting the preferences.management.cattle.io custom resource that stores the "Show the last" log setting for the user within the Rancher local cluster.

Execute kubectl -n <user-id> delete preferences logs-range against the Rancher local cluster where is the Rancher user id of the affected user, e.g. u-nxm8r, user-7m7zc etc.

As the affected user reload Rancher UI in the browser and successfully view Pod logs

This worked for me, thanks a lot @axeal 💐

martialblog commented 2 years ago

@axeal Workaround looks good! Thanks

Torkolis commented 2 years ago

While it does work as a workaround this bug still needs to be fixed.

hbudde commented 2 years ago

Here a one liner if you have to do it for multiple users: eval "$(kubectl get preferences.management.cattle.io -A --field-selector metadata.name=logs-range -o jsonpath='{range .items[*]}{"kubectl -n "}{.metadata.namespace}{" delete preferences.management.cattle.io logs-range;"}{end}')"

Torkolis commented 2 years ago

Rancher 2.6.6 still has the bug.

niranjan94 commented 2 years ago

v2.6.6 still has the bug. And the workarounds mentioned above don't seem to be very reliable. (For me, It fixes it for a couple of minutes and somehow the issue comes back)

erSitzt commented 2 years ago

https://github.com/rancher/dashboard/issues/5984

If the last comment is correct this should be fixed in 2.6.7 And this is a duplicate..

richard-cox commented 2 years ago

Thanks for raising this issue, however as mentioned above it's a duplicate of https://github.com/rancher/dashboard/issues/5984 and will be resolved in 2.6.7. There's some info about workarounds in https://github.com/rancher/dashboard/issues/5984#issuecomment-1180191702

rancher / dashboard

[UI] Unable to see logs in UI #6453