nil pointer dereference in prometheus/histogram.go

[agnivade]

We have received the following stack trace from Sentry.

runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "github.com/prometheus/client_golang@v1.11.0/prometheus/histogram.go", line 177, in newHistogram
  File "github.com/prometheus/client_golang@v1.11.0/prometheus/histogram.go", line 431, in NewHistogramVec.func1
  File "github.com/prometheus/client_golang@v1.11.0/prometheus/vec.go", line 428, in (*metricMap).getOrCreateMetricWithLabels
  File "github.com/prometheus/client_golang@v1.11.0/prometheus/vec.go", line 222, in (*MetricVec).GetMetricWith
  File "github.com/prometheus/client_golang@v1.11.0/prometheus/histogram.go", line 481, in (*HistogramVec).GetMetricWith
  File "github.com/prometheus/client_golang@v1.11.0/prometheus/histogram.go", line 504, in (*HistogramVec).With
  File "github.com/mattermost/mattermost-server/v6/enterprise/metrics/metrics.go", line 958, in (*MetricsInterfaceImpl).ObserveStoreMethodDuration
  File "github.com/mattermost/mattermost-server/v6/store/timerlayer/timerlayer.go", line 6007, in (*TimerLayerRoleStore).ChannelHigherScopedPermissions
  File "github.com/mattermost/mattermost-server/v6/app/role.go", line 93, in (*Server).mergeChannelHigherScopedPermissions
  File "github.com/mattermost/mattermost-server/v6/app/role.go", line 112, in (*App).mergeChannelHigherScopedPermissions
  File "github.com/mattermost/mattermost-server/v6/app/role.go", line 70, in (*App).GetRolesByNames
  File "github.com/mattermost/mattermost-server/v6/app/authorization.go", line 223, in (*App).RolesGrantPermission
  File "github.com/mattermost/mattermost-server/v6/app/authorization.go", line 167, in (*App).HasPermissionToTeam
  File "github.com/mattermost/mattermost-server/v6/app/authorization.go", line 190, in (*App).HasPermissionToChannel
  File "github.com/mattermost/mattermost-server/v6/app/webhook.go", line 778, in (*App).HandleIncomingWebhook
  File "github.com/mattermost/mattermost-server/v6/web/webhook.go", line 90, in incomingWebhook
  File "github.com/mattermost/mattermost-server/v6/web/handlers.go", line 298, in Handler.ServeHTTP
  File "github.com/gorilla/mux@v1.8.0/mux.go", line 210, in (*Router).ServeHTTP
  File "net/http/server.go", line 2049, in HandlerFunc.ServeHTTP
  File "net/http/server.go", line 2867, in serverHandler.ServeHTTP
  File "net/http/server.go", line 1932, in (*conn).serve

The code is from this commit: https://github.com/mattermost/mattermost-server/tree/0077fb2b04feeb5425dced76b633393b8a4ea2d0

Library version is at 1.11.0.

Let me know if you need anything else.

[yolossn]

@agnivade Looks like the implementation of (*MetricsInterfaceImpl).ObserveStoreMethodDuration is not publicly available. Can you share the implementation to reproduce the issue ?

I am getting a 404 on this URL. https://github.com/mattermost/mattermost-server/blob/0077fb2b04feeb5425dced76b633393b8a4ea2d0/enterprise/metrics/metrics.go

[agnivade]

Hi @yolossn - Unfortunately, I cannot share the code due to legal reasons. It belongs in our enterprise repo. But it's just a one line method which goes something like histogram.With(..).Observe(...), where histogram is a *prometheus.HistogramVec.

[keremgocen]

Hey @agnivade, can you share an example or at least the types of parameters you’re passing to the histogram builder? I could not find in client_golang version 11’s newHistogram code where it could possibly end up being a nil reference.

If this issue is not valid anymore it might as well be stale, it was created long time ago.

[agnivade]

It is basically like <*prometheus.HistogramVec>.With(prometheus.Labels{"string": stringVal, "string": stringVal}).Observe(float64)

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for the last 3 months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next 4 weeks, this issue will be closed (we can always reopen an issue if we need!).

[sarthaksarthak9]

@bwplotka is this issue still up ?

[bwplotka]

Seems so, but ideally we have test reproducing this from the author of this issue.

Thanks!