Getting per interaction metrics and stats about Panel servers

nishikantparmariam commented 1 year ago

Is your feature request related to a problem? Please describe.

Our use-case is to log and store how Panel servers are performing (cpu, memory), stats (like total and active sessions), how much time an interaction took e.g. when a button is clicked in frontend which triggers some callback in Python, we would like to log how much time it took and do this preferably for each such interaction/widget.

Describe the solution you'd like

I'd prefer a solution which would allow me to hook into Panel and intercept requests to time interactions for logging.

Describe alternatives you've considered

Patching

This request is similar to https://discourse.holoviz.org/t/performance-metrics-logging-with-panel/4355 and I've gone through the recommendations there. I patched these two functions in a starter script (that will be run before panel serve) - https://github.com/holoviz/panel/blob/f7f9ea921e9f62890fea2d4f6039c05ce1415fd9/panel/reactive.py#L408 and https://github.com/holoviz/panel/blob/f7f9ea921e9f62890fea2d4f6039c05ce1415fd9/panel/reactive.py#L441 like this -

from panel.reactive import Syncable

original_change_event = Syncable._change_event

def _change_event(*args, **kwargs):
  start = time.time()
  res = original_change_event(*args, **kwargs)
  print(f'End change event : {time.time() - start}s')
  # Do logging stuff etc.
  return res

Syncable._change_event = _change_event

Using this I was able to log timings for button clicks, select etc. and this also included the time for any registered callback to complete e.g. on_click on Button. Ultimately I'd like to standardize these kind of logs and store them in a fixed format.

Questions -

Are these methods enough if I want to intercept an interaction or request?
Should I be looking into some other methods?
Is there a cleaner way to do this rather than patching private functions e.g. if Panel can accept callback similar to pn.state.on_session_created.

Admin Page

Admin page seems great here but it becomes problematic when using nginx as reverse proxy with multiple Panel servers architecture - which we are using. This makes the /admin request go to one of the Panel servers every time you open it and stats seem not that meaningful. Is there a way I can tell admin page to show combined stats across Panel servers?

I was able to mimic similar problem with num-procs in Panel where admin page shows stats for different Panel processes (I think?) after multiple refreshes (notice total and active sessions) -

num-procs-panel

For my use-case, other than capturing start & end logs and calculating timings from them (similar to how admin page's timeline is calculated), are there other solutions here?

Additional context

If it helps, we can control how panel serve is invoked by our users e.g. passing some args for all users is feasible.

MarcSkovMadsen commented 1 year ago

+1

ability to add middleware
ability to collect logs/ metrics across pods, processes etc.

nishikantparmariam commented 1 year ago

I'll get back here with a proposal.

nishikantparmariam commented 1 year ago

I've raised a draft PR to get initial feedback - https://github.com/holoviz/panel/pull/5273. The proposal is to simply expose middleware interfaces that users can implement themselves, namely BokehEventMiddleware and PropertyChangeEventMiddleware (better names can be chosen). Users should then pass these using pn.state.add_bokeh_event_middleware(my_middleware).

I would also like to propose that there should be a way to give a human-readable name to each component. This would be very useful for logging purposes e.g. consider a Panel app has many buttons or select widgets whose data I want to log using these middlewares, giving business specific name to these components would really help to distinguish and visualize logs later.

For my use-case, for getting stats which Panel's admin page gives I found this alternative while exploring the codebase here. This will work if panel is launched using --session-history -1 :

def log_server_details():
    # get process stats like memory, CPU
    print(pn.state.session_info) # session stats - rendering time, live, total sessions etc.

pn.state.add_periodic_callback(log_server_details, 5000)

These will be logged for individual servers and then I should combine them as needed.

Would be happy to hear other thoughts/ideas/feedback!

nishikantparmariam commented 1 year ago

@philippjfr a quick question - is there a way to get a list of all current users at any point in Panel?

philippjfr commented 1 year ago

@philippjfr a quick question - is there a way to get a list of all current users at any point in Panel?

Should be able to get the sessions but user information is only really available when you have an OAuth provider configured. What exactly would you want to do?

nishikantparmariam commented 1 year ago

only really available when you have an OAuth provider configured.

I am using plugins approach and it exposes a custom OAuth handler.

What exactly would you want to do?

As a Panel app developer, I wanted to know all the current users connected and possibly log them. I guess in my OAuth handler, I should store who logs in and store it somewhere?

holoviz / panel