search

Netflix / flamescope

FlameScope is a visualization tool for exploring different time ranges as Flame Graphs.
Apache License 2.0
3k stars 168 forks source link

cache / preprocess heatmap response data #58

Open iori-yja opened 5 years ago

iori-yja commented 5 years ago

Every time people access the flamescope's heatmap, the server processes the perf output data which makes the user awaited. So I am considering to cache preprocessed or first-returned data.

Structure of the response is quite simple;

{
    "columns": [0, 1, .....],
    "rows": [980, 960....],
    "values": [ [2, 2, ...], [...], [...], ....],
    "maxvalue": 16,
}

The most quick hack to cache it is storing in sessionStorage in local browser. Better idea is to save the data as a metadata beside actual data (In that case, we need to decide naming convention for metadata). Other idea is to store data in DB.

I am going to implement a cache anyway, but I want to ask this before starting, which is, what kind of cache is easier to accept for you?.

spiermar commented 5 years ago

When you mention cache, is that from going back and forth between the heatmap and flame graph visualization, or changing profiles? Browser session storage might work, but it's very limited. A preprocessed output might be stored next to the actual profile files, in disk. I would avoid adding an external dependency, like a database, since one of the goals is to have a solution that is very simple to deploy.

spiermar commented 5 years ago

Preprocess in the other hand is a different problem. I definitely wouldn't do it in the browser session storage, given the size limits, but might be Ok to have a task to preprocess all heatmaps and store them on disk.

iori-yja commented 5 years ago

My scope is only on the processing time when user is "going back and forth between the heatmap and flame graph" or just opening one of them.

I agree with you on both points of avoiding using databases and browser storages for each reason. I will move on to this in a few weeks. Is it OK to be default behavior to store preprocessed data next to the profile? I don't want to have a slow start-up time, so it should be to keep the first response rather than preprocessing all data files.