ETS node view

[mkacper]

Guys,

I've been working on the ets node view for a few days and I've just finished the first prototype. It shows all the ETS tables present on a particular node. Each table is connected to its owner's process. Tables which "can cause problems" are coloured red and yellow. Tables to be coloured are choosen based on the size of the memory allocated for a particular table. The five biggest tables are coloured red and the next five are coloured yellow.

Here are some screenshots:

Here is the code: https://github.com/mkacper/erlangpl/tree/ets-node-view-proto

I encourage you to build this branch and check it out. Please let me know what you think :)

Genereal idea for this view is to make it possible to analyse ETS tables from different perspectives and help users to find suspicious tables. I think it would be nice to add some kind of switch in the UI to allow users change the view's mode (change what we analyse in terms of ETS e.g. memory usage or size of the table etc.).

What to analyse:

• Allocated memory usage - already prototyped. Shows the amount of the memory allocated for a particular table. I talked to @michalslaski about that and we agreed that it would be propably better to show this kind of information in a sorted table instead on a Vizceral graph and using colours. So why did I do this prototype? Because I talked to some Erlangers and they said that ETS memory size can often provide us with the information that something is wrong. I decided to show this data on a Vizceral graph because it gives us a nice context (especially if we could double click on a suspicious table and see who and how comunicates with the table) and I have a Vizceral at hand :) What do you think about that?

• Memory utilization/fragmentation - I've mentioned about it during ErlangPL meetup. I have some doubts about this metric. Again, I've talk to some Erlangers and none of them (excluded @michalslaski :)) have never met this kind of problem with ETS tables. Moreover, getting this metric seems to be not so easy. I don't know if it's worth the effort at this stage.

• Size - numbers of rows in a table. Some people say this metric can be helpful. Again, does it make sense to show it on Vizceral graph?

• Lock statistics - statistics about locks protecting ETS tables (I don't know what exactly yet). Need compiling ErlangVM with --enable-lock-counter flag (unless you use OTP 20). I heard that even if the lock profiler is disabled it degradates ErlangVM perfromance for about 30%.

• Mean access time - how much time do you need (on average) to access the table.

The most important questions (in my opinion) are:

• Which of the aformentioned ideas make sense?
• How should we visualise patricular metrics?
• What else could we measure (in terms of ETS performance)?

I would be very grateful for your feedback/suggestions/ideas :)

[mkacper]

I've just implemented size mode of working for ets node view. Build the ets-node-view-proto branch on my erlangpl fork to see how it works. If you want to switch between size and memory modes simply edit this line.

[arkgil]

First of all - great job!

IMO the most obvious (and probably most desired, but I'm only speaking for myself) metrics are memory usage, size and mean access time (maybe for reads and writes separately?).

I have concerns about colouring top five tables red etc. It might be useful if those tables have similar sizes, but if one of them is exceptionally big it wouldn't stand out, thus would remain unnoticed. Maybe in such cases table view could be more appropriate, but I am not sure how to represent that if we are to keep the graph.

Do you plan on implementing a switch in the UI allowing to change what metric we observe?

[mkacper]

@arkgil I'm glad you like it :)

It's a good idea to divide the access time into reads and inserts.

You're right - the way of presenting "problematic" ETS tables is not so smart now. We could try to present this kind of information in a table but how to combine it with a Vizceral graph? We use the top-level Vizceral's view and (IMO) it would be nice to use the deepest one for visualising which processes and how they communicate with a particular table. I also think that the lowest view should be showed when you click on a particular ETS table. Maybe @Baransu knows if we can combine "table view" as the middle one with the Vizeral graph?

If we want to keep using Vizeral for visualising the aformentioned metrics what criteria should we use to decide which tables should be coloured? Maybe we should present the metric on the graph in a different way?

When it comes to a switch in the UI - I don't know yet, maybe @Baransu can help us with that? I'm sure that will be a piece of cake for him :wink:

[arkgil]

Maybe there is no need to render the view in its current form at all (ETS tabled connected to owning processes). A simple, sortable table with memory usage, owning process etc. would be sufficient and IMO more readable. Clicking on a row of the table would open the next level view.

Unfortunately I don't know much about Vizceral and if it is possible, so my ideas may sound ignorant (sorry if that's the case).

[mkacper]

Do you mean we could try table view as a second view? When you click on a particular node in ets cluster view an ordered table will be showed and then, when you click on a particular row the third Vizceral's view will be showed? Am I understand you correctly? :)

[Hajto]

Also for learning purposes showing n first records in table could be nice. People could look how stuff exactly work for example how digraph works.

[arkgil]

@mkacper yes, that's what I meant 🙂

[mkacper]

So let's try this approach out :smiley: I've been working on measuring access time to a table so I'll try present this data (memory usage and size too) in a table. Maybe it would be separated from a Vizceral graph on the first try.

[mkacper]

I was playing with the Vizceral's Events a little bit and it seems that it's probably possible to put the table view between the ets cluster view and ets detailed view.

Vizceral provides us with the viewChanged event so we can detect when and which node from the ets cluster view is clicked and then can show the table contains ETS tables from the clicked node.

Another useful thing in this case is the Vizceral's setView. Using it we can set the current Vizceral view to the one we want to show. If a user clicks on a particular ets table in the table view we can set the Vizceral view to ets detailed view for the clicked ets table.

Here are the links to the Vizceral docs:

• https://github.com/Netflix/vizceral/blob/master/USAGE.md#viewchanged
• https://github.com/Netflix/vizceral/blob/master/USAGE.md#setview

I'm not the Vizceral and JS expert but I think it could work. What do you think @Baransu? :smiley:

[mkacper]

@Baransu and I have implemented the very first prototype of ets table view. Here is the screenshot:

Each row in the table shows basic information about a particular ETS and, what's more interesting, statistics (min, max, percentile) about access time to the ETS table (inserts and lookups). The table shows infromation in recent 5 seconds.

If you want to try the view out here is the code: https://github.com/mkacper/erlangpl/tree/ets-node-view-tab The table is visible in the side panel in ETS section. You have to click on the table's left edge and holding left mouse button switch left in order to make the table bigger.

You have to enable particular functions calls tracing to see access time stats. To do that run on the erlangpl node: epl_tracer:enable_ets_call_tracing(ObservedNode).

To generate some inserts and lookups run on the ObservedNode:

ets:new(test, [named_table, public]).
ets:new(test2, [named_table, public]).
[spawn(fun() -> [ets:insert(test, {Y,1})|| Y <- lists:seq(1,5)], [ets:insert(test2, {Z,1}) || Z <- lists:seq(1,5)] end) || X <- lists:seq(10,20)].
[spawn(fun() -> [ets:lookup(test, Y)|| Y <- lists:seq(1,5)], [ets:lookup(test2, Z) || Z <- lists:seq(1,5)] end) || X <- lists:seq(10,20)].

What do you think? Remember that it's only a prototype :slightly_smiling_face:

[mentels]

I would say that the biggest problem with it is the readability. Too many columns, columns labels are hidden... even if the information presented in the table are valuable, we need a better way to present them.

[baransu]

Make columns width different for different columns. Also you can try better styling for different columns to differ more important data. Also search input would be great, something like name:test2, size: 5 for example (look at github search, I have something like this in mind).

[mkacper]

I absolutely agree. We could e.g. aggregate data related to particular functions somehow. When it comes to the table's appearance (labes etc.) it obviously needs improvements. I didn't focused on the appearance at this stage but when we decide that ets table view should be our final implementation of the ets node view we have to deal with it.

[mkacper]

Possibility of sorting the table by columns stored numerical values would be nice too.

[arkgil]

+1 to @mentels feedback

[mkacper]

Any ideas how to improve readability of the table are welcome :slightly_smiling_face:

[mkacper]

I've been working on ets table view in terms of its readability for a few days. I changed the layout of the table to increase its clarity. There is a possibility now to choose which columns are showed in a table. It contains only the most important columns by default to avoid confusion. I've also tried to make it possible to sort data in particular columns but it dosen't work yet.

Screenshots:

[baransu]

A lot cleaner in my opinion 🙂

[mkacper]

Close via #76