Improve network monitor

[alexisbatyk]

It'll be sending more data to network monitor. Creating more types and displaying it better in the web app.

There was a PR associated with this task. Im continuing with it. https://github.com/freenet/freenet-core/pull/1052

Frontend - Typescript

• [x] Make another index around contract key.
• [x] Make a Modal to filter all the contract info.
• [x] Connect TransactionDetails with data. Make a group for each tx with its changes and data.
• [x] Handle PutRequest and PutSuccess on transactions-data.ts
• [x] add timestamp column to frontend tables
• [x] add order by timestamp feature on tables
• [x] Change all any to actual types.
• [x] Move mocks to mocks.ts
• [x] Move interfaces to type_definitions.ts
• [x] improve UI for clear filter buttons
• [x] improve UI for timestamp order direction toggle
• [x] Add pagination to tables.
• [x] In order to represent how the transaction were flowing through different peers add node ring on Transaction Details.

Backend

• [x] add timestamp column in Rust code when sending Events to network-metricts-server
• [x] Add contracts_data to ServerState in network_metrics_server.rs in fdev.
• [x] Store contract data on fn save_record when pushing a new event.
• [x] Add BroadcastEmitted / BroadcastReceived to transactions and contracts table

Do we need these?

• [ ] In order to be able to use the correct parser and be more deterministic, in fbs define Change union table that will include ContractChange and PeerChange.
• [ ] Propagate Change union to metrics-server Backend
• [ ] Propagate Change union to network-monitor Frontend

added on 29/08

Contract Ring

• [x] In order to show the distribution between the different peers that are caching one contract when opening contract details, add contract ring displaying one orange dot for ideal location and purple dots for peer caching it locations.

Transaction Tree

• [x] In order to represent how the data is propagated via different messages going to many peers in a Put Op, create a transaction tree connecting the peers.

Main Nodes Ring Enhancement

sketch https://excalidraw.com/#json=g2UXiuA2T-P3gg9LY1JdR,Pjg-xsQFuxMNQGGphbxAKA

• [x] In order to have central visibility place the node ring on the landing page, below the peers histogram.
• [x] In order to enable or disable each data group (peers or contracts). Add two toggle buttons, one for Peers and one for Contracts. You'll we able to see peers and/or contracts at the same time.
• [ ] In order to represent the peers data add a point with color pink point for each peer, and one segment between each peer when there are two connected.
• [ ] In order to represent the contracts data add an orange dot for each contract ideal localization and a light green dot for each contract cached localization. There will be a segment between an ideal and a cached localization of the same contract to represent that relation.
• [ ] In order to be able to create peers nucleus look for a clustering algorithm to transform a closely distributed group of peers on the ring into just 1 point. The nucleus data will include how many peers are on it and how many connections exists between them, that will reflect the amount of points that the cluster has. Also It'll have the number of peers connecting to others clusters.
• [ ] In order to represent these nucleus data graphically transform one group of near points into one big circle. The circumference will reflect the amount of points on it. The color of the big circle will be the way to indicate the density of internal connections. Red, yellow and green for low, medium and high.
• [ ] In order to represent the amount of connections of peers from cluster A to cluster B use different segment widths.
• [ ] In order to be able to create contract nucleus look for an algorithm to create a cluster when a contract is cached by a closely distributed group of peers. This group of positions will be a nucleus, including into its data how many peers are caching it
• [ ] In order to represent contract nucleus in a graphical way draw a big or small circle depending on how many cached locations are included. Ideally a peer nucleus will be very close to a contract nucleus. And should have similar size.
• [ ] node ring improvement: show the distance and distribution of nodes around the ring where an X contract is stored.

[alexisbatyk]

TODO: improve node ring, improve clear filters button, add timestamp column in Rust code when sending Events to network-metricts-server, add that column to frontend UI and by clicking order by that

[alexisbatyk]

node ring improvement: show the distance and distribution of nodes around the ring where an X contract is stored.

[alexisbatyk]

1 The node ring is going to be place on the landing page. Below the peers histogram. 2 It's going to have 2 toggle buttons, to enable or disable each of the data groups (node peers or contracts). You'll we able to see peers and/or contracts at the same time.

3 One pink point for each peer, one segment between each peer when there are two connected.

4 An orange dot for each contract real localization and a light green dot for each contract cached localization. There'll be a segment between a real and a cached localization of the same contract to represent that relation.

5 Look for a clustering algorithm to transform a closely distributed group of peers on the ring into just 1 point. That will be a nucleus. That nucleus data will include how many peers are on it and how many connections exists between them, that will reflect the amount of points that the cluster has. Also It'll have the number of peers connecting to others clusters.

6 This clustering will be graphically represented by transforming one group of near points into one big circle. The circumference will reflect the amount of points on it. The color of the big circle will be the way to indicate the density of internal connections. Red, yellow and green for low, medium and high.

7 To represent the amount of connections of peers from cluster A to cluster B will be through the segment width.

8 For contracts the clustering will happen when they are closely situated. This group of positions will be a nucleus, including into its data how many contracts are on it.

9 It will be represented with a big or small circle depending on how many points are included.

Ideally a peer nucleus will be very close to a contract nucleus. And should have similar size.

[alexisbatyk]

Add pagination to tables.

[alexisbatyk]

Add contracts_data to ServerState in network_metrics_server.rs in fdev. Store contract data on fn save_record when pushing a new event.

[alexisbatyk]

Add BroadcastEmitted / BroadcastReceived to transactions and contracts table

[alexisbatyk]

I've spent some time working on these:

• In order to be able to use the correct parser and be more deterministic, in fbs define Change union table that will include ContractChange and PeerChange.
• Propagate Change union to metrics-server Backend
• Propagate Change union to network-monitor Frontend And then figured out how much potential problems and bugs could introduce. Do we need these tasks?

The problem this intended to solve When parsing a fbs message both in networ_metrics_server (backend) and network_monitor (frontend) we do it on a try and fail way:

msg
  - try to parse with parser A
      if failed push error message to error_messages
      if ok -> return with content
  - try to parse with parser B
      if failed push error message to error_messages
      if ok -> return with content
  - try to parse with parser C
      if failed push error message to error_messages
      if ok -> return with content

if error_messages.len > 1 {
  print error_messages;
}

Fixed would be: have a wrapper type with content_type and content properties that will let us know which parser to use

parse msg with wrapper type
match content_type {
    "type_A": try to parse with parser A
    "type_B": try to parse with parser B
    "type_C": try to parse with parser C
}

so if is type_A and doesnt work well with parser A will immediately raise that error and will not be wasting time.

right now there are just 2 parsers, PeerChanges and ContractChanges.

Is it worth it to spend time on this?

[alexisbatyk]

This happens on the network_metrics_server (backend) on crates/fdev/network_metrics_server.rs@113 at async fn push_interface

on the frontend is on network-monitor/src/app.ts@36 at function handleChanges

[iduartgomez]

so if is type_A and doesnt work well with parser A will immediately raise that error and will not be wasting time. right now there are just 2 parsers, PeerChanges and ContractChanges. Is it worth it to spend time on this?

I think it makes sense if we are going to keep making changes or adding more message types in the future, I am in favour of doing the change as it shouldn't be that bad and will clean up code

[alexisbatyk]

Understood, will do it!