Transfer Monitoring

[maany]

Description

Discussions about the requirements for the WebUI in relation to https://github.com/rucio/rucio/issues/6189

Motivation

No response

Change

No response

Additional Information

• [X] The Rucio API needs to be changed.

[rcarpa]

Hi @fno2010.

Following your comment https://github.com/rucio/webui/issues/301#issuecomment-1729596149

Can you please elaborate on the need to add the type for the request? Are you intending to stream all requests and then compute statistics in javascript?

[fno2010]

@rcarpa, I think streaming original requests directly is the easiest way. Just like the current web-ui does for DIDs, RSEs, and Rules. But we don't need to stream all requests. I think we can query the subset of requests on demand, e.g., filtered by request_type, request_state, account, etc. Is there any privacy issue?

[rcarpa]

I'm thinking more about the performance issues. The "normal" size of the requests table is a couple million rows. It frequently grows even bigger. I think it's wiser to rely on aggregated statistics which is computed (and, most probably, cached) on server side. It's better to limit ourselves to the data returned by the following SQL query : https://github.com/rucio/rucio/blob/c3ded056c52b99ef7d692dde8c947f2280c3c2b1/lib/rucio/core/request.py#L1290

[fno2010]

Ah, good point. What is the scale of the active requests (state = SUBMITTED) for a specific source-dest RSE pair? I'm thinking that we can provide the aggregated statistics for the overview, and provide more detailed attributes for each given source-dest RSE pair if possible. I have been working on the similar visualization for FTS3. The scale of the active transfers per source-dest pair in the fts3-atlas server is acceptable. But maybe the scale in Rucio is larger because the submitted requests from the Rucio view may be queued from the FTS3 view.

[rcarpa]

The order of magnitude between rucio and fts is the same.

Considering the visualisation, did you already have an idea in mind on how to represent it ? After speaking with the Operations team, I'm pretty sure that showing the data on a World map is a bad idea. Because of the high density of RSEs in some areas and low density in others. I didn't think it through very much, but something like a circle layout, cise layout, or the wine and cheese demo from the cytoscape library seem interesting example for visualizations. It's not an endorsement of this library at all. I never used it. Just some examples I found on the web. Or do you have better ideas?

[fno2010]

@rcarpa Sorry about missing your follow-up discussions. As a starting point, I would like to finish the table view first. But I do think about some potential graph views.

I agree with you that the RSE-level topology is highly dense. The RSEs from the same organization usually have close geolocations on a world map. I think we can add a group-level topology as an overlay of the RSE-level topology. The group can be based on the site (from the CRIC database), country, or some other meta attributes. The group can be folded/unfolded.

Several years ago, we used cisco's next library to realize this effect. Like this demo. But it seems that the next library has no longer been maintained. I see G6 has a similar feature. Maybe cytoscape.js can do a similar thing. I will investigate a little bit.

[fno2010]

Since we discussed the selection of third-party libraries for graph-based visualization, I just investigate a little bit the activity, maintenance, popularity, and performance of some top graph libraries.

Although I'm using G6 right now, the front-end development is still in the early stage. Changing the library will not be very painful.

Library	License	Latest Publish	Popularity	Unpacked Size	Native Typescript Support	Comments
G6	MIT	2023-11-01	1.3k Forks / 10.3k Stars / 82k Weekly Downloads	6.35 MB	✅
cytoscape.js	MIT	2023-10-30	1.6k Forks / 9.4k Stars / 413k Weekly Downloads	4.4 MB + 232 kB (for typescript)	Require additional package
GoJS	Enterprise License	2023-09-12	2.9k Forks / 7.2k Stars / 55k Weekly Downloads	46 MB	✅
sigma.js	MIT	2023-03-22	1.6k Forks / 10.8k Stars / 7k Weekly Downloads	689 kB	✅
graphology	MIT	2023-08-01	72 Forks / 1k Stars / 211k Weekly Downloads	2.73 MB	✅	Graph-related extension on top of sigma.js

[maany]

@fno2010 thanks for the research. The final option (graphology) seems like the safest bet. I believe that implies using sigma.js in conjunction, which is okay too.

[fno2010]

@maany Thanks for sharing the opinion. I'm not a safety expert. Just wonder how to judge the safety.

[maany]

Closing this discussion. We should restart it when we incorporate @fno2010's work in the webui