search

ros-infrastructure / rosindex

The source code for generating index.ros.org
https://index.ros.org
GNU General Public License v3.0
12 stars 24 forks source link

index.ros.org/stats/ takes a very long time to render #337

Closed tfoote closed 1 month ago

tfoote commented 9 months ago

The venn.js library is now overloaded with rendering the combinatorics of our overlaps. The generated file is some 10000 lines long most of which are entries for the venn.js diagram. So it spends many minutes at 100% cpu crunching to try to produce the diagram that's too complicated to effectively render anyway.

I saw between 10 and 13 overlaps trying to be rendered per field.

We should find a way to down scope this to keep the rendering interesting or else just remove it. David has a bunch of good stats at metrics.ros.org that might make sense to push this over there instead.

gabrielhsdev commented 6 months ago

@tfoote

Is this issue still open for work? If so, could I be assigned to it?

Thank you.

tfoote commented 6 months ago

Yeah, I think that at a high level I'd actually suggest that we remove the distro statistics section from the website. And look to contribute them to the https://github.com/DLu/ros_metrics analysis available here https://metrics.ros.org/ If it's pre-rendered the rendering time won't matter as much. But in general these venn diagrams don't make a lot of sense at the scale of thousands.

gabrielhsdev commented 6 months ago

Hello @tfoote

I appreciate your thorough explanation, and I'll take some time to review the ros_metrics repository to better understand how the data can be integrated into that framework.

I agree that pre-rendering the data could significantly alleviate the rendering time concerns.

I'll keep you updated on my progress and reach out if I have any further questions.

Any additional information you can provide would be greatly appreciated. I want to ensure that I approach these issues in the most effective manner possible.

Thank you again for your guidance on this matter.

rkent commented 2 months ago

IIUC the number of intersections is going to grow at a rate of n^2 in distro count. The issue is only getting worse.

But that also means that you do not have to reduce the number of distros by very much to make the venn diagrams work just fine.

I don't think that rosindex should be continuing to show ancient old ros and ros2 distros, as that clutters things up in more ways than just here. If the old repos are reduced to just melodic and galactic, then stats works just fine. I think that is the proper solution to this.