compspec / compspec-go

Prototype compatibility plugin in go for testing compspec descriptive metadata
MIT License
0 stars 0 forks source link

wip: add support for node extraction -> cluster metadata #26

Closed vsoch closed 6 months ago

vsoch commented 6 months ago

Problem: we need a extract metadata for nodes and then parse into a cluster graph Solution: create a compspec create nodes subcommand.

This is for the registration step of a cluster to rainbow. Information about the design / idea planning is here.

In this PR I am adding a ClusterGraph, which still needs work to improve the output to easily map into a JGF (right now it has elements that can support any type that need further parsing). I am also generalizing the idea of plugins more, so we will have extractors and converters (that run create) but I need to finalize the design for the latter, right now the create commands are very separate. I am opening the PR sooner than later in case my computer explodes. A few problems I have run into is that NFD does not have cpu counts, let along physical vs. logical. This information is in /proc/cpuinfo for x86 but not arm. We also do not have a way to get socket -> core mapping. So likely we do need to add the hwloc extractor, and provide an automated build for doing that since it requires hwloc on the system. I will put some thought into this.

Dinosaur TODO

Update for the above - hwloc won't be possible soon, there is a bug in the library. I've opened an issue, and I'll remove the bullets to #19

image

Update: JGF looks good! It was... a pointer bug!

image

vsoch commented 6 months ago

This is ready to go.

The current issues with the second are that node feature discovery is not giving us the topology of sockets -> cores, nor is it able to give us basic counts of virtual vs physical cores. We will need hwloc for that (I hope/think). I can't add the bindings yet because there is a bug, but I wrote the Makefile (added here) that will make that possible.

This should be enough to prototype with rainbow, so I'm going to merge. Will pick up more after dinner.