How do you add collated stats for providers like Hetzner ?

[mrchrisadams]

Problem

We use Hetzner for our cloud compute services, not AWS, and we want to try collecting figures for the environmental impact of running our existing services. We also use Scaleway for object storage, but much less of it, and there appear to be some issues already for them,

We're not sure how do to this with Boavista right now.

Solution

We'd like to understand how to use Boavista's API to collate figures on a per-host basis, so we can create similar metrics and dashboards as seen on the link below in the recent SDIA Green Coding workshop in Berlin:

https://docs.google.com/presentation/d/1yNR_buB9bQWlDzteBH06WYRWyjW0_sb7qm2q2X82j4U/edit#slide=id.g29e1bc192dd_0_196

Alternatives

We've also been looking at doing this in Cloud Carbon footprint, because they provide some scope for per-service level reporting, and (I think) reporting based on tags we apply to resources.

There's a fairly detailed issue linked below, and I started a PR a while ago, but haven't found the time to build a first version yet:

https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/issues/512#issuecomment-946609102

Additional context or elements

We use Hetzner Cloud, not their physical servers via H-Robot.

This means we have limited access to the hardware, but we can still make some assumptions.

Hetzner provide a number of helpful APIs, some of which have been converted to expose some relevant info for grafana / prometheus style setups, although this may overlap with boavista's per-host agent approach.

https://promhippie.github.io/hcloud_exporter/

I believe this exporter queries the API exposed by Hetzner below: https://docs.hetzner.cloud/

If it helps, we'd be happy to contribute an example to the docs of what a a setup using Boavista looks like, once we know how to do it.

[da-ekchajzer]

Hi @mrchrisadams, Your suggestion triggered a lot of work ;) see https://github.com/Boavizta/boaviztapi/pull/258 & https://github.com/Boavizta/boaviztapi/issues/252.

We have done some development on the API to make the contribution process easier. Take a look at the new documentation and try to contribute: https://doc.api.boavizta.org/contributing/cloud_provider/. Please let us know if the documentation is unclear so we can improve it.

[mrchrisadams]

Oh, wow @da-ekchajzer this looks really cool!

So if I understand this correctly, adding support would be something like:

1. add a file, hetzner.csv
2. inside that file, define a one or more platforms, - these are a bit like the physical machine a VM might run on
3. inside the same file, define one or more instances - these might correspond to the VMs that are sold, and run inside the platform defined in the previous step.

Let's say I'm looking at the new arm64 servers tha Hetzner offer (mainly because there are fewer of them and it makes for a simpler example). I've attached a screenshot for clarity:

I might need to figure out what the platform is, that actually 'contains' these instance types. For simplicity, I'm pretending that the largest instance type can also be the platform, a bit like the linked the AWS examples,where really big instances can effectively take up the platform's worth of capacity too.

So, maybe something like this for the platform:

- id: CAX41 
- manufacturer: Hetzner
- CASE.case_type: rack
- CPU.units: 16 (let's put aside CPU vs vCPU for now)
- CPU.name: Arm Ampere Altra
- RAM.units: 8
- RAM.capacity: 32
- SSD.units: 0
- SSD.capacity: 0
- HDD.units: 8 (just a guess)
- HDD.capacity: 320 (just guess)

... plus whatever relevant stats I can find for the physical server

I can't remember if the CAX41 ones use SSDs or HDDs, but for the purposes of this example, it doesn't matter that much - I'd check the specs to see, and then add these.

And then for the instances I need to list all the instance types. So for the smallest instance, it might look like CAX11, and we'd need one of these for each possible instance type (CAX11, CAX21, CAX31, etc)

- id: CAX11 
- manufacturer: Hetzner
- CASE.case_type: rack
- CPU.units: 2
- CPU.name: Arm Ampere Altra
- RAM.units: 1
- RAM.capacity: 4
- SSD.units: 0
- SSD.capacity: 0
- HDD.units: 1 (just a guess)
- HDD.capacity: 40

For the Ampere range it would end up as a probably a CSV file which is about 4-5 lines long, based on the different instances offered.

Is that in the right direction? I appreciate there is waaay more detail to add, but if that's broadly how it ought to work, then I can try contacting Hetzner to ask for help getting this together, and then thinking about how to contribute to the documentation.

[da-ekchajzer]

Here are some precisions/corrections :

The process

1. add "hetzner" in boaviztapi/data/archetypes/cloud/providers.csv (see doc : https://doc.api.boavizta.org/contributing/cloud_provider/)
2. add a file, hetzner.csv in boaviztapi/data/archetypes/cloud/ with the same column as boaviztapi/data/archetypes/cloud/aws.csv see doc : https://doc.api.boavizta.org/contributing/cloud_provider/)
3. ~inside that file, define a one or more platforms~ inside the server.csv define on or more platform., - these are a bit like the physical machine a VM might run on (see doc : https://doc.api.boavizta.org/contributing/server/).
4. ~inside the same file, define one or more instances~ in boaviztapi/data/archetypes/cloud/hetzner.csv define one or more instances. You will refer to the platform with its name on column platform - these might correspond to the VMs that are sold, and run inside the platform defined in the previous step. (see : https://doc.api.boavizta.org/contributing/cloud_instance/)

I might need to figure out what the platform is, that actually 'contains' these instance types. For simplicity, I'm pretending that the largest instance type can also be the platform, a bit https://github.com/Boavizta/boaviztapi/issues/252#issue-2026729673,where really big instances can effectively take up the platform's worth of capacity too.

This is a good option. You can also use a generic server (with no manufacturer) such as platform_compute_medium. You should just be sure that your instance specs are never higher than the platform spec (which would make no sense physically).

So, maybe something like this for the platform:

• id: CAX41
• manufacturer: Hetzner
• CASE.case_type: rack
• CPU.units: 16 (let's put aside CPU vs vCPU for now)

Should be the number of CPU on the server 16 seems to be the number of vCPU

• CPU.name: Arm Ampere Altra

You should check the availability of the CPU in https://github.com/Boavizta/boaviztapi/blob/main/boaviztapi/data/crowdsourcing/cpu_specs.csv. If not available, you should add its specs. (see : https://doc.api.boavizta.org/contributing/cpu/)

• RAM.units: 8
• RAM.capacity: 32
• SSD.units: 0
• SSD.capacity: 0
• HDD.units: 8 (just a guess)
• HDD.capacity: 320 (just guess) ... plus whatever relevant stats I can find for the physical server I can't remember if the CAX41 ones use SSDs or HDDs, but for the purposes of this example, it doesn't matter that much - I'd check the specs to see, and then add these.

And then for the instances I need to list all the instance types. So for the smallest instance, it might look like CAX11, and we'd need one of these for each possible instance type (CAX11, CAX21, CAX31, etc)

Nop. The instance info should be define by the charachterictis define here : https://doc.api.boavizta.org/contributing/cloud_instance/

Column name	Required	Unit	Description	Example
id	Required		Instance identifier	c5.12xlarge
vcpu	Required	unit	Number of vCPU	48
memory	Required	GB	RAM quantity	96
ssd_storage		GB	SSD storage quantity (can be 0)	0
hdd_storage		GB	HDD storage quantity (can be 0)	0
gpu_units		unit	GPU quantity (not supported yet)	0
platform	Required			c5.metal

If you have some suggestions to improve the contribution documentation, feel free to give feed back or to improve it directly.

[mrchrisadams]

Thanks @da-ekchajzer - I think this gives me enough to have a first go at getting this together 👍

I'll shoot for late next week.

[da-ekchajzer]

Hello Chris, any pain points collecting the data ? I stay available for further explanations.