search

openstreetmap / operations

OSMF Operations Working Group issue tracking
https://operations.osmfoundation.org/
99 stars 13 forks source link

Containerise small services #807

Open grischard opened 1 year ago

grischard commented 1 year ago

There's a number of small services that currently live on a full server, and could live inside a container.

Firefishy commented 1 year ago

I am going to test OSQA as a container first.

tomhughes commented 1 year ago

Yes sure why not pick literally the hardest possible thing to try first.

Firefishy commented 1 year ago

Choose container system... examples could be:

tomhughes commented 1 year ago

Well there's actually really two separate questions - what to use to build images and what to use to deploy them.

As I understand there are other less horrible languages that Dockerfile for describing images and those images can be deployed with any of the major systems just as podman for example can deploy an image built from a Dockerfile.

tomhughes commented 1 year ago

Some articles on container systems and orchestration systems.

I think https://buildah.io/ is what I was thinking of as the main alternative to Dockerfile - it can actually use Dockerfiles but the native way is just based around writing a script that uses buildah commands to manipulate the image.

Firefishy commented 1 year ago

Dockerfiles are the beast I know and by a million miles has more adoption than the alternatives.

There are alternatives for building OCI compatible container images with varying levels completeness. Other than https://buildah.io/ there is also https://github.com/genuinetools/img (uses Dockerfile) and https://github.com/GoogleContainerTools/jib (java images)

grischard commented 1 year ago

The stuff that runs on naga (redirectors, blog aggregator, munin server that we're going to demise (#501)) might be good.

grischard commented 1 year ago

https://hardware.osm.org which runs on idris even already has a dockerfile in https://github.com/osmfoundation/osmf-server-info

Firefishy commented 1 year ago

The following sites are now containers:

jpds commented 1 year ago

You probably want to deploy https://github.com/google/cadvisor on every Docker host and point Prometheus at that,

tomhughes commented 1 year ago

Well that rather depends how it works - if it depends on the docker daemon then it probably won't work for us.

tomhughes commented 1 year ago

A quick test on naga suggests it doesn't manage to collect anything useful in our setup - all it finds is some basic host hardware metrics:

# HELP machine_cpu_cores Number of logical CPU cores.
# TYPE machine_cpu_cores gauge
machine_cpu_cores{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 64 1681303543392
# HELP machine_cpu_physical_cores Number of physical CPU cores.
# TYPE machine_cpu_physical_cores gauge
machine_cpu_physical_cores{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 16 1681303543392
# HELP machine_cpu_sockets Number of CPU sockets.
# TYPE machine_cpu_sockets gauge
machine_cpu_sockets{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 2 1681303543392
# HELP machine_dimm_capacity_bytes Total RAM DIMM capacity (all types memory modules) value labeled by dimm type.
# TYPE machine_dimm_capacity_bytes gauge
machine_dimm_capacity_bytes{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447",type="Registered-DDR4"} 2.06158430208e+11 1681303543392
# HELP machine_dimm_count Number of RAM DIMM (all types memory modules) value labeled by dimm type.
# TYPE machine_dimm_count gauge
machine_dimm_count{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447",type="Registered-DDR4"} 12 1681303543392
# HELP machine_memory_bytes Amount of memory installed on the machine.
# TYPE machine_memory_bytes gauge
machine_memory_bytes{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 2.02673983488e+11 1681303543392
# HELP machine_nvm_avg_power_budget_watts NVM power budget.
# TYPE machine_nvm_avg_power_budget_watts gauge
machine_nvm_avg_power_budget_watts{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 0 1681303543392
# HELP machine_nvm_capacity NVM capacity value labeled by NVM mode (memory mode or app direct mode).
# TYPE machine_nvm_capacity gauge
machine_nvm_capacity{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",mode="app_direct_mode",system_uuid="32353537-3835-584d-5136-303130324447"} 0 1681303543392
machine_nvm_capacity{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",mode="memory_mode",system_uuid="32353537-3835-584d-5136-303130324447"} 0 1681303543392
# HELP machine_scrape_error 1 if there was an error while getting machine metrics, 0 otherwise.
# TYPE machine_scrape_error gauge
machine_scrape_error 0
tomhughes commented 1 year ago

Also generating metrics with timestamps (rather than letting the server add them) is generally considered bad form and can cause problems with metric ingestion.

jpds commented 1 year ago

generating metrics with timestamps (rather than letting the server add them) is generally considered bad form

Do you have a source for this? Because just searching for "timestamp" on my own Grafana brings up:

tomhughes commented 1 year ago

I mean that fact that each of those metrics has a timestamp like 1681303543392 after the metric value at the end of the line - when that is there the server will use that as the timestamp to associate with the value instead of using it's own clock to get a scrape time to associate with the value.

See https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus/#buggy-client-side-timestamps for some discussion of the potential issues with it.

jpds commented 1 year ago

Ah yes - that's already discussed on cAdvisor issue #2526.

However, if you don't see one of these https://github.com/google/cadvisor/blob/master/metrics/prometheus.go#L138 metrics here, then something is misconfigured.

tomhughes commented 1 year ago

Well I didn't do any configuration. I just ran the executable as there didn't seem to be any clear instructions telling me to do anything else.

As I say because we are using podman if it relies on talking to dockerd to get statistics then it's probably not going to work.

Firefishy commented 1 year ago

dmca.osm.org now a container: https://github.com/openstreetmap/chef/commit/4ac7cf54e6e68995fe0d1c020e044184432d79e3

grischard commented 12 months ago

Left to containerise on Ridley: tracked in #1028

Firefishy commented 1 month ago

https://github.com/openstreetmap/birthday20-website/ is a wordpress to static site I generated using wp2static.

wp2static did a good job. Some cleanup was required, but it does a reasonable job.

Linked issue: https://github.com/openstreetmap/operations/issues/1125

Firefishy commented 1 month ago

SoTM 2007, 2008 and 2009 are now containers.

Firefishy commented 1 week ago

I need to work out what the list to completion is here.