[job-components, terafoundation, teraslice] Remove prom metrics assignment default label, exclude master pod from active execution list

busma13 commented 1 month ago

This PR makes the following changes:

Remove prom metrics assignment default label. This should never have been included.
Exclude master pod from active execution list. We didn't realize this was included in the cluster/state endpoint active array for the node that had the master.
bump teraslice from 2.6.0 to 2.6.1
bump terafoundation from 1.5.1 to 1.5.2
bump job-components from 1.5.1 to 1.5.2

Ref: #3743

godber commented 1 month ago

I think I know how that assignment got in there ... don't we have two different usecases for exporters? One where its the master sharing cluster wide statistics, and one when it's a worker level exporter, sharing job level metrics?

busma13 commented 1 month ago

Yes. The execution controller also has metrics, so there are 3 uses cases I guess.

busma13 commented 1 month ago

The now removed job_metric_api in standard-assets prefixed all job and worker metric names with teraslice_job_ and added the pod_name label to all metrics of the workers. That would lead me to believe that these will be expected on those jobs when prod switches over to prom metrics exporters internal to teraslice. Currently the prefix and labels are set in the PromMetricsInitConfig for the exc and workers. If we don't want these prefix and labels on all jobs, then would it make sense to add fields to the terafoundation and/or jobConfig to pass in these values? If we don't, then these will have to be on all workers and execution controllers or none of them.

godber commented 1 month ago

We had discussed you adding documentation that would document what both exporters are exporting, I don't see that in this PR.

busma13 commented 1 month ago

These are what we want the metrics to look like:

Master:

teraslice_master_info{arch="arm64",clustering_type="kubernetes",name="ts-dev1",node_version="v22.5.1",platform="linux",teraslice_version="2.6.1",url="localhost"} 1

teraslice_slices_processed{name="ts-dev1",url="localhost"} 214

teraslice_slices_failed{name="ts-dev1",url="localhost"} 0

teraslice_slices_queued{name="ts-dev1",url="localhost"} 0

teraslice_workers_joined{name="ts-dev1",url="localhost"} 0

teraslice_workers_disconnected{name="ts-dev1",url="localhost"} 0

teraslice_workers_reconnected{name="ts-dev1",url="localhost"} 0

teraslice_controller_workers_active{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 1
teraslice_controller_workers_active{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 1

teraslice_controller_workers_available{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 0
teraslice_controller_workers_available{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 0

teraslice_controller_workers_joined{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 1
teraslice_controller_workers_joined{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 1

teraslice_controller_workers_reconnected{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 0
teraslice_controller_workers_reconnected{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 0

teraslice_controller_workers_disconnected{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 0
teraslice_controller_workers_disconnected{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 0

teraslice_execution_info{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",image="teraslice-workspace:e2e-nodev22",version="2.6.1",name="ts-dev1",url="localhost"} 1
teraslice_execution_info{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",image="teraslice-workspace:e2e-nodev22",version="2.6.1",name="ts-dev1",url="localhost"} 1

teraslice_controller_slicers_count{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 1
teraslice_controller_slicers_count{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 1

teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="pending",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="scheduling",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="initializing",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="running",name="ts-dev1",url="localhost"} 1
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="recovering",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="failing",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="paused",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="stopping",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="completed",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="stopped",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="rejected",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="failed",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",status="terminated",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="pending",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="scheduling",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="initializing",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="running",name="ts-dev1",url="localhost"} 1
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="recovering",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="failing",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="paused",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="stopping",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="completed",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="stopped",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="rejected",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="failed",name="ts-dev1",url="localhost"} 0
teraslice_execution_status{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",status="terminated",name="ts-dev1",url="localhost"} 0

teraslice_controller_slices_processed{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 28
teraslice_controller_slices_processed{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 35

teraslice_controller_slices_failed{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 0
teraslice_controller_slices_failed{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 0

teraslice_controller_slices_queued{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 2
teraslice_controller_slices_queued{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 3

teraslice_execution_created_timestamp_seconds{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 1729290554.8
teraslice_execution_created_timestamp_seconds{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 1729290554.627

teraslice_execution_updated_timestamp_seconds{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 1729290568.565
teraslice_execution_updated_timestamp_seconds{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 1729290568.043

teraslice_execution_slicers{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 1
teraslice_execution_slicers{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 1

teraslice_execution_workers{ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",name="ts-dev1",url="localhost"} 1
teraslice_execution_workers{ex_id="006ebee7-321b-4c16-a603-b2b7478fb709",job_id="5ed22a2b-023f-473b-a3b9-8fb571908539",job_name="peter-kafka-to-es",name="ts-dev1",url="localhost"} 1

Execution Controller:

teraslice_job_execution_controller_info{arch="arm64",clustering_type="kubernetes",name="ts-dev1",node_version="v22.5.1",platform="linux",teraslice_version="2.6.1",url="localhost",ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",assignment="execution_controller",pod_name="ts-exc-peter-datagen-to-kafka-1ba6f6ac-40e1-6rplj"} 1

teraslice_job_slices_processed{name="ts-dev1",url="localhost",ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",assignment="execution_controller",pod_name="ts-exc-peter-datagen-to-kafka-1ba6f6ac-40e1-6rplj"} 127

Worker:

teraslice_job_worker_info{arch="arm64",clustering_type="kubernetes",name="ts-dev1",node_version="v22.5.1",platform="linux",teraslice_version="2.6.1",url="localhost",ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",assignment="worker",pod_name="ts-wkr-peter-datagen-to-kafka-1ba6f6ac-40e1-649b7558f-6dcp8"} 1

teraslice_job_slices_processed{name="ts-dev1",url="localhost",ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",assignment="worker",pod_name="ts-wkr-peter-datagen-to-kafka-1ba6f6ac-40e1-649b7558f-6dcp8"} 523

teraslice_job_kafka_bytes_produced{op_name="kafka_sender",name="ts-dev1",url="localhost",ex_id="18872374-ac97-44be-bd77-1780f2aba10f",job_id="1ba6f6ac-40e1-48f1-b5ea-4d2b7704788e",job_name="peter-datagen-to-kafka",assignment="worker",pod_name="ts-wkr-peter-datagen-to-kafka-1ba6f6ac-40e1-649b7558f-6dcp8"} 949471967

busma13 commented 1 month ago

Results testing this build in a test environment:


Master:

teraslice_master_info{arch="x64",clustering_type="kubernetesV2",name="teraslice-dev1",node_version="v22.9.0",platform="linux",teraslice_version="2.6.1",url="https://ts-dev1.tera4.lan"} 1

teraslice_slices_processed{name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 6

teraslice_slices_failed{name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0

teraslice_slices_queued{name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 3

teraslice_workers_joined{name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1

teraslice_workers_disconnected{name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0

teraslice_workers_reconnected{name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0

teraslice_controller_workers_active{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1

teraslice_controller_workers_available{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0

teraslice_controller_workers_joined{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1

teraslice_controller_workers_reconnected{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0

teraslice_controller_workers_disconnected{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0

teraslice_execution_info{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",image="ghcr.io/terascope/teraslice:v2.6.1-nodev22.9.0",version="2.6.1",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1

teraslice_controller_slicers_count{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1

teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="pending",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="scheduling",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="initializing",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="running",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="recovering",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="failing",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="paused",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="stopping",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="completed",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="stopped",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="rejected",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="failed",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0
teraslice_execution_status{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",status="terminated",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0

teraslice_controller_slices_processed{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 65

teraslice_controller_slices_failed{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 0

teraslice_controller_slices_queued{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 3

teraslice_execution_created_timestamp_seconds{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1729637884.485

teraslice_execution_updated_timestamp_seconds{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1729637956.801

teraslice_execution_slicers{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1

teraslice_execution_workers{ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",name="teraslice-dev1",url="https://ts-dev1.tera4.lan"} 1

Execution_controller:

teraslice_job_execution_controller_info{arch="x64",clustering_type="kubernetesV2",name="teraslice-dev1",node_version="v22.9.0",platform="linux",teraslice_version="2.6.1",url="https://ts-dev1.tera4.lan",ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",assignment="execution_controller",pod_name="ts-exc-peter-datagen-to-noop-da2d0a46-da70-t426b"} 1

teraslice_job_slices_processed{name="teraslice-dev1",url="https://ts-dev1.tera4.lan",ex_id="69d44a9e-4894-429d-817d-789088b067ff",job_id="da2d0a46-da70-450a-af21-d9120f145701",job_name="peter-datagen-to-noop",assignment="execution_controller",pod_name="ts-exc-peter-datagen-to-noop-da2d0a46-da70-t426b"} 183

Worker:

teraslice_job_worker_info{arch="x64",clustering_type="kubernetesV2",name="teraslice-dev1",node_version="v22.9.0",platform="linux",teraslice_version="2.6.1",url="https://ts-dev1.tera4.lan",ex_id="85e56ccb-4d7a-42ac-82e3-747d7790fa75",job_id="f9958158-52d6-460a-90d1-e0733852d9cd",job_name="peter-datagen-to-kafka",assignment="worker",pod_name="ts-wkr-peter-datagen-to-kafka-f9958158-52d6-75c9b44bb9-fzpjl"} 1

teraslice_job_slices_processed{name="teraslice-dev1",url="https://ts-dev1.tera4.lan",ex_id="85e56ccb-4d7a-42ac-82e3-747d7790fa75",job_id="f9958158-52d6-460a-90d1-e0733852d9cd",job_name="peter-datagen-to-kafka",assignment="worker",pod_name="ts-wkr-peter-datagen-to-kafka-f9958158-52d6-75c9b44bb9-fzpjl"} 69

teraslice_job_kafka_bytes_produced{op_name="kafka_sender",name="teraslice-dev1",url="https://ts-dev1.tera4.lan",ex_id="85e56ccb-4d7a-42ac-82e3-747d7790fa75",job_id="f9958158-52d6-460a-90d1-e0733852d9cd",job_name="peter-datagen-to-kafka",assignment="worker",pod_name="ts-wkr-peter-datagen-to-kafka-f9958158-52d6-75c9b44bb9-fzpjl"} 125036729

terascope / teraslice

[job-components, terafoundation, teraslice] Remove prom metrics assignment default label, exclude master pod from active execution list #3799