VictoriaMetrics / operator

Kubernetes operator for Victoria Metrics
Apache License 2.0
424 stars 142 forks source link

[VMAgent] source_labels or sourceLabels | Operator claims spec.remoteWrite in managedFields #946

Closed sdomme closed 4 months ago

sdomme commented 5 months ago

Hi,

I have a question regarding using the source_labels/sourceLabels and/or target_label/targetLabel in a spec.remoteWrite inlineUrlRelabelConfig (probably global RelabelConfig is also affected).

version: victoriametrics/operator:v0.42.3

It doesn't matter if you deploy a VMAgent with the underscore version for inlineUrlRelabelConfig like this:

- inlineUrlRelabelConfig:
  - action: replace
    regex: vmagent_remotewrite_bytes_sent_total
    replacement: new_label_value
    source_labels:
      - __name__
    target_label: new_label
  url: https://url.xyz

or the version without underscore like this:

- inlineUrlRelabelConfig:
  - action: replace
    regex: vmagent_remotewrite_bytes_sent_total
    replacement: new_label_value
    sourceLabels:
      - __name__
    targetLabel: new_label
  url: https://url.xyz

it ends up building this:

- inlineUrlRelabelConfig:
  - action: replace
    regex: vmagent_remotewrite_bytes_sent_total
    replacement: new_label_value
    sourceLabels:
      - __name__
    source_labels:
      - __name__
    targetLabel: new_label
    target_label: new_label
  url: https://url.xyz

The issue we have now is, in any case the operator claims the spec.remoteWrite field in managedFields:

managedFields:
- ...
- apiVersion: operator.victoriametrics.com/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:operator.victoriametrics/last-applied-spec: {}
          f:finalizers:
            .: {}
            v:"apps.victoriametrics.com/finalizer": {}
        f:spec:
          f:remoteWrite: {}
      manager: manager
      operation: Update
      time: '2024-05-06T13:35:58Z'

that leads to an apply conflict for remoteWrite changes as per this definition for the manager that handled the initial creation of the VMAgent object with a Server-Side Apply.

│ Error: victoria/application failed to run apply: Apply failed with 1 conflict: conflict with "manager" using operator.victoriametrics.com/v1beta1: .spec.remoteWrite
│ Please review the fields above--they currently have other managers. Here
│ are the ways you can resolve this warning:
│ * If you intend to manage all of these fields, please re-run the apply
│   command with the `--force-conflicts` flag.
│ * If you do not intend to manage all of the fields, please edit your
│   manifest to remove references to the fields that should keep their
│   current managers.
│ * You may co-own fields by updating your manifest to match the existing
│   value; in this case, you'll become the manager if the other manager(s)
│   stop managing the field (remove it from their configuration).
│ See http://k8s.io/docs/reference/using-api/server-side-apply/#conflicts

I can't remember having this issue before, but we also didn't change remoteWrite for a longer time.

Also existing VMAgents aren't affected by this. The operator does not set this double version parameters. Only if we create new ones.

f41gh7 commented 5 months ago

Hello, thanks for reporting. It's a bug, we're going to fix it at the next release.

f41gh7 commented 4 months ago

Changes included to the v0.44.0 release

Feel free to re-open if it doesnt help