VMAgent Deployment add -promscrape.cluster.name to distinct pods

[okzheng]

When I deploy a vmagent deployment with two replicas, two vmagent pods will scrape same targets and remote write to same vmcluster with same tenant. If vmagents scraped different value at a same metric with same timestamp, then different value will be write vmcluster. It was likely to take a mistake. Even if setting deduplicate flag, it cloud not resolve the problem. So vmagent already provided a flag -promscrape.cluster.name to set different timing offsets for scraping the same targets. It's greate! But the flag could not be yet added pods which came from a deployment, when we deploy vmagent with deployment mode by vm-operator. Have any resolutions? Thanks.

[arctan90]

Updated：My bad, I'm so sorry. shardCount for sharding, not replicaCount.

When I deploy a vmagent deployment with two replicas, two vmagent pods will scrape same targets and remote write to same vmcluster with same tenant. If vmagents scraped different value at a same metric with same timestamp, then different value will be write vmcluster. It was likely to take a mistake. Even if setting deduplicate flag, it cloud not resolve the problem. So vmagent already provided a flag -promscrape.cluster.name to set different timing offsets for scraping the same targets. It's greate! But the flag could not be yet added pods which came from a deployment, when we deploy vmagent with deployment mode by vm-operator. Have any resolutions? Thanks.

hi, okzheng I came to the same thing about some months ago. You can set spec.replicaCount in VMAgent for the effort of
-promscrape.cluster.name. While you can find more infos in HA DOC about high available.

Have Fun

[okzheng]

Thanks, however I can't get the meaning. In my example, the spec.replicaCount was set to 2. So a Deployment has two pods. But the two pods have same spec with same -promscrape.cluster.name. I attempted to add POD_NAME as the value of the flag In VMAgent.

extraArgs: ... ... promscrape.cluster.name: "$(POD_NAME)" extraEnvs:

• name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name

It was valid for my case, every Pod(vmagent) would get different scrape offset. If vm-operator could provide a setting about the flag In VMAgent CRD, I think it would be perfect.

[artmakh]

I've encountered same problem debugging deduplication of data in this issue.

According to docs about VMAgent HA, we should pass different promscrape.cluster.name for each replica of VMAgent we have. But it can't be done in VMAgent CD (we can pass only one, cluster.name with same value to all replicas), so logical solution would be to create N VMAgent CD's with different promscrape.cluster.name, where N will be number of desired replicas. But this will be resulted in multiplication of data, because each VMAgent adds it's own name in vmAgentExternalLabelName generated label, and as we have not a 1 VMAgent in HA, but N different VMAgents we will have N copies of data.

Maybe it's possible to implement this parameter as optional to VMAgent crd, with some kind of automatic counter, which will create N deployment of VMAgent with different promscrape.cluster.name parameters based on number of desired replicas?

[Amper]

To use vmagent in cluster mode with operator you need to setup sharding, in this case, the operator will automatically set the cluster mode parameters. We will try to make this section clearer in a future version of the documentation.