Add support for new prometheus RemoteWrite configurations in tidb-monitor

[BenMusch]

Feature Request

Is your feature request related to a problem? Please describe:

Prometheus has updated to expand support for new configurations around remote write. For example, more recent versions of prometheus support a headers configuration which we use at Airbnb to attribute our metrics to different sources.

TiDB's remote write config uses an old version of the prometheus config which only supports a subset of these options: https://github.com/pingcap/tidb-operator/blob/aa375936589f4e668f3c47393536bd383eb5c2cf/docs/api-references/docs.md#remotewritespec

Currently, the prometheus library used in operator is quite old (v 1.8.2 while current versions are already on v2). As far as I can tell, the github.com/prometheus/prometheus library is only used for the configuration structs and does not affect the actual version of prometheus that is being run: https://github.com/pingcap/tidb-operator/pull/1474/files

Describe the feature you'd like: Update to a new (v2+) version of the prometheus library with support for more of the remote_write configurations

Describe alternatives you've considered: There are workarounds like patching our own configuration, but first-class support would be much cleaner

Teachability, Documentation, Adoption, Migration Strategy: Users would use it just like the existing config, but with the newly-available options

[BenMusch]

For what it's worth I've gotten started on this on my own, but the go dependency issues are something I'm struggling with: https://github.com/BenMusch/tidb-operator/compare/master...monitor-remote-write-header-support

[mikechengwei]

We also found this problem, and we will upgrade the prometheus version to support more configurations.

[mikechengwei]

/assign

[mikechengwei]

We have support this feature.