[pyro UI] Tags don't work as expected

[ralucacocioban]

Describe the bug

Tags don't show up consistenty in 'Select Tag' pyro UI dropdown.

More on behaviour:

• tags might show up after many refreshes
• sometimes it only shows up a subset of tags or 0 tags
• the Tag Explorer view also needs few refeshes until the tags show in the UI
• the problematic API seems to be /querier.v1.QuerierService/LabelValues but there are situations where /LabelNames also return nothing {}

To Reproduce

To repro, we need to publish 3 different tags dummyApp.live{myTag=1}, dummyApp.live{myTag=2}, dummyApp.live{myTag=3} with /injest API:

printf "foo;bar 100\n foo;baz 200" | curl \ -X POST \ --data-binary @- \ 'http://localhost:4040/ingest?name=dummyApp.live%7BmyTag=1%7D&from=1715831660&until=1715831668'

printf "fff;bbb 100\n moo;boz 50" | curl \ -X POST \ --data-binary @- \ 'http://localhost:4040/ingest?name=dummyApp.live%7BmyTag=2%7D&from=1715831669&until=1715831675'

printf "oo;bar 600\n foo;baz 500" | curl \ -X POST \ --data-binary @- \ 'http://localhost:4040/ingest?name=dummyApp.live%7BmyTag=3%7D&from=1715831674&until=1715831688'

Might need to adjust from and until to see it in the UI.

Leave it to run for at least 20 mins and then refresh the pyro UI. Try many refreshes to see the behaviour of the tags.

Also, we didn't experience this issue in pyroscope < 1.0