Okay after much back and forth I understand a lot better what is required from the tags to reliably get metrics working across the board.
At a minimum we need the hostname / pod_name identifier to be set as a tag to that metrics can be delineated at this level. This is much more important on GCP (though also important on AWS) as they limit aggressively how many datapoints for a particular metric + tags (dimensions) pair can be pushed per interval. in AWS you can publish pretty well as many datapoints as you like but in Stackdriver you need to be more careful here.
How we were solving that was to aggregate up all metrics that had the same name - this was in relation to the publishing issue above. What I got wrong at the time was that different tags allowed it to work correctly. So as long as there is one different tag in the mix it should all work correctly!
Okay after much back and forth I understand a lot better what is required from the tags to reliably get metrics working across the board.
At a minimum we need the hostname / pod_name identifier to be set as a tag to that metrics can be delineated at this level. This is much more important on GCP (though also important on AWS) as they limit aggressively how many datapoints for a particular metric + tags (dimensions) pair can be pushed per interval. in AWS you can publish pretty well as many datapoints as you like but in Stackdriver you need to be more careful here.
How we were solving that was to aggregate up all metrics that had the same name - this was in relation to the publishing issue above. What I got wrong at the time was that different tags allowed it to work correctly. So as long as there is one different tag in the mix it should all work correctly!