search

cloudfoundry-community / stackdriver-tools

Stackdriver Nozzle for Cloud Foundry Loggregator, Host Monitoring Agents BOSH Release
Apache License 2.0
21 stars 13 forks source link

Support exporting metrics to Stackdriver Logging #99

Closed johnsonj closed 6 years ago

johnsonj commented 7 years ago

Today we split events from the Loggregator into Stackdriver Logging/Monitoring. It is desirable to have the metric data available in Stackdriver Logging so users can use exports for further analysis. An example would be exporting to BigQuery to perform custom calculations on metrics not available in Stackdriver Monitoring.

Design Considerations:

mrdavidlaing commented 7 years ago

Should it be a 'All or Nothing' to log to both Stackdriver Logging and Monitoring or should each endpoint have it's own list of events?

We were certainly surprised that when adding CounterEvent,ValueMetric,ContainerMetric events to firehose.events: sent them to a different place ( Stackdriver Monitoring rather than Stackdriver Logging ).

We'd like to propose having 2 separate config settings ( in both the BOSH release and the Tile )

  firehose.events_to_stackdriver_logging:
    description: Whitelisted event types to send to Stackdriver Logging (comma separated). Valid values include LogMessage,Error,HttpStartStop,CounterEvent,ValueMetric,ContainerMetric
    default: LogMessage,Error
  firehose.events_to_stackdriver_monitoring:
    description: Whitelisted event types to send to Stackdriver Monitoring (comma separated). Valid values include CounterEvent,ValueMetric,ContainerMetric
    default: CounterEvent,ValueMetric,ContainerMetric

For the Pivotal Tracker on GCP use case; we'd probably send everything to Stackdriver Logging -> because we want the ability to send some of the data into BigQuery, and CounterEvent,ValueMetric to Stackdriver Monitoring

mrdavidlaing commented 7 years ago

@johnsonj Thanks for acting on this issue so quickly.

Will this change make its way into v1.0.6 of the GCP Stackdriver Nozzle for PCF tile ?

johnsonj commented 7 years ago

Yes! I just need to do some manual validation before releasing. I think it will be v1.1.0.

On Wed, Aug 23, 2017, 11:35 PM David Laing notifications@github.com wrote:

@johnsonj https://github.com/johnsonj Thanks for acting on this issue so quickly.

Will this change make its way into v1.0.6 of the GCP Stackdriver Nozzle for PCF tile https://network.pivotal.io/products/gcp-stackdriver-nozzle ?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/cloudfoundry-community/stackdriver-tools/issues/99#issuecomment-324354705, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZo5c825cDGPerTO3qb0v5W8MnlFHxbks5sbDjHgaJpZM4O6m9_ .

johnsonj commented 7 years ago

Released as v1.0.6. Pivnet update pending approval.

johnsonj commented 7 years ago

Re-opening since v1.0.6 was rolled back due to performance issues

johnsonj commented 6 years ago

Released as v2.0.1.