[Spike] Investigate making reservoir sizes for major services changable at runtime

[jtduffy]

Description

Currently, none of the agent's services reservoir size can be dynamically changed (no app restart). We need to investigate the feasibility of adding this behavior to the insight, logging, transaction and span events service.

• Do these other service currently implement the AgentConfigListener interface?
• If not, is there a reason why these services do not?
• Is these a technical reason why the reservoir size can't be dynamically be changed?
• Identify a level of effort (story points) to implement the changes, if it's technically possible to implement the change

[workato-integration[bot]]

https://new-relic.atlassian.net/browse/NR-280261

[jtduffy]

tl;dr

The TransactionEventService, LogSenderService, SpanEventsService and InsightsService can have their reservoir sizes updated by forcing a disconnect/reconnect action.

TransactionEventService

Initial Config

• The configured reservoir max size is configured via the transaction_events.max_samples_stored setting
• The max size for this setting appears to be 10000 (enforced by the back end)
• This setting is sent to the collector in the connect message
• The response payload contains a harvest_limits object which will specify the maximum number of events stored per harvest cycle in the harvest_limits.analytic_event_data property. This will be the max_samples_stored value devided by 12.

  "harvest_limits":{"error_event_data":8,"log_event_data":500,"analytic_event_data":833,"custom_event_data":833}

Updating the Setting During Runtime

• Updating the max_samples_stored value in the yml does update the value in the AgentConfig object, however the max_samples_stored value is not updated in the service itself so the harvest cycle reservoir size is not changed
• Forcing a re-connect of the agent to the back end will basically perform the "initial config" flow, which will update the max_samples_stored in the service correctly.

LogSenderService

Initial Config

• The configured reservoir max size is configured via the application_logging.forwarding.max_samples_stored setting
• The max size for this setting appears to be 100000 (enforced by the back end)
• This setting is sent to the collector in the connect message
• The response payload contains a harvest_limits object which will specify the maximum number of events stored per harvest cycle in the harvest_limits.log_event_data property. This will be the max_samples_stored value devided by 12.

  "harvest_limits":{"error_event_data":8,"log_event_data":8333,"analytic_event_data":833,"custom_event_data":833}

Updating the Setting During Runtime

• Updating the application_logging.forwarding.max_samples_stored setting does adjust the reservoir size per harvest without needed to do a reconnect
• This is done in the service's configChanged method; it assigns the service's max_samples_stored value by taking the value from the config and doing this calculation: maxSamplesStored = (int) (appLoggingConfig.getMaxSamplesStored()*(reportPeriodInMillis / 60000.0));
• In addition, the reservoir size can be updated via a reconnect using the response in the connect flow (like txn service above)

SpanEventsServiceImpl

Initial Config

• The configured reservoir max size is configured via the span_events.max_samples_stored setting
  • The max size for this setting appears to be 10000 (enforced by the back end)
  • This setting is sent to the collector in the connect message
  • The response payload contains a span_event_data object which will specify the maximum number of events stored per harvest cycle in the span_event_data.max_samples_stored property.

    "span_event_data":{"report_period_in_seconds":60,"max_samples_stored":100}

Updating the Setting During Runtime

• Updating the max_samples_stored value in the yml does update the value in the AgentConfig object, however the max_samples_stored value is not updated in the service itself so the harvest cycle reservoir size is not changed
• Forcing a re-connect of the agent to the back end will basically perform the "initial config" flow, which will update the max_samples_stored in the service correctly.

InsightsServiceImpl

Initial Config

• The configured reservoir max size is configured via the custom_insights_events.max_samples_stored setting
  • The max size for this setting appears to be 100000 (enforced by the back end)
  • This setting is sent to the collector in the connect message
  • The response payload contains a harvest_limits object which will specify the maximum number of events stored per harvest cycle in the harvest_limits.custom_event_data property. This will be the max_samples_stored value devided by 12.

    "harvest_limits":{"error_event_data":8,"log_event_data":500,"analytic_event_data":833,"custom_event_data":833}

Updating the Setting During Runtime

• Updating the max_samples_stored value in the yml does update the value in the AgentConfig object, however the max_samples_stored value is not updated in the service itself so the harvest cycle reservoir size is not changed
• Forcing a re-connect of the agent to the back end will basically perform the "initial config" flow, which will update the max_samples_stored in the service correctly.