Problem

We were passing only dialog values. These Input values are small and did not require too much memory.

After https://github.com/ManageIQ/manageiq/pull/23039, we are now passing in all RequestTask#options, the context has ballooned in size.

Proposed Solution

Input is typically the Output from the previous State in StateHistory, so it may be as simple as dropping Input from all records in StateHistory.

To be complete: for Retries (has RetryCount), Input is the Input from the previous State. For Parallel States, the previous State in StateHistory is not necessarily the previous State. For the first State, Input is the execution#Input

If we deem that Input is easy enough to derive, or if we won't be accessing it, then dropping that should result in a record that is a little over 1/2 the size.

Not sure if we can also drop Output for records where Input == Output.

Dev Math

Context

script = ConfigurationScriptBase.last
script.context.each { |k, v| puts "#{k}: #{v.to_s.length}" }.size
Task:              2
State:           260
Execution:     5,306 # <===
StateHistory: 58,654 # <===
StateMachine:      2

Context Input

All of this data can be attributed to the Input:

script.context["Execution"].each { |k, v| puts "#{k}: #{v.to_s.length}" }.size
Input:             5,107 # <===
EndTime:              20
StartTime:            20
_object_id:            2
_object_type:         52
_manageiq_api_url:     0

State History

Since each history record records the input and the output, this gets quite large. For errors, the Output will change will vary, but for test workflows, the Input tends to be the Output with possibly a few values changed or added:

script.context["StateHistory"].each_with_object({}) { |sh, h| sh.each { |k, v| h[k] ||= 0 ; h[k] += v.to_s.length  } }
=>
{
 "Guid"=>           326,
 "Name"=>           105,
 "Input"=>       35,764, # <===
 "Output"=>      20,537, # <===
 "Duration"=>        70,
 "NextState"=>       87,
 "EnteredTime"=>    180,
 "FinishedTime"=>   180,
 "RetryCount"=>       3,
 "RunnerContext"=>  244
}

For Envelope math:

input_size     = execution.input.size # 5,107
output_size    = input_size           # 5,107
error_size     = 20
total_states   = 9
error_states   = 3
success_states = total_states - error_states # 6

history_memory = success_states * 2 * input_size # 61,284
actual: 58,654

ManageIQ / manageiq-providers-workflows

Reduce Size of ConfigurationScriptBase record #99