Now sparkProperties fields are casted using a predefined map of fields to types
More types are included: int, boolean, file size string, time interval string
executor and driver memory are now casted to integer number of bytes
New config property skip_exceptions. When set to True, execeptions during run processing do not terminate the program. The errors are logged to a separate index for later debug.
New calculated field 'est_peak_memory_usage' includes max input size + max output size among stages.
Updated field storage_memory_usage now compares the 'est_peak_memory_usage' to total storage memory at all executors. (previously only input size was considered)
Closes #41
Now sparkProperties fields are casted using a predefined map of fields to types
New config property skip_exceptions. When set to True, execeptions during run processing do not terminate the program. The errors are logged to a separate index for later debug.
New calculated field 'est_peak_memory_usage' includes max input size + max output size among stages.
Updated field storage_memory_usage now compares the 'est_peak_memory_usage' to total storage memory at all executors. (previously only input size was considered)