With the current action framework, output data is collected in memory and then written at once to an output file or console using the write instruction. At the moment, the write instruction overwrites any existing output file; there's no support for appending to an existing file.
We should consider adding an explicit append: true|false property to write instructions, to allow existing files to be appended to. If we do so, we also need to have an ability to clear (contents of) an existing file, i.e., on action start we'd clear (contents of) any existing file, and then in later steps append data to this clean file.
In addition or alternative to the above, we should also consider adding support for record writers that allow for appending individual records in various output formats (JSON, CSV, YAML, table, ...), similar to the existing fcli output framework. This serves 2 main purposes:
Data can be gradually written during action execution, rather than having to collect all data in memory first, and therefore results in lower memory consumption.
Data can be easily written to various output formats, for example actions won't need to manually construct CSV records and then use generic text output to write these records.
Potentially we could reuse functionality provided by the fcli output framework for implementing this feature.
If we implement this functionality, ideally we should also provide support for writing partial output to temporary files, and then combining these files to generate the full output. For example, for SARIF files, we separately collect rules and results, which now get combined in memory when writing the output. Ideally, we'd have separate JSON record writers for rules & results, and then stream the contents of those temporary files into the final SARIF output file.
With the current action framework, output data is collected in memory and then written at once to an output file or console using the
writeinstruction. At the moment, thewriteinstruction overwrites any existing output file; there's no support for appending to an existing file.We should consider adding an explicit
append: true|falseproperty towriteinstructions, to allow existing files to be appended to. If we do so, we also need to have an ability to clear (contents of) an existing file, i.e., on action start we'd clear (contents of) any existing file, and then in later steps append data to this clean file.In addition or alternative to the above, we should also consider adding support for record writers that allow for appending individual records in various output formats (JSON, CSV, YAML, table, ...), similar to the existing fcli output framework. This serves 2 main purposes:
Potentially we could reuse functionality provided by the fcli output framework for implementing this feature.
If we implement this functionality, ideally we should also provide support for writing partial output to temporary files, and then combining these files to generate the full output. For example, for SARIF files, we separately collect
rulesandresults, which now get combined in memory when writing the output. Ideally, we'd have separate JSON record writers for rules & results, and then stream the contents of those temporary files into the final SARIF output file.