[OCU-97] make putil save intermediate results to more than "file://"

[northdpole]

Thanks for the patience and the questions. Replies below and in the RFC

Why do we need this?

• security tools are not always idempotent, hence a workflow that failed for whatever reason cannot always be restarted with the same results
• deduplication enrichers save things to a database, if you fail at the consumer stage everything then is a duplicate but has not been reported
• the ability to restart a workflow from the last successful stage is a core component of any workflow engine

this is the first step at implementing retryable steps

Is it something urgent?

No, future looking for now, necessary after v1.0

How can we prevent the storage backend from being spammed with huge amounts of data?

I have yet to see a security tool that provides huge numbers of data without the tool itself dying trying to export it. We can design for a scale of low Mb.

How is it enabled thoughout a pipeline?

Every single component takes --in and --out parameters, part of the component definition. These parameters are currently paths. In the future they should be urls

What are the new methods that will be added? What are their signatures?

updated the RFC.

writeToDB(dbConnStr, object)error
readFromDBG(dbConnStr)(object,error)

writeToFS(path,object)error
readFromFS(path)(object,error)

What will be the names of the tables where the output will be written?

implementation detail, in the rfc

Do we need to refactor the components or just the SDK?

just the SDK