Implement helper functions to collect Boefje job output information

minvws / nl-kat-coordination

Repo nl-kat-coordination for minvws

European Union Public License 1.2

123 stars 56 forks source link

Implement helper functions to collect Boefje job output information #1198

Open Darwinkel opened 1 year ago

Darwinkel commented 1 year ago

Parent task: https://github.com/minvws/nl-kat-coordination/issues/1069

Requirements

[ ] Should capture the Boefje's errors (stderr)
[ ] Should capture the Boefjes regular output (stdout)
[ ] Should provide a structured, smart way to collect and report stderr and stdout simultanously (to my knowledge, there is currently no meaningful difference between the two channels)

Darwinkel commented 1 year ago

After consulting with @underdarknl, the plan is as follows:

As Boefjes can return multiple raw/mimetype outputs, we can abuse it it to output an additional boefje/info mimetype with info/debug/warning info. This means that no modifications need to be made in Bytes, as boefje_meta already has a many-to-many relationship with boefje_raw.

@underdarknl suggests building a logger facade as part of a small boefje-utils module or similar, and provide boefjes developers with convenient helpers to output boefje/info.

Darwinkel commented 1 year ago

@Donnype rightly noted that introducing Python helper functions (e.g. a boefje.utils logger facade) become an issue when we move to containerized boefjes. Even assuming that those boefjes will be Python-based, they would have to be installed inside the container through pip or something.

Darwinkel commented 1 year ago

We will turn the boefje.utils into a pip package. I'll put some work in building a pipeline to push the tool automatically to pypy.

@ammar92 will write a more concrete proposal regarding the formal protocol describing the special mimetypes.

Darwinkel commented 1 year ago

Blocked until the design for containerized boefjes are ready.