OpenFn / kit

The bits & pieces that make OpenFn work. (diagrammer, cli, compiler, runtime, runtime manager, logger, etc.)
11 stars 9 forks source link

Runtime: workflows throw for circular references on state #219

Closed josephjclark closed 1 year ago

josephjclark commented 1 year ago

Using the salesforce adaptor in a workflow, we see:

[CLI] ✘ TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'ClientRequest'
    |     property 'socket' -> object with constructor 'TLSSocket'
    --- property '_httpMessage' closes the circle
[CLI] ✘ Took 3.401s.

The salesforce adaptor is writing some http request stuff to state. We then try and clone that, before passing it into the next job, which fails.

I think we need to use fast-safe-stringify instead of JSON.stringify to clone state

This job using http lets us reproduce:

{
  "start": "job1",
  "jobs": {
    "job1": {
      "adaptor": "http",
      "expression": "get('https://api.chucknorris.io/jokes/random');",
      "next": { "job2": true }
    },
    "job2": {
      "adaptor": "common",
      "expression": "fn(s => s)"
    }
  }
}
josephjclark commented 1 year ago

This issue is only occurring on workflows, but I think there's a wider story here.

The runtime right now will return whatever state was output from the last job. This could contain circular references, functions, and other non-serializable stuff.

The CLI can handle this "unclean" output because it runs fast-safe-stringify on the output. So it gets non-serialzable output from the runtime, and deals with it itself.

Really, each expression we execute should return a safely serializable object. Functions removed (maybe stubbed in with [function]. Circular references replaced. So the runtime guarantees a safe result.

Basically fast-safe-stringify needs to move down to the runtime.

josephjclark commented 1 year ago

This is now fixed on the workflow branch: the runtime will always return serializable state, making everyone's life easier.