Support custom function datasource

[Janpot]

The REST datasource is on a pathway of configuration bloat. There's just so many possible combinations of urls, path parameters, headers, query parameters, authentication, method, preprocessing, postprocessing, error handling,... as well as, no clear way to combine or chain requests.

We should allow the user to express their rest calls in plain javascript. e.g. allow the user to write queries of the form:

export default async function (input) {
  const response = await fetch(`https://dog.ceo/api/breed/${input.selectedBreed}/images/random`)
  if (response.ok) {
    return response.json()
  }
  throw new Error(`HTTP ${response.status}: ${response.statusText}`)
}

This would execute on the server and user will be presented with UI where they can define input to be passed from the page to the server function. The user would have full control over how urls are created, pre/post-processing, chaining multiple requests. Error handling,...

We can allow them to set up authentication headers per origin and have the server automatically use them when making requests.

We can also add a wizard that asks for a few basic parameters and generates the function code to start from.

This would make the following tickets obsolete:

• https://github.com/mui/mui-toolpad/issues/611
• https://github.com/mui/mui-toolpad/issues/610

benchmark:

• https://pipedream.com/
• https://segment.com/docs/connections/functions/source-functions/

Todo: evaluate methods of running untrusted code securely on the server side:

• https://github.com/laverdet/isolated-vm (importing modules will require a bundling step, e.g. using esbuild)
• https://github.com/justjake/quickjs-emscripten (difficult and error-prone to create bindings to serverside apis)
• https://github.com/endojs/endo/tree/master/packages/ses (not sure how secure is the polyfill + how to protect against memory overflow, infinite loop,...?)
• ...

[apedroferreira]

+1, sounds like a great and important feature, it would also easily allow for a lot of integrations

[apedroferreira]

still, maybe having a simple, limited, non-bloated way of making REST requests wouldn't be bad either

[prakhargupta1]

Interesting! As it has the potential to cover much of user's expectations around customization. It seems some WRITE use cases would also be possible through this.

[Janpot]

Sufficiently implemented by https://github.com/mui/mui-toolpad/pull/641 and follow-up PRs