Datadance

Simple, open-source JSON data transformation tool

Playground • API Reference • Docs

DataDance is a versatile data processing package that makes handling JSON transformations straightforward and efficient. Our package accepts JSON input and allows you to define transformations using a code-like format. Provide your data and transformation rules, and DataDance will process them to deliver the desired output.

Install

Install DataDance from npm:

npm install datadance

Then you can import it in your code :

import { transform } from "datadance";

or

const datadance = require("datadance");
const transform = datadance.transform;

You can use the transform method :

async function test() {
    var res = await transform({
        input: { x: 2 },
        transforms: [{ x: "input.x+8" }],
        settings: {
            merge_method: "overwrite",
        },
    });
    console.log(res); // { x : 10 }
}
test();

Local Setup

• Datadance uses Deno, so you need to install Deno in your local.
  
  
  • On Linux :

    curl -fsSL https://deno.land/install.sh | sh

  • On Windows :

    irm https://deno.land/install.ps1 | iex

• Clone the Datadance repo locally :

  git clone https://github.com/yakshavingdevs/datadance.git && cd datadance

• And then start the server :

  deno task start

• Then you can either play with the API http://localhost:8000/process or experience playground http://localhost:8000/design/build.

Where Datadance Shines?

In many ETL pipeline scenarios, executing third-party or user-provided code can be both an operational and security challenge. Managing and isolating this code often becomes a complex task. DataDance solves this problem by offering a unique solution where third-party or user logic is expressed as DataDance transforms. These transforms are easy to maintain and eliminate the need for a traditional programming language shell. Internally, DataDance uses a parser that executes the specified transforms on the input JSON and produces an output JSON, ensuring both simplicity and security.

Internals

• Parser & Evaluation Engine : Datadance at its core uses MozJexl expression language. We can use expression language to parse a given expression and evaluate it.
• Backend : Deno + Fresh
• Frontend: Preact + Bootstrap + Ace Editor

CLI

• Compiling the binary :

  deno task compile # builds the binary

• Executing the binary :

  ./bin/datadance -i '{"hello": "world"}' -t '[{"also": "\"hello \" + input.hello"}]' -s '{"merge_method": "overwrite"}'
  

  Expression language reference

• You can check the Mozjexl expression language reference here : https://github.com/mozilla/mozjexl/blob/master/README.md

Example

• You can make POST call to the http://localhost:8000/process with below JSON :

  {
  "input": {
    "name": {
        "first": "Malory",
        "last": "Archer"
    },
    "exes": [
        "Nikolai Jakov",
        "Len Trexler",
        "Burt Reynolds"
    ],
    "lastEx": 2
  },
  "transforms": [
    {"lastEx": "input.lastEx + 5"}
  ],
  "settings": {
    "merge_method": "overwrite"
  }
  }

• And the output will be :

  {
    "name": {
        "first": "Malory",
        "last": "Archer"
    },
    "exes": [
        "Nikolai Jakov",
        "Len Trexler",
        "Burt Reynolds"
    ],
    "lastEx": 7
  }

Contributing

Development of Datadance happens in the open on GitHub. You can read the contributing guide here : CONTRIBUTING.md.

License

MIT