[experiment] Polars-based JSON normalizer

dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

https://dlthub.com/docs

Apache License 2.0

2.65k stars 175 forks source link

[experiment] Polars-based JSON normalizer #1793

Open jorritsandbrink opened 2 months ago

jorritsandbrink commented 2 months ago

Hacked-together rudimentary Polars-based JSON normalizer as experiment to learn what it takes to replace current normalizer and which speedups to expect.

approach: process single JSON line with original normalizer to infer/create dlt schema, then process all JSON lines with Polars to create normalized files
basic functionality seems to work, but there's no feature parity with current normalizer
Polars code can probably be optimized to run faster

netlify[bot] commented 2 months ago

Deploy Preview for dlt-hub-docs ready!

Name	Link
Latest commit	79c03ac62cbe31599cea3dcc37abaa371618eabd
Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/66df0f8b79097e0008c3d16b
Deploy Preview	https://deploy-preview-1793--dlt-hub-docs.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

dlt-hub / dlt

[experiment] Polars-based JSON normalizer #1793

✅ Deploy Preview for dlt-hub-docs ready!

Deploy Preview for dlt-hub-docs ready!