Open jrasband-dev opened 1 month ago
@ritchie46 Super excited for the new json_normalize function in polars 1.0.0! Would love to hear your thoughts on this as a further development item.
For additional context, a little over a year ago I put together an in depth video on how the pandas json_normalize function works (something you don't always get from just reading the documentation). https://www.youtube.com/watch?v=5JyLCsQ2JNc
~Just on the topic of that specific dataset: I think Polars may have an issue with the first example as the lists are of mixed types.~
~You may need to manually transpose them (e.g. with zip):~
>>> pl.DataFrame(zip(*data["data"]), orient="col")
shape: (191_407, 28)
┌───────────────┬───────────────┬──────────┬────────────┬───┬─────────────┬───────────┬───────────┬───────────┐
│ column_0 ┆ column_1 ┆ column_2 ┆ column_3 ┆ … ┆ column_24 ┆ column_25 ┆ column_26 ┆ column_27 │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 ┆ i64 ┆ ┆ str ┆ str ┆ str ┆ str │
╞═══════════════╪═══════════════╪══════════╪════════════╪═══╪═════════════╪═══════════╪═══════════╪═══════════╡
│ row-d77h.fy7… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53033006500 ┆ 3009 ┆ 7 ┆ 4 │
│ row-95h2~76n… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53077002201 ┆ 2746 ┆ 4 ┆ 22 │
│ row-7cud~im3… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53033029306 ┆ 3009 ┆ 9 ┆ 46 │
│ row-3ey5-tfy… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53061052107 ┆ 3213 ┆ 1 ┆ 45 │
│ row-h6ea_9na… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53077000902 ┆ 2746 ┆ 4 ┆ 20 │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ row-j544~qpj… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53053073302 ┆ 3210 ┆ 10 ┆ 15 │
│ row-ertp~645… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53053070206 ┆ 3210 ┆ 8 ┆ 15 │
│ row-5wyy~zwx… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53033032601 ┆ 3009 ┆ 8 ┆ 6 │
│ row-mevd_k52… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53027001100 ┆ 2983 ┆ 6 ┆ 26 │
│ row-6n89~ue2… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53005010811 ┆ 2955 ┆ 4 ┆ 12 │
└───────────────┴───────────────┴──────────┴────────────┴───┴─────────────┴───────────┴───────────┴───────────┘
Does record_path
do something differently compared to indexing the object directly?
>>> pl.json_normalize(data["meta"]["view"]["columns"]).columns
['id',
'name',
'dataTypeName',
'fieldName',
'position',
'renderTypeName',
'flags',
'description',
'tableColumnId',
'format.align',
'computationStrategy.source_columns',
'computationStrategy.type',
'computationStrategy.parameters.region',
'computationStrategy.parameters.primary_key']
I didn't think of indexing the object directly. That definitely works when getting the column names.
I suppose the one advantage of the record_path
argument in pandas is that when you specify the "data" key, it will correctly parse and read the data into a DataFrame. Whereas polars requires the use of the zip function that you mentioned and indexing doesn't work.
import json
import polars as pl
f = open("../../Datasets/Electric_Vehicles.json",'r')
data = json.load(f)
df = pl.json_normalize(data["data"])
df
Output
shape: (28, 191_407)
┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐
│ column_0 ┆ column_1 ┆ column_2 ┆ column_3 ┆ … ┆ column_19 ┆ column_19 ┆ column_19 ┆ column_1 │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ 1403 ┆ 1404 ┆ 1405 ┆ 91406 │
│ struct[1] ┆ struct[1] ┆ struct[1] ┆ struct[1] ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ ┆ ┆ ┆ ┆ ┆ struct[1] ┆ struct[1] ┆ struct[1] ┆ struct[1 │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ] │
╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
│ {null} ┆ {null} ┆ {null} ┆ {null} ┆ … ┆ {null} ┆ {null} ┆ {null} ┆ {null} │
└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘
And using the zip function certainly isn't hard to do. Now that you've shared that code snippet it makes a lot of sense, but perhaps its more of an ease of use situation for users. I don't know that I would have known how to read the data in correct if you didn't share that with me.
Ah.. I see.
So without it, it behaves similar to the Polars example:
>>> pd.json_normalize(data["data"])
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
0 {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {}
1 {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {}
[191407 rows x 28 columns]
I was being silly with the zip
:
>>> pl.DataFrame(data["data"], orient="row")
shape: (191_407, 28)
┌───────────────┬───────────────┬──────────┬────────────┬───┬─────────────┬───────────┬───────────┬───────────┐
│ column_0 ┆ column_1 ┆ column_2 ┆ column_3 ┆ … ┆ column_24 ┆ column_25 ┆ column_26 ┆ column_27 │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 ┆ i64 ┆ ┆ str ┆ str ┆ str ┆ str │
╞═══════════════╪═══════════════╪══════════╪════════════╪═══╪═════════════╪═══════════╪═══════════╪═══════════╡
│ row-d77h.fy7… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53033006500 ┆ 3009 ┆ 7 ┆ 4 │
│ row-95h2~76n… ┆ 00000000-000… ┆ 0 ┆ 1718719976 ┆ … ┆ 53077002201 ┆ 2746 ┆ 4 ┆ 22 │
└───────────────┴───────────────┴──────────┴────────────┴───┴─────────────┴───────────┴───────────┴───────────┘
So it seems record_path
may also enforce what Polars calls orient="row"
Exactly. So I guess the question would be: is it worth building out a new parameter for json_normalize?
It looks like it works fine when you index the object directly and when you have mixed types you can use the "row" orient method. Seems to accomplish the same goal.
Description
Request
Would be great if we could select a path in the JSON file to start parsing records at. Pandas has this functionality (see below).
Usage
data source: https://catalog.data.gov/dataset/electric-vehicle-population-data
Using Polars
OUTPUT
Using Pandas with record path argument
OUTPUT
Pandas with nested paths
OUTPUT