search

amirziai / flatten

Flatten JSON in Python
https://pypi.python.org/pypi/flatten_json
MIT License
541 stars 98 forks source link

flatten_preserve_lists is failed to flatten json data which includes an array with more than 10 rows #69

Open kanelee7512 opened 3 years ago

kanelee7512 commented 3 years ago

I have a json data, the orderItems has more than 10 items. it looks like: { "orderHeader":{ "orderEvent":"STATUS_UPDATE_IN", "orderNumber":"099432727" }, "orderItems":[ { "itemLineNumber":1, "itemSku":3822, "itemQuantity":468 }, { "itemLineNumber":2, "itemSku":8805, "itemQuantity":414 }, { "itemLineNumber":3, "itemSku":10045, "itemQuantity":24 }, { "itemLineNumber":4, "itemSku":10150, "itemQuantity":24 }, { "itemLineNumber":5, "itemSku":10212, "itemQuantity":36 }, { "itemLineNumber":6, "itemSku":10218, "itemQuantity":24 }, { "itemLineNumber":7, "itemSku":10224, "itemQuantity":84 }, { "itemLineNumber":8, "itemSku":10226, "itemQuantity":60 }, { "itemLineNumber":9, "itemSku":10227, "itemQuantity":42 }, { "itemLineNumber":10, "itemSku":10242, "itemQuantity":84 }, { "itemLineNumber":11, "itemSku":10444, "itemQuantity":12 }, { "itemLineNumber":12, "itemSku":10507, "itemQuantity":12 }, { "itemLineNumber":13, "itemSku":10583, "itemQuantity":6 }, { "itemLineNumber":14, "itemSku":11661, "itemQuantity":396 }, { "itemLineNumber":15, "itemSku":11693, "itemQuantity":48 }, { "itemLineNumber":16, "itemSku":11776, "itemQuantity":24 }, { "itemLineNumber":17, "itemSku":11811, "itemQuantity":42 }, { "itemLineNumber":18, "itemSku":11927, "itemQuantity":24 }, { "itemLineNumber":19, "itemSku":12195, "itemQuantity":732 }, { "itemLineNumber":20, "itemSku":12334, "itemQuantity":24 } ] }

rows = flatten_preserve_lists(json_data, separator='.', max_depth=5, max_list_index=100) len(rows) Out[13]: 11

after flatten, it should be 20 flatted rows. I did some troubleshooting and found that there is a bug in your code.

  global_max_record = int(max(list(
      list_prebuilt_flattened_dict.keys())))

global_max_record is always 9 once list_prebuilt_flattened_dict.keys reach '10'. So you cannot generate more than 11 rows. keys() are '0','1','2','3','4','5','6','7','8','9','10'. max() function always get '9' among the keys because it get the lexicographically-largest value from the list.

i changed it to: global_max_record = int(max(list( list_prebuilt_flattened_dict.keys()), key=int)) then it works.

rows = flatten_preserve_lists(json_data, separator='.', max_depth=5, max_list_index=100) len(rows) Out[16]: 20

amirziai commented 3 years ago

thanks for reporting. feel free to create a PR.