Divide Large JSON Files into Small Script/Module

bhargavnova commented 1 year ago

Objective

Develop a Python script or module that allows users to divide large JSON (JavaScript Object Notation) files into smaller, more manageable chunks. This tool will help users process and work with large JSON datasets efficiently.

Features

Chunk Size Specification: Allow users to specify the desired size of each resulting JSON chunk. (KB, MB....)
Output File Naming: Provide options to customize the naming convention for the generated JSON chunks.
Handle Nested JSON: Ensure that the script can handle cases where the JSON data has nested structures.
Preserve Valid JSON: Guarantee that each generated chunk remains a valid JSON object.

Example Usage

import json_divider

# Divide a large JSON file into smaller chunks
json_divider.divide('large_data.json', chunk_size=1000, output_folder='output_folder')

# Customize the naming convention for output files
json_divider.divide('large_data.json', chunk_size=500, output_folder='output_folder', custom_names=['part1.json', 'part2.json'])

Difficulty: Beginner/Intermediate

Tags: Python, JSON, Data Processing, Chunking

Additional Information

Consider using libraries like json in Python for handling JSON operations.
Ensure that the script provides informative error messages in case of invalid JSON input or other issues during the dividing process.

Contribution Guidelines

The updated guidelines can be found here.

Note:

folder_name: divide_json
script_name: divide_json

aecuto commented 12 months ago

i'll take this

bhargavnova commented 12 months ago

Sure!

aecuto commented 12 months ago

too hard for me :)

bhargavnova commented 12 months ago

okay @aecuto , so it will be like a list of dictionaries if any JSON file has lots of rows, that can be divided into small batches, that was the concept, if anyone wants to take over, just comment.

Happy Coding!

aecuto commented 12 months ago

the json have many formats such as [{},{}] {keys: []} {} that why tis too hard

bhargavnova commented 12 months ago

as of now, let's go with the first, because that's common when it comes to large file,

aecuto commented 12 months ago

lets me try aging :)

bhargavnova commented 12 months ago

Hi @aecuto, Thanks for your contribution,

You can take a look at other open issues or, you can also submit one of your own code/ideas.

Thanks :) Happy coding!

bhargavnova / python-helper-modules