Omnivore export batches entries, difficult to use with Polyglot. Wrote a merger script.

[apotenza92]

Hi, love your tool, used it to get off Omnivore and onto Raindrop.

One small snag I came across was that Omnivore batches its entires into groups of 20 for some reason. So you end up with a folder that looks like this:

To get around this I just wrote a tiny merger python script that will take all the jsons and make a merged_output.json that then works great with Polyglot. Would be cool to have it integrated into Polygot for Omnivore exports somehow? Like maybe you could point Polyglot to the folder that Omnivore makes and it will automatically merge all these jsons?

import json
import glob

# Initialize an empty list for merged data
merged_data = []

# Get all JSON files in the current directory
json_files = glob.glob("*.json")

for file in json_files:
    with open(file, "r") as f:
        data = json.load(f)

        # Append dictionary or list items from each JSON file to merged_data list
        if isinstance(data, dict):
            merged_data.append(data)
        elif isinstance(data, list):
            merged_data.extend(data)

# Save merged data to a new file
with open("merged_output.json", "w") as f:
    json.dump(merged_data, f, indent=4)

print(
    "All JSON files have been merged into merged_output.json in the specified list format."
)

[tdhooten]

I'm currently working on this, would you be willing to provide me with two of those JSON files to test with?

[ocean]

Love your work on this tool @tdhooten ! And the extension @apotenza92 👏

I posted an Omnivore processing jq command in a gist here though I know that's not directly applicable to you 😁

Found your question though, so here are two of the Omnivore JSON files for you to experiment with 👍

metadata_180_to_200.json metadata_60_to_80.json

[tdhooten]

@ocean Thanks so much for the files!

Native support for uploading multiple files at once has now been released.