search

jdrumgoole / gdelttools

Scripts to load the GDELT data set into MongoDB
Apache License 2.0
7 stars 6 forks source link

Faster mongoimport #8

Closed judy2k closed 2 years ago

judy2k commented 2 years ago

I've modified mongoimport.sh to only run mongoimport once, and all the CSV files are piped in. Makes it quite a bit faster, especially with lots of small files.

I've also added some comments about how to run it, and enabled it to be run from another directory (by tracking down where the script is, and loading the .ff file from there).

judy2k commented 2 years ago

One other change here is that the database name is no longer hard-coded, and so the script should be called with a valid --uri parameter.