edcorcoran / internet_archive

0 stars 0 forks source link

Make database incremental #1

Open edcorcoran opened 2 weeks ago

edcorcoran commented 2 weeks ago

Instead of blowing away the database and starting over each time, I should retain the database. This will also speed up querying IA because I can first query the DB to see if that item_id exists. If it does, I just skip. If it doesn't, then I query the item and add it to the DB.