Manual source refresh

[D0nPiano]

Description

This is not meant to be merged (like this at least), but a proof of concept for refreshing with an extra url /refresh.

• Only adds new sources (that are not yet in the database)
• Option to refresh existing items using override param refresh/?override
• slightly ajus

Limitations:

• Just imports function from @opowell28 (threw out pandas + csv)
• It doesn't add locations or translations
• locations / tags / translations might be overwritten when refreshed with overwrite param
• items that are manually edited in source/text/interface are not detected by override function and will be duplicated after a refresh

Checklist

• [x] I installed pre-commit hooks (pip install pre-commit && pre-commit install), and ran black on my code.
• [x] Please squash merge this pull request (uncheck if you'd like us to merge as multiple commits)

[j-bennet]

Note to self: look at

https://django-postgres-extra.readthedocs.io/en/master/conflict_handling.html

for upsert functionality.

[j-bennet]

@D0nPiano I made some changes to your branch here, to hopefully make this more production-ready. The most important change, I'm using external_id (which should be unique) to ensure items are not duplicated.

• used django-postgres-extra to be able to update or ignore on conflict
• added test with mock data for refresh endpoint
• renamed override=true to overwrite=true
• batch insert items, instead of one-by-one
• if a batch failed to insert, exception will be returned as part of errors response item.
• it's not possible to know which items were created and which were updated, but that's not terribly important anyway.

Example response:

{
        "liveuamap": {
            "detail": "Refresh completed",
            "errors": {"exceptions": [], "total": 0},
            "overwrite": "true",
            "processed": 3,
        }
    }

Please take a look.