Use a separate table to ingest data, then merge

[jacopofar]

The current ingestion process populates the card table from scratch. This has two problems:

• it takes a long time (20 minutes to many hours), during which the app is unusable
• it doesn't account for updates, only insertion
• if the table is truncated and recreated and the process is interrupted, it's left like that

it will get worse when adding lemmatization, which will increase the size and time.

So, a solution is to first create a staging table with the data and only after that merge it with the current one using SQL transactions.

[jacopofar]

Done, together with the new partitioning it makes it a bit faster but most importantly allows incremental update while the app is running