Data source implementation currently requires a full search of the data source table linked with every content table in order to add an item without risking duplication. Instead of a sequential reference ot the data_source table, use a 128- or 256-bit SHA-1 as key in each substantive table (i.e. discourse part, content, contribution, annotation, user, etc), so that a simple upsert can be done on new data without checking if the data source entry exists already for every kind of entity.
Data source implementation currently requires a full search of the data source table linked with every content table in order to add an item without risking duplication. Instead of a sequential reference ot the data_source table, use a 128- or 256-bit SHA-1 as key in each substantive table (i.e. discourse part, content, contribution, annotation, user, etc), so that a simple upsert can be done on new data without checking if the data source entry exists already for every kind of entity.