Retool data source approval: check for duplicates

[josh-chamberlain]

Context

Follows #74

We have a retool interface for data source intake. Generally, we're taking steps to reduce the number of duplicates that make it into this pipeline. However, if one makes it in, I'd like to save the reviewer time.

Requirements

• [ ] check whether the URL is a duplicate using our endpoint; if it is, add a text label saying "Duplicate URL(s) found with <status, status> status"
• [ ] it would then be incumbent upon the checker to locate the other source(s) and see if any new information has been added. but, information is power!

Open questions

• Does this make the workflow notably slower? Can we do it in the background with a little "checking for duplicates..." message?

[maxachis]

So I've found myself with an interesting dilemma: the unique url checker was designed to check for unique urls already in the database. The issue, at least right now, with the data source intake form is that anything in the intake form is, by definition, already in the database -- just not approved.

Now there can be some workarounds to this -- I could, for example, add a filter to check only for ones which are approved or rejected, and make that customizable.

So that if the user wants this checked against all data sources, they can have that. But if they only want it checked against approved data sources, they can have that as well.

What do you think, @josh-chamberlain?

[josh-chamberlain]

@maxachis oh, interesting! I suppose that retool could just ignore anything returned with an intake status? I don't think we the filter—that sounds a little fancy.

[maxachis]

@josh-chamberlain Well, that took less time than I thought it would. Done!

[josh-chamberlain]

@maxachis excellent, thank you!