Refactor Data Sources Logic to Reduce Redundancy

[maxachis]

Data sources logic as it currently exists is redundant, with multiple endpoints performing considerably similar functions and operating out of different endpoints.

To give an example, the data-sources-need-identification endpoint and the data-sources-map endpoints all retrieve variants of the data retrieved via data-sources. However, considerable portions of the backend logic are separate where they could be consolidated for easier maintainability. A few ideas include:

• Creating a view which various data source queries could draw from, rather than duplicating the SQL queries with their joins.
• Creating more dynamic queries that enable requesting only a certain set of columns, rather than hardcoding different column combinations in separate lists or queries.
• Consolidate endpoints into fewer endpoints with more parameters to specify different logic (for example, whether to get approved or unapproved endpoints).

This one is more open ended, motivated more by an awareness of redundancies in the backend, rather than any specific prescription.

TODO

• [ ] Create an issue listing all changes (including this one) that are likely to break the frontend

[josh-chamberlain]

This is a good idea—probably made easier by the database_client?

[maxachis]

This is a good idea—probably made easier by the database_client?

Oh yes, very much easier. The separation of concerns makes it much easier to consolidate at the middleware, database, or even Resource level.

[maxachis]

Implementation Notes

Some column names might be changed, which could break the frontend

When creating a view in SQL (or at least PostgreSQL), all column names must be unique. Previously, some column names were the same (for example, both data_sources and agencies have a last_approval_editor, so queries which selected both had two columns named last_approval_editor). That can't be done in a view, so they'll need to be modified in some way. Any code which is expecting an exact match on those columns will break.

(In some cases, such as last_approval_editor, we may want to eventually normalize them to a separate table called approval_information or something like that, which can indicate approval/rejection information for all things that we can approve or reject, including agencies and data sources).

Known Column Name Changes:

./data-sources

• name column changed to data_source_name
• submitted_name column changed to data_source_submitted_name
• last_approval_editor changed to data_source_last_approval_editor