cityofaustin / atd-data-tech

Austin Transportation Data & Technology Services
17 stars 2 forks source link

Migrate 311 > Knack integration to PostgREST #7405

Open johnclary opened 2 years ago

johnclary commented 2 years ago

TLDR: Replace the current 311 > Knack-Proxy integration with a central PostgREST datastore.

We have been thinking internally about a better way to manage our ESB integration. As you know, we have periodic outages for which we haven't been able to identify a solution. We experienced one of these outages over the weekend, in fact.

We're also thinking about how we can more efficiently scale our integration to incorporate additional Knack applications as well as manage internal reporting needs.

The solution we've zeroed in on is to have a centralized PostgRESTdata store of 311 SRs that sits between our applications and the ESB. This will address two main issues.

Outages

The ESB will interface directly with our own database API. We suspect that the current outages stem from latency issues between the ESB, our proxy service, and the Knack API. We unfortunately have very little control/visibility of the Knack API.

Under this new scheme, the ESB can make a POST request directly to the data store. The database technology we have in mind is extremely resilient and fast—we currently use it for thousands of transactions/day.

Scalability

This intermediary data store will allow us to make modifications to our Knack apps, and add new ones, without needing modify the ESB integration. You will only need to maintain one set of mappings, tokens, etc. for all of our integration needs.

Todo

Current state Future state

johnclary commented 2 years ago

I contacted Omar and Robert for their feedback. Will schedule a meeting with them once I hear back.