We will use an interoperability layer between the NWS public API and the website

[greg-does-weather]

Context

The public NWS API is primary data source for all observation and forecast data. The way it is architected, it requires multiple roundtrip REST requests to obtain all of the information needed to display our website. The data that comes back from the API is not precisely the structure that works best for our website, so we do some amount of processing on it before using it to render the site. We currently cache API responses with Drupal caching mechanisms, but we do not cache the processed results.

Decision

We will introduce a new API interop layer that sits between Drupal and the public NWS API. This layer will be built in Node.js to take advantage of its multithreading support and low memory footprint.

The interop layer will present a single endpoint that returns all of the data relevant to rendering the website. Behind the scenes, the interop layer will make the necessary calls to the public API and the local geospatial database tables. The interop layer will also handle most of the processing of the raw data into structures that directly serve the website.

Consequences

Positive

• In the immediate term, our Drupal modules for fetching data will be dramatically simplified if not entirely replaced. Because Drupal is single-threaded, offloading this functionality should have a positive impact on page load times.
• In the longer term, the interop layer can add caching for processed data, allowing much faster responses to page requests.
• The most significant portion of backend code is moved out of PHP and into Javascript, a language that is easier to support by 18F staff. Javascript also has very broad industry support (as does PHP), so this should not have any downstream negative consequences.
• We should have more flexibility over how we handle error cases. Currently our response to errors from the API is to simply show a red banner for whichever data component failed. Because of how the site was built with intertangled blocks, an error in one data component could cascade into errors for other components (e.g., if the forecast causes an error, we also don't show alerts). With the interop layer, we have an opportunity to separate the data components more cleanly and have more fine-grained error handling.

Neutral

• Additional cloud resources will be required. It seems likely to be a relatively small impact since the total amount of computation work will remain roughly the same.

Negative

• The interop layer introduces an additional programming language and runtime environment, increasing complexity.

[github-actions[bot]]

This issue appears to be a draft or in-progress ADR. When it is completed and accepted, add the ADR: Accepted label and close the issue. This will start a process to create the ADR document and add it to the repo automatically. If you close the issue before adding the label, simply re-open and re-close it.

[jamestranovich-noaa]

Can we add something about how errors are handled -- e.g., will the beta.weather.gov site degrade gracefully in the face of API errors? How can the interop layer help (or not) with error handling of external sources, and in turn, how are these errors going to be reflected on beta.weather.gov?

I'm thinking of the user experience when someone visits beta.weather.gov and the weather.gov API is acting funky. What happens? Can we provide a nicer "something is going wrong behind the scenes" experience?

[greg-does-weather]

😅 I'll add it as a potential consequence. Exactly how we'll do that, however, is probably a separate body of work. Currently we show an error banner for any components of data that we couldn't compute. I think that also be our expectation for the interop layer in this first pass, but I think you're right to raise this because the interop layer will hopefully 🤞 give us more options for error handling and graceful degradation.

[jamestranovich-noaa]

I think we're good to go on the error front, especially as we have template updates to display errors from the api interop layer (and we have NR logging now besides). @eric-gade what sayest thou? Does this ADR look good?

[eric-gade]

This looks good to me!