DataJunction / dj

A metrics platform.
http://datajunction.io
MIT License
34 stars 15 forks source link

Improve node status display and fix status update bugs #981

Closed shangyian closed 4 months ago

shangyian commented 4 months ago

Summary

This PR makes a number of improvements related to node status.

Fix node update status propagation bug

We were seeing an issue with node update propagation, where after a node is updated, its downstreams' statuses were often not updated to the right status.

The primary cause of this issue is that we weren't updating the downstreams in the right order. We need to topologically sort the downstreams, such that if node A queries from node B, B always comes before A in the ordering. We were instead sorting by level order, which is the wrong ordering and may lead to children being processed before parents.

A secondary issue is that we were only updating statuses and not columns. If a node's query changes, it may cause changes to downstream nodes' columns, which also needs to be saved in order for downstream nodes to pick up on them.

Display detailed node status

This change modifies the node validation endpoint to include a list of errors if the node's status is invalid.

We update the node info UI to (a) always revalidate the node (this can be changed later), and (b) show detailed status information. If a node is valid, we continue to display just the checkmark, but if a node is invalid, we additionally display a list of errors that explain why the node is invalid. This allows users to understand why a node's status is what it is, so that they can fix issues to correct it.

Screenshot 2024-05-04 at 10 38 06 AM

Test Plan

Deployment Plan

netlify[bot] commented 4 months ago

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
Latest commit 45a972f4109fe22db4f8ba789c4692cfcb79185a
Latest deploy log https://app.netlify.com/sites/thriving-cassata-78ae72/deploys/66387b52b9d90300078b0eb2