[Enhancement] Display cluster-level information in the ReactFlow workspace

ohltyler commented 2 months ago

The current view of ingest and search pipelines in the ReactFlow workspace is ambiguous on where/when these resources are created. A few ways this can be improved.

(first goal) Where the resources are created. We should display the cluster (local or remote) within the workspace in a visual way. Perhaps a larger parent node encapsulating both the ingest and search flows (see example below). At the least, have more helpful text indicating that these pipelines are being built within the specified cluster.

(stretch goal): When the resources are created. Users will iteratively build out these pipelines, and we should reflect what is created only after the users create them, not just based on the UI configuration. This could mean showing the pipelines as greyed-out/disabled, or not shown at all, until they are concretely provisioned.

Example:

dblock commented 1 month ago

[Catch All Triage - 1, 2, 3, 4]

ohltyler commented 3 weeks ago

Another potential option for Item 2 is adding labeling/status per pipeline (or per-component) indicating if it has been provisioned or not. Overall, the "Preview" is supposed to show the proposed configuration, and does not need to be one-to-one parity with what is provisioned.

kamingleung commented 1 week ago

@ohltyler What are the statuses and states we have for Ingest/Search pipeline? Do the two pipelines have different states?

ohltyler commented 1 week ago

For each pipeline individually, we currently track:

If it has been provisioned or not
If the form differs than what's been provisioned (unsaved "provision" changes - e.g., configure and provision processor A, configure processor B, but not yet provisioned processor B)

Further per-processor granularity is available for ingest, and for search, there is ongoing work for that - see https://github.com/opensearch-project/OpenSearch/issues/14745. I'd consider per-processor state a P2+-type item based on the backend dependencies.

kamingleung commented 6 days ago

For First goal, we can leverage the existing multiple data source picker on the top menu bar. It displays which cluster this workflow and resources are belonging to:

kamingleung commented 6 days ago

For the Second goal, we can consider displaying statuses to the pipelines.

We should dive deep into the different possible states and status before implementing this.

One area to display this is in the preview of a pipeline like this with OuiHealth:
We can also consider displaying the statuses in the resources lists: we can add a Status column to the resource table:

kamingleung commented 6 days ago

To help exploring the states, I am mapping out the lifecycle of the pipeline as users are working through the workflows:

Proposing: Pipeline status:

Inactive: The pipeline has not been created.
Provisioning: When the pipeline is building or updating. (Are we able to get this state? How long does it usually take to build?)
Active: The pipeline is running.

Save status

Pending changes: Perhaps we can show a secondary status for tracking changes that has not been provisioned yet.

ohltyler commented 6 days ago

Thanks Ming. Regarding the ‘Provisioning’ state, it will be quite immediate, in my opinion I don’t think we want to / should show. It would just flash for a moment and could cause user confusion. The rest of those states make sense to me.

From: "Ka Ming L." @.> Reply-To: opensearch-project/dashboards-flow-framework @.> Date: Thursday, November 7, 2024 at 7:16 PM To: opensearch-project/dashboards-flow-framework @.> Cc: "Ohlsen, Tyler" @.>, Mention @.***> Subject: Re: [opensearch-project/dashboards-flow-framework] [Enhancement] Display cluster-level information in the ReactFlow workspace (Issue #377)

To help exploring the states, I am mapping out the lifecycle of the pipeline as users are working through the workflows: image.png (view on web)https://github.com/user-attachments/assets/96ab8380-dda5-438e-b05a-1a90aa3fdf36

Proposing: Pipeline status:

Inactive: The pipeline has not been created.
Provisioning: When the pipeline is building or updating. (Are we able to get this state? How long does it usually take to build?)
Active: The pipeline is running.

Save status

Pending changes: Perhaps we can show a secondary status for tracking changes that has not been provisioned yet.

— Reply to this email directly, view it on GitHubhttps://github.com/opensearch-project/dashboards-flow-framework/issues/377#issuecomment-2463675419, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOZ55TN7UTNVXMK3C6VFHRDZ7QUHHAVCNFSM6AAAAABOGCR4USVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRTGY3TKNBRHE. You are receiving this because you were mentioned.Message ID: @.***>

opensearch-project / dashboards-flow-framework

[Enhancement] Display cluster-level information in the ReactFlow workspace #377