search

kedro-org / kedro-viz

Visualise your Kedro data and machine-learning pipelines and track your experiments.
https://demo.kedro.org
Apache License 2.0
672 stars 110 forks source link

Research: Viz usage as a React component #954

Closed tynandebold closed 2 years ago

tynandebold commented 2 years ago

Description

We'd like to know how many users are using Viz as a React component.

Context

Do we need to continue supporting this use case?

Outcome

An answer to the question above.

Checklist

tynandebold commented 2 years ago

On 5 July, 2022, the following question was posed to the appropriate users channels in Discord and Slack:

Hello, everyone! We're looking to understand your usage of Kedro-Viz. Specifically, do you use it as a React component? https://www.npmjs.com/package/@quantumblack/kedro-viz

👍 for yes, 👎 for no

After ~48 hours, we have:

It's also thought that these two products might be using Viz as a React component:

Next step: follow up with each positive (yes) response, plus the two products linked above, and as them about how they're using Viz. Specifically:

921kiyo commented 2 years ago

@tynandebold Thank you for tagging me! I am not using Viz as a React component, I just used the Viz pipeline image in my website (sorry for the confusion...)

tynandebold commented 2 years ago

No confusion! Thank you for clarifying :)

datajoely commented 2 years ago

Thanks for pushing this @tynandebold I have a couple of thoughts on this.

In my eyes, much of the value of Viz has been providing largely back-end facing engineers with the ability to communicate to state/function/purpose of their data engineering / science efforts to non-technical audiences. I think there is potential to strive further in this direction.

In my own experience this medium article was a fun but painful experience to prepare. I had to work closely with @studioswong to find a way of compiling Viz as a SPA so that I could embed a version of the flowchart and also disable the sidebar behaviour. In my eyes, it should be simple to just say Viz component + JSON = Visualization.

With this in mind, where do I see real potential for this direction? DOCS

I think there are many places where a user may want to share a sub-graph of their pipeline and annotate / describe the purpose embedded in their docs. The Python world uses Sphinx as the primary documentation generator and this provides a plugin entrypoint where we could a native way of adding a Kedro Viz widget within the docs.

There Mermaid plugin feels like a good analogue for what we would need to do this properly, although I think shorter routes to MVP will exist.

tynandebold commented 2 years ago

Great context @datajoely, thank you.

Some thoughts and questions:

  1. When you say, "This is where the React component would be start" do you mean if we had Viz a React component?
  2. For your medium article, why didn't you use the export functionality of Viz and then add those output images to your article? Did you want the embed to be interactive? Your article was all about "layered thinking" and thus you showed and referenced the layers in Viz. The image exports don't actually include those, though if they did, would you have used them?
  3. We do actually have this: Viz component + JSON = Visualization. That's our React component. Perhaps that isn't ideal, as you need to create a small standalone application to get the output. Instead, and still using your formula, what if we output a standalone and single HTML file with the necessary CSS and JS included to display the Visualization? Would that be better and more usable? Could you have used that in Medium? Or even in a GitHub readme?
  4. For use in Docs, I think the image output of Viz is feasible, though maybe not feature rich right now? We have a feature request from a user (#638) where they mention they'd like a way to programmatically export/save the visualization for exactly this reason ("automatic documentation updates"). We're thinking about how to do this now.
studioswong commented 2 years ago

Hi folks - nice discussion! joining in the fun with my 2 cents here.

@tynandebold to your second and third point, my understanding is that @datajoely wants to showcase the interactive kedro-viz pipeline using medium embeds via a glitch hosted link (see this article). I had a brief look at embedding and rendering an html file as an embed in medium and don't think Medium supports it natively - i.e doing so will still require you to upload and host the html file somewhere which is probably the same amount of effort as setting it up as a react component export.

My curiosity is - what is the reason to not support the react component usage? It might seem counter-intuitive to look into setting up a new function to export and output a standalone HTML file in kedro-viz rather than utilizing the npm export setup which we have already setup nicely within our CI pipeline. There are other QB products who are consuming kedro-viz via npm ( such as Brix) - does stopping the npm support means they won't get the new goodies from latest versions of kedro-viz?

In terms of the the user request from #638 , to me it seems like a different use case as it looks like the user just wants a fast way to export svg/png directly via the terminal rather than via the Kedro-Viz UI, which is an entire different use case from those who consumes kedro-viz via an npm component ( which most likely they utilize it to showcase the interactive flowchart)

datajoely commented 2 years ago

@tynandebold

  1. Yes I mean, IF we had it, we could potentially do some cool things.
  2. My thinking is that a when describing a complex pipeline the ability to interactively fold and interrogate is super useful and better than any static image.
  3. For Medium you need to use one of their accredited embed providers - Ideally I would have used Codeopen to just pass the JSON to a component and it would work fine. In this case I needed to do a much more involved process to get Glitch working.
  4. I think this is the same point as 2, we can do static images fine - interactivity is the exciting part.
tynandebold commented 2 years ago

@datajoely

  1. We do have that though. Am I missing something or are we speaking of different things?
  2. Agreed.
  3. Good point. I've tried using the Viz React component in CodePen and it doesn't work.
  4. Noted.
tynandebold commented 2 years ago

@studioswong thanks for chiming in!

My curiosity is - what is the reason to not support the react component usage?

The short answer is to reduce complexity of the app in tandem with reducing the workload when building some potential future features.

There are other QB products who are consuming kedro-viz via npm ( such as Brix) - does stopping the npm support means they won't get the new goodies from latest versions of kedro-viz?

No, we still want to support other teams using Viz while trying to determine if the way they're using it is most optimal. For the Brix case and their setup, there was no other way to use it, so they took what we offered and ran with that. Yet what if there was a more suitable alternative (e.g. a HTML-generated view of the pipeline, a Kedro-Viz-Lite NPM package only showing the flowchart, a more rich static image export, etc., etc.)? Would that have been preferable?

This are some of the things we're trying to answer.

datajoely commented 2 years ago

Re 1 - As a novice JS user my understanding of 3 (Codepen) thought this wasn't valid.

tynandebold commented 2 years ago

You can basically import anything from NPM into CodePen. Here's a small React app running in there. I've tried using Viz in here and unfortunately we get an error with one of our dependencies. If we didn't, it should work just fine.

tynandebold commented 2 years ago

Given the feedback from user interviews, we'll continue to support using Viz as a React component.

We should strive to better our user's experiences with things like modularity, being lighter weight, adding access to click handlers, and more.

tynandebold commented 2 years ago

Closing this issue as the conclusion was reached that we'll still support this use case.