search

flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.2k stars 551 forks source link

[Feature] FlyteAdmin should support a GraphQL endpoint for entities - readonly #561

Open kumare3 opened 3 years ago

kumare3 commented 3 years ago

Motivation: Why do you think this is important? To add new features to UI, it would be the best to use GraphQL based endpoints, that would allow filtering entities and projecting columns.

Goal: What should the final outcome look like, ideally? UI can be improved at a faster rate

Describe alternatives you've considered Add new endpoints to REST/gRPC Specifically summary endpoints for

  1. Projects/domains
  2. Workflows
  3. Executions

Flyte component

kumare3 commented 3 years ago

cc @gagoar what do you think about this?

gagoar commented 3 years ago

cc @gagoar what do you think about this?

I think in general terms this is would be a great addition.

GraphQL in this particular space will provide flexibility for both parts of the stack. It will also provide ergonomics to understand the data that is offered and available even before the UI is built which will increase leverage.

Finally, if typescript is also one of the mid-term objectives, having GraphQL alongside TS will produce a leap in maintainability over time.

My grain of salt here would be:

1) Is this a priority right now? 2) Is there a direct need that we know that GraphQL will unlock or help solve? 3) This is a scheduling problem more than anything else IMO.

kumare3 commented 3 years ago

cc @gagoar what do you think about this?

I think in general terms this is would be a great addition.

GraphQL in this particular space will provide flexibility for both parts of the stack. It will also provide ergonomics to understand the data that is offered and available even before the UI is built which will increase leverage.

Finally, if typescript is also one of the mid-term objectives, having GraphQL alongside TS will produce a leap in maintainability over time.

My grain of salt here would be:

  1. Is this a priority right now?
  2. Is there a direct need that we know that GraphQL will unlock or help solve?
  3. This is a scheduling problem more than anything else IMO.

@gagoar great points. So TBH we do not have bandwidth to work on this and I question the need. This was something someone in the team raised. the data-model for Flyte controlplane is complex. Rendering one entity could take many calls and stitching them together in the UI. The engineer suggested use of GraphQL to solve this issue. Do you think this is accurate and even needed?

gagoar commented 3 years ago

cc @gagoar what do you think about this?

I think in general terms this is would be a great addition. GraphQL in this particular space will provide flexibility for both parts of the stack. It will also provide ergonomics to understand the data that is offered and available even before the UI is built which will increase leverage. Finally, if typescript is also one of the mid-term objectives, having GraphQL alongside TS will produce a leap in maintainability over time. My grain of salt here would be:

  1. Is this a priority right now?
  2. Is there a direct need that we know that GraphQL will unlock or help solve?
  3. This is a scheduling problem more than anything else IMO.

@gagoar great points. So TBH we do not have bandwidth to work on this and I question the need. This was something someone in the team raised. the data-model for Flyte controlplane is complex.

This is actually a good reason to use GraphQL. if the need is to represent a complex entity in the frontend. not implementing a layer like GraphQL will mean that we are pushing that complexity to the Frontend, having more chances of a bad experience and harder to optimize.

Rendering one entity could take many calls and stitching them together in the UI. The engineer suggested the use of GraphQL to solve this issue.

It will solve this Partially. cardinality here will be always the issue, It doesn't matter what happens (at the end of the day data is in different places and the round trips are gonna have to happen) but having a singular layer will allow a different place to cache data on.

I will add that GraphQL can be a way to expose entities in a way that is useful for the consumer and it doesn't necessarily represent the entity as is in the backend.

Do you think this is accurate and even needed?

I think it will solve the cardinality partially, it will def give you a caching layer and a simplification on how clients consume whatever data you want to expose. the data layer where Frontend gets plugged is something that becomes complex over time. This is the case for many possible reasons, but the most common I've seen is having too many views with too many particular requirements (producing API extensions that start breaking the consistency of what the API is or serves).

the idea behind a layer like GraphQL in this context (IMO) is the ability to create a layer where: 1) FE connects for every data need. 2) a caching point for elements that don't really change that much 3) A consumer-oriented endpoint, not a data exposition ( I call data exposition in GraphQL: GraphQL as a feature) 4) A contract/agreement (via GraphQL schema) between backend and frontend to what is needed and what needs to be done (which allows for parallelism without too much back and forth)

All these will allow the APIs to remain cohesive and optimized while allowing through GraphQL the expressivity and ergonomics you need.

github-actions[bot] commented 10 months ago

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

github-actions[bot] commented 10 months ago

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

kumare3 commented 6 months ago

This issue is still relevant and we will probably add this to a larger effort of supporting graphQL for ui