dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.6k stars 1.46k forks source link

Metadata for code location #15090

Open tnk-dev opened 1 year ago

tnk-dev commented 1 year ago

What's the use case?

Based on this conversation: https://dagster.slack.com/archives/C01U954MEER/p1688064447664719

Here is what I need. A column that shows metadata like custom version, authors, etc.

Also a description column for each code location would be a great addition.

Ideas of implementation

Basically a pretty formatted Python Dictionary that shows up in Dagit -> Deployment -> Code Location in a new column

Please don't make use the workfile.yaml or the Helm deployment for it, but Python code. Maybe some Class or Class extension that takes in new parameters like metadata and description.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

sryza commented 1 year ago

This sounds like a reasonable direction to me. One way to implement it would be a metadata argument on Definitions.

hellendag commented 1 year ago

@sryza Which team would own adding that argument? Can you triage to them? Feel free to reassign to us once we've got something we can use for the UI.

tnk-dev commented 1 year ago

Hey any updates here ? 😄

sryza commented 1 year ago

Hey @tnk-dev - this isn't yet on our near-term roadmap. Additional upvotes would make us more likely to prioritize it. If you were interested in implementing something like this yourself, we could also give pointers on how to do so.

elementljarredc commented 1 year ago

@braunjj this is pretty close to the idea that we've discussed around having a home page for a code location.

tnk-dev commented 1 year ago

@sryza I'd be happy to contribute.

Some pointers would be ideal like local setup, MR conventions and whatever you can think of that would speed up my experience😁

sryza commented 1 year ago

@tnk-dev awesome. Here is our contribution guide if you haven't seen it already: https://docs.dagster.io/community/contributing.

In terms of the actual changes, here are some of the places that I think will need to receive changes:

I believe the type for the attribute you'll want to add is Mapping[str, MetadataValue]

tnk-dev commented 1 year ago

@sryza thanks, I'll work on it in the upcoming week. You can assign it to me if possible :)

tnk-dev commented 1 year ago

@sryza PR created. Let me know what ya think.

@hellendag You could point me to the JS files that would be affected to make the metadata visible :) I didn't get how the GQL query in dagster/js_modules/dagit/packages/core/src/workspace/WorkspaceContext.tsx came about.

tnk-dev commented 1 year ago

Hey @sryza, could you pleased tell me the current state of this issue

sryza commented 1 year ago

Hey @braunjj - do you know where we stand on designs for code location-level metadata? I know there's been a lot on your plate recently.

braunjj commented 1 year ago

@sryza we're working on it but it won't land until after Launch Week next week.

tnk-dev commented 9 months ago

@sryza we're working on it but it won't land until after Launch Week next week.

Hey @braunjj Any chance to add this one to next release ?

hellendag commented 2 weeks ago

@braunjj Is this part of the new code location page design?