DAGWorks-Inc / hamilton

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
https://hamilton.dagworks.io/en/latest/
BSD 3-Clause Clear License
1.89k stars 126 forks source link

@ignore #1172

Closed jernejfrank closed 1 month ago

jernejfrank commented 1 month ago

Solves #1168.

Changes

How I tested this

Checklist

jernejfrank commented 1 month ago

Looks good! Small nits. One thought -- is @ignore (the name I suggested) the best name?

Other thoughts (would love your ideas):

  • @hamilton_ignore
  • @exclude
  • @hamilton_exclude

And another thought -- should we have an option in @mutate and step to not ignore them? E.G. is there any way we may want to also include it in the pipeline (instinct says no here...)

Having a look with fresh eyes I agree, we should make it explicit that it is for hamilton to avoid confusion. How about hamilton_skip?

zilto commented 1 month ago

Looks good! Small nits. One thought -- is @ignore (the name I suggested) the best name? Other thoughts (would love your ideas):

  • @hamilton_ignore
  • @exclude
  • @hamilton_exclude

And another thought -- should we have an option in @mutate and step to not ignore them? E.G. is there any way we may want to also include it in the pipeline (instinct says no here...)

Having a look with fresh eyes I agree, we should make it explicit that it is for hamilton to avoid confusion. How about hamilton_skip?

Thoughts on naming:

Other nit:

elijahbenizzy commented 1 month ago

I think it should be very specific that Hamilton is the one doing it -- so I'm in favor of @hamilton_exclude.

That said, I don't think this should be the first way to do things (it should be with _), but there are times it'll be useful (the docs here hit it right, it should show up in the docs, but not in the "How-to" section)

elijahbenizzy commented 1 month ago

Looks good, let's just change the wording of the docs to indicate that _ is preferred!