Document how to pass data between steps in a job

[chrishiestand]

I would expect to see how to pass data between steps in a job here: https://concourse-ci.org/jobs.html and/or https://concourse-ci.org/steps.html

But both are silent on this important detail. We should document that the way to pass data between steps is file-based. Each job is run in a dedicated directory where get resources and task outputs are stored, that later steps can access data written by earlier steps, in sub directories named after the name of the get resource or task output. Does that sound about right? I will try to create a PR for this later if nobody beats me to it.

[osis]

Would https://github.com/concourse/docs/blob/examples/lit/examples/task-passing-artifact.lit or https://github.com/concourse/docs/blob/examples/lit/examples/trigger-then-continue.lit via https://github.com/concourse/concourse/issues/2898 satisfy your needs?

Or are you looking for an example of a created artifact that's outputted to a resource and passed on to another job?

[chrishiestand]

Thanks @osis. I think examples are an invaluable help. The former is the area that needs more documentation that I'm describing in this issue (https://github.com/concourse/docs/blob/examples/lit/examples/task-passing-artifact.lit). In addition though, we should describe conceptually how data is supposed to be passed between steps in a job.

I think we should roughly cover these conceptual points:

• How to get data in a job (A: get resource)
• How to pass data between steps in the same job (A: directories/files relative and with specific names within the build directory)
• Which steps can access data from previous steps (A: all subsequent steps in a job)
• Best practices for passing data between jobs - e.g. is there a preference for storing data in a step's input directory, or using a dedicated output directory? Any other trade-offs to mentions?

[osis]

Cool. For what it's worth. I've found it helpful to reframe the my internal view to something like "How does my resource flow through my pipeline." vs looking at it via the job perspective.

For me, this has helped me align with one of Concourse's principals of not holding state. But I also realize that there are lots of ways of looking at a pipeline and docs should strive to cover as many angles as possible.

I plan on adding more examples via https://github.com/concourse/concourse/issues/2898 and eventually want to explain some of the concepts and principals in the right sidebar and doing some cross-referencing magic so that people can shift in-between the reference docs and working examples.

I think your last point depends on your use case, and Concourse lacks sometimes lacks an opinion with the hope to not limit use cases. Would be interesting to surface some of the internal constructs to bring some clarification so users can do what works for them. 👍

[taylorsilva]

This is covered in two areas now:

• https://concourse-ci.org/tutorial-inputs-outputs.html
• https://concourse-ci.org/exploring-task-input-and-output-scenarios.html