Currently, if we want to run a series of Jobs where the input of one jobs is the output of the previous one, we need to materialize the output on disk for the next one to be able to read it. People using Spark or Hadoop usually run a series of jobs rather than a single big job. Thus it is important to support a functionality of caching the results of one job to be consumed by the next job.
Currently, if we want to run a series of Jobs where the input of one jobs is the output of the previous one, we need to materialize the output on disk for the next one to be able to read it. People using Spark or Hadoop usually run a series of jobs rather than a single big job. Thus it is important to support a functionality of caching the results of one job to be consumed by the next job.