chaps-io / gush

Fast and distributed workflow runner using ActiveJob and Redis
MIT License
1.04k stars 104 forks source link

Accessing job output inside workflow #49

Closed dmitrypol closed 6 years ago

dmitrypol commented 6 years ago

I have one job that downloads file and then I queue multiple jobs to process each row in the file. How do I access the file path inside the Workflow (not descendant job) so I can loop through it? I need to get payloads output and assign it to variable but after studying the code I do not see a solution.

class DownloadCsvJob < Gush::Job
  def perform
    output('path/to/data.csv')
  end
end
class ImportWorkflow < Gush::Workflow
  def configure    
    run DownloadCsvJob
    csv_jobs = CSV.foreach("path/to/data.csv").map do |row|
      run ImportCsvRowJob, params: row, after: DownloadCsvJob
    end
    run GenReportJob, after: csv_jobs
  end
end

Another use case is when one job performs some kind of check and the subsequent job(s) are queued based on the results of that.

Thank you very much.

pokonski commented 6 years ago

It is not possible because Workflow and its jobs are defined only once, during the creation. So it has no access to the output of jobs, because they are not even executed at that moment :)