thbar / kiba

Data processing & ETL framework for Ruby
https://www.kiba-etl.org
Other
1.75k stars 87 forks source link

Allow block-form for source #8

Closed thbar closed 8 years ago

thbar commented 9 years ago

I find myself wrapping sources like this:

require 'oj'

class JSONSource
  def initialize(file)
    @file = file
  end

  def each
    File.open(@file) do |file|
      Oj.load(file).each do |row|
        yield row
      end
    end
  end
end

where I'd prefer to be able to write this (at least for one-off scripts):

source do
  File.open(source_file) do |file|
    Oj.load(file).each do |row|
      yield row
    end
  end
end

Here the semantic would be that the block must yield each row. Another possible semantic would be this:

source { Oj.load(IO.read(source_file)) }

where the block is expected to return something that would respond to .each.

Maybe both semantics would be nice to have, with maybe a different keyword or some kind of parameter.

I must think with more depth about this for now, just dropping a note.

eduardodeoh commented 9 years ago

:+1:

thbar commented 9 years ago

The yield construct is afaik not possible to achieve (see discussion). Passing a block as a param should work though. That, or using enumerables/enumerators.

Radagaisus commented 9 years ago

A magic process method can do the trick.

source do
  File.open(source_file) do |file|
    Oj.load(file).each do |row|
      process row
    end
  end
end
thbar commented 9 years ago

@Radagaisus good point - I'll try that out when I look more closely at this again. Thanks!

thbar commented 8 years ago

While this seems a "nice to have" idea, in months of production use of Kiba, I never truly felt the actual need to have that. I always end up writing sources as real classes, without feeling any urge to do differently. Closing for now, I'll re-open if I change my mind!