search

datahq / dataflows

DataFlows is a simple, intuitive lightweight framework for building data processing flows in python.
https://dataflows.org
MIT License
194 stars 39 forks source link

exceptions from generators shouldn't be silently ignored #54

Closed OriHoch closed 5 years ago

OriHoch commented 5 years ago

reproduction

from dataflows import Flow, printer

def generator():
  for i in range(5):
    raise Exception()
    yield {"i": i}

Flow(generator(), printer()).process()

Expected

Exception

Actual

no exception, empty resource

OriHoch commented 5 years ago

related to inferring schema, moving exception below the yield raises it

from dataflows import Flow, printer

def generator():
  for i in range(5):
    yield {"i": i}
    raise Exception()

Flow(generator(), printer()).process()
akariv commented 5 years ago

Interesting find! Are you looking into it?

OriHoch commented 5 years ago

not at the moment (probably not anytime soon)

akariv commented 5 years ago

👍 assigning to me then