datahq / dataflows

DataFlows is a simple, intuitive lightweight framework for building data processing flows in python.
https://dataflows.org
MIT License
195 stars 40 forks source link

Add computed field with callable #70

Open akariv opened 5 years ago

akariv commented 5 years ago

Add a computed field that is computed using a Python callable This is very useful and can be reused. It might make sense to merge with dataflows.add_computed_field. h/t @shevron

def add_computed_field_callable(name, type, callback, **options):
    def func(package):
        # Alter the schema to add a field
        for resource in package.pkg.descriptor['resources']:
            resource['schema']['fields'].append(dict(name=name, type=type, **options))
        yield package.pkg

        def value_setter(rows):
            for row in rows:
                row[name] = callback(row)
                yield row

        for resource in package:
            yield value_setter(resource)

    return func
akariv commented 5 years ago

Fixed via #78