frictionlessdata / frictionlessdata.io

The main repository of the Frictionless Data project. Website, issues, and discussions
http://frictionlessdata.io
MIT License
140 stars 53 forks source link

Add configurability to custom Steps #915

Open neolefty opened 3 months ago

neolefty commented 3 months ago

Overview

A research team within the Purdue College of Agriculture has started to use Frictionless for its data pipelines, and we would like to write custom Steps that receive configuration options from the Pipeline YAML file. For example:

steps:

  # Standard Step -- ensure ID is an integer
  - type: field-update
    name: id
    descriptor:
      name: farm_animal_id
      type: integer

  # Custom Step -- normalize funky date formats
  # Currently, in our testing, values (below) such as field_name or descriptor.name
  # are not getting passed to the custom step, and it appears they are not kept by
  # the custom Step mechanism. We want to capture them and pass them into the
  # custom Step initialization.
  - type: date-normalize
    field_name: Date
    descriptor:
      name: Date
      type: string
neolefty commented 3 months ago

Hi Frictionless! We want to develop and submit a solution, and also plan to attend the hangout tomorrow, to get connected to the Frictionless community.

neolefty commented 3 months ago

Question: Should this move over to frictionless-py? https://github.com/frictionlessdata/frictionless-py/issues