search

BCODMO / pipeline-generator

Generates a pipeline .yml file
2 stars 2 forks source link

test unpivot/pivot from dataflows #6

Closed adyork closed 5 years ago

adyork commented 5 years ago

Make a dataflow .py for pivot/unpivot and see if it will work for our identified use-cases.

If this works out we can add flows directly into the pipeline yamls.

Example custom flow called from a pipeline: https://github.com/frictionlessdata/datapackage-pipelines#dataflows-integration

adyork commented 5 years ago

I have to do this for the ProteOMZ proteins dataset so this will get done during my normal DM duties for https://www.bco-dmo.org/dataset/737620.

adyork commented 5 years ago

Done, works easily enough in dataflows. Nice regex functionality to specify which columns to flip. see notes in https://github.com/BCODMO/Proteomz-Saito/tree/DM_structure/737620/3#tips-learned for https://github.com/BCODMO/Proteomz-Saito/commit/9fec20be211b66357e3f45ce0aab0dcdca435fa5.

However, we don't need to use dataflows for this, pipelines has unpivot too.