BCODMO / pipeline-generator

Generates a pipeline .yml file
2 stars 2 forks source link

Test the find/replace in dataflows to see if it also has the "None" replacement issue #7

Closed adyork closed 5 years ago

adyork commented 6 years ago

If we decide to integrate dataflows into our pipelines there could potentially be three different ways to find and replace.

The current way we are using, pipeline built-in processor way, has an issue where non-matches that are also blank cells are filled in with the string "None." Currently the only way to avoid this is to do a second find and replace using find pattern: "None" (without quotes) and leave the replace with field blank.

  1. pipeline processor - has "None" fill issue.
  2. dataflow "find_replace" - unknown whether same issue as pipeline find replace.
  3. custom flow. If all else fails we can write a custom find replace (python re) which seems silly to do but if it works then ....
adyork commented 5 years ago

Not doing this.