Open hrshdhgd opened 2 months ago
Koza's architecture has a baked in separation of concerns that lets the transform.py really only transform individual rows and pass them to the writer, but everything about the configuration has already happened by the time transform.py runs. The yaml is where the process starts, not the python code.
As the issue is, all that I can say is, "correct, you can't set those things in the transform.py, it's too late" but I bet that we can figure out a solution to the actual problem that fits into Koza's architecture, or handle it before or after Koza using other tooling.
Okay, thanks @kevinschaper that makes sense
I guess separating the concern of config parsing also has the advantage that all the config stuff lives in one place (the config file), which makes it easier to see how the transform was done
As of now, Koza assumes that an ingest project has
transform.yaml
located in the same directory as the code that is generating thekoza_app
object. I seems like attributes of this object cannot be set via code.Example: Setting the
output_dir
ordelimiter
attribute like this:OR
cc: @caufieldjh , @justaddcoffee