Closed ericbusboom closed 8 years ago
Create a stronger separation of the ingestion phase.
Don't auto create source tables. Create them only when the wranger requests it. The source tables are always loaded in from config.
So, the wrangler runs an ingest, then exports the source tables, and edits them. Then they are synced in.
When a source is run, if a source table exists, a new pipe segment is created to map the source headers and column types.
This change would also make it easier to load sources directly into build pipelines.
Create a stronger separation of the ingestion phase.
Don't auto create source tables. Create them only when the wranger requests it. The source tables are always loaded in from config.
So, the wrangler runs an ingest, then exports the source tables, and edits them. Then they are synced in.
When a source is run, if a source table exists, a new pipe segment is created to map the source headers and column types.
This change would also make it easier to load sources directly into build pipelines.