mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

parallel_tasks\files.py ParallelReadFile copies data into master table instead of child tables #10

Closed JohannaRaab closed 5 years ago

JohannaRaab commented 6 years ago

ReadFile.shell_command is called by ParallelReadFile.read_command with the master table as param target_table. Postgres does not redirect insert/copy from master table to child tables automatically.

In _ReadFile the correct child name is writen to a local param. l.140: target_table = self.targettable + '' + day.strftime("%Y%m%d") The param is not passed to function parallel_task, it is only used to analyze the (empty) partition.

Possible solutions could be:

JohannaRaab commented 5 years ago

Thanks