meltano / files-airflow

MIT License
0 stars 5 forks source link

Scheduling jobs throws KeyError: 'start_date' #31

Closed r-nyq closed 2 years ago

r-nyq commented 2 years ago

Scheduled jobs does not include start_date but _meltano_job_generator in meltano.py looks for start_date key. This throws an KeyError: 'start_date'

File "project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/models/dagbag.py", line 318, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "project/orchestrate/dags/meltano.py", line 193, in <module>
    create_dags()
  File "project/orchestrate/dags/meltano.py", line 187, in create_dags
    _meltano_job_generator(schedule_export["schedules"].get("job"))
  File "project/orchestrate/dags/meltano.py", line 128, in _meltano_job_generator
    if schedule["start_date"]:
KeyError: 'start_date'

This is the schema for scheduled jobs from meltano schedule

def _format_job_list_output(entry: Schedule, job: TaskSets) -> dict:
    return {
        "name": entry.name,
        "interval": entry.interval,
        "cron_interval": entry.cron_interval,
        "env": entry.env,
        "job": {
            "name": job.name,
            "tasks": job.tasks,
        },
    }
r-nyq commented 2 years ago

This is a copy of issue #29 and fixed by pull request #30