search

astronomer / dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
MIT License
1.09k stars 167 forks source link

Init containers is not converted to V1 kubernetes models #180

Open dimon222 opened 6 months ago

dimon222 commented 6 months ago

Init containers currently only passes basic objects defined in task (dicts, lists, etc). It should be handling stuff in more complex way to accommodate conversion of lists of dicts to things like V1 VolumeMount and etc.

siddu2020 commented 6 months ago

Can you explain why this is required ? are you looking at a specific use case ? I am more curious on context. I am thinking of contribution to this.

dimon222 commented 6 months ago

Refer to these two code blocks: Volume mount - https://github.com/ajbosco/dag-factory/blob/master/dagfactory/dagbuilder.py#L444-L448 Init containers - https://github.com/ajbosco/dag-factory/blob/master/dagfactory/dagbuilder.py#L502-L506 The problem is that volume mounts can be also not just on container level, but also on init containers level, and you can't just pass raw dict there that you got from reading yaml files (the underline casting will lead to incorrect serialized object that k8s api will not accept), it needs to match the V1VolumeMount class structure to be used properly. There might be other complex objects with similar problem that are missing conversion prior passing to constructors like V1Container.