search

awesome-kyuubi / hadoop-testing

Testing Sandbox for Hadoop Ecosystem Components
Apache License 2.0
32 stars 13 forks source link

Introduce docker compose profile feature to isolate and compose different component groups for different purpose #12

Closed yanghua closed 10 months ago

yanghua commented 10 months ago

As adding more components to Hadoop testing, it would be better to introduce a solution to isolate and compose different component groups, e.g (mini-hadoop-cluster, lakehouse-iceberg, and so on).

IMO, the docker-compose profile feature is a good choice, just like the maven profile.

yanghua commented 10 months ago

@pan3793 WDYT?

pan3793 commented 10 months ago

I write my original thoughts in README:

... we may want to leverage the template engine like Ansible plus Jinja2 to generate the docker-compose.yml file and configuration files for each component, so that user could easily customize the testing environment by modifying the configurations.

In fact, a lot of Hadoop ecosystem components usually mix-deploys in a group of machines, we can not achieve that by using docker-compose profile

yanghua commented 10 months ago

I write my original thoughts in README:

... we may want to leverage the template engine like Ansible plus Jinja2 to generate the docker-compose.yml file and configuration files for each component, so that user could easily customize the testing environment by modifying the configurations.

In fact, a lot of Hadoop ecosystem components usually mix-deploys in a group of machines, we can not achieve that by using docker-compose profile

Agree, the mechanism of profiles can only solve some issues. Its flexibility is limited. My original thought only is for picking some of those components that we have supported for different scenarios.