search

rajasekarv / vega

A new arguably faster implementation of Apache Spark from scratch in Rust
Apache License 2.0
2.23k stars 206 forks source link

Provide configs throught env vars only #73

Closed iduartgomez closed 4 years ago

iduartgomez commented 4 years ago

Removes clap and parses configs from env vars only.

Closes #54

rajasekarv commented 4 years ago

Actually this is not a good idea. For distributed mode, why would we want to configure every single machine? We should get configuration for executors from command line arguments only. Did you test this in distributed mode? plus ENV variables should not represent program-specific arguments.

iduartgomez commented 4 years ago

I reverted the change, is true the arguments must be propagated to the workers and I had to fix that made the push too late yesterday haha.

Will look into it later.

rajasekarv commented 4 years ago

Haha. No issues. I think we should make distributed mode testing also as part of CI/CD pipeline

iduartgomez commented 4 years ago

Haha. No issues. I think we should make distributed mode testing also as part of CI/CD pipeline

Yeah that would be definitively a good idea! When we make shut down in distributed mode ok we could enable it.

Shouldn't be too bad changing the docker set up a bit (e.g. is not really necessary to build the image as you can execute the compiled tests when mounting the volume, this is what i use to test distributed mode at least).