search

rajasekarv / vega

A new arguably faster implementation of Apache Spark from scratch in Rust
Apache License 2.0
2.23k stars 207 forks source link

Remove requirement to copy the host conf file manually to executors #115

Closed iduartgomez closed 4 years ago

iduartgomez commented 4 years ago

Closes #109

I think this is all what was needed to change in that issue, correct me otherwise.

rajasekarv commented 4 years ago

This is ok. However, I changed the context logic completely. So this won't be required. Even hosts.conf is not necessary anymore.

context.rs mapoutputtracker.rs scheduler/* cachetracker.rs executor.rs shufflefetcher.rs shufflemanager.rs

These files and basically anything with networking is undergoing full revamp. So for some time it would be good to not touch these. At max 2 weeks. It would be good to finish all other missing APIs in rdd and pair rdd except for orderedrdd. Or I would do something. I will create new branch and put all changes there so that you can track the changes and you can commit there if you want.

iduartgomez commented 4 years ago

Yes please open a new branch so we can keep track. Thanks!