search

chrislusf / glow

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
3.19k stars 246 forks source link

[feature] rsync executables to agents #2

Open chrislusf opened 8 years ago

chrislusf commented 8 years ago

Currently the "rsync" folder is not using rsync. I attempted, but later just used normal http copy. I would like someone to help me to use real rsync.

A better idea may be using bittorrent DHT to distribute the binary file.

JensRantil commented 8 years ago

@chrislusf Are you referring to the "rsync" directory under "driver"? Specifically, which function would require reimplementation using rsync? Also, is there an interface that could be used to swap out rsync for something like bittorrent?

chrislusf commented 8 years ago

Thanks for your interest!

There is no clear interface yet, but the files under "rsync" folder should be fairly simple to read. The "fetch.go" file is not working. The "fetch_url.go" currently is the entry point to copy binaries.