search

reddmetrics / forma-deploy

Deployer for the FORMA Clojure code.
4 stars 0 forks source link

github integration #2

Open danhammer opened 12 years ago

danhammer commented 12 years ago

Currently, on our local clone of this project, we change this line to point to a file called forma_bootstrap_robin.sh, which contains Robin's github credentials. We need this in order to clone the forma-clj project on the cluster. He doesn't want these credentials to go in this repo, which makes sense.

Ideally, we would have a way to configure github using our own credentials which could be stored or copied into the forma-deploy repo on our local machine. Or maybe in ~/.ssh or something. I don't know. We can talk more about this if I am being unclear.

sritchie commented 12 years ago

I think we should never clone the repo onto every machine in the cluster. See my rsync comment.

On Wed, Jul 18, 2012 at 3:13 PM, Dan < reply@reply.github.com

wrote:

Currently, on our local clone of this project, we change this line to point to a file called forma_bootstrap_robin, which contains Robin's github credentials. We need this in order to clone the forma-clj project on the cluster. He doesn't want these credentials to go in this repo, which makes sense.

Ideally, we would have a way to configure github using our own credentials which could be stored or copied into the forma-deploy repo on our local machine. Or maybe in ~/.ssh or something. I don't know. We can talk more about this if I am being unclear.

Reply to this email directly or view it on GitHub: https://github.com/reddmetrics/forma-deploy/issues/2

Sam Ritchie, Twitter Inc 703.662.1337 @sritchie09

(Too brief? Here's why! http://emailcharter.org)

robinkraft commented 12 years ago

Note that our bootstrap script has placeholders for your private key and public key git credentials.

I'm not sure what the right way to do this would be.

danhammer commented 12 years ago

Fair enough. Where is the rsync comment, @sritchie?

sritchie commented 12 years ago

Search email for "rsync"

danhammer commented 12 years ago

nothing comes up. paste in the comment here, if you think it's worth the time.

robinkraft commented 12 years ago

Here 'tis

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for quickly switching between branches, etc. I'm not convinced rsync is the way to go.

robinkraft commented 12 years ago

I don't care about having it on every machine, but having it on the master node is pretty sweet.

sritchie commented 12 years ago

Try rsync once, then judge for yourself. I'm totally convinced.

On Wed, Jul 18, 2012 at 3:59 PM, Robin Kraft < reply@reply.github.com

wrote:

Here 'tis

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for quickly switching between branches, etc. I'm not convinced rsync is the way to go.

Reply to this email directly or view it on GitHub: https://github.com/reddmetrics/forma-deploy/issues/2#issuecomment-7085061

Sam Ritchie, Twitter Inc 703.662.1337 @sritchie09

(Too brief? Here's why! http://emailcharter.org)

robinkraft commented 12 years ago

Deal!! Great idea.

On Jul 18, 2012, at 4:02 PM, Sam Ritchie wrote:

Try rsync once, then judge for yourself. I'm totally convinced.

On Wed, Jul 18, 2012 at 3:59 PM, Robin Kraft < reply@reply.github.com

wrote:

Here 'tis

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for quickly switching between branches, etc. I'm not convinced rsync is the way to go.

Reply to this email directly or view it on GitHub: https://github.com/reddmetrics/forma-deploy/issues/2#issuecomment-7085061

Sam Ritchie, Twitter Inc 703.662.1337 @sritchie09

(Too brief? Here's why! http://emailcharter.org)

Reply to this email directly or view it on GitHub: https://github.com/reddmetrics/forma-deploy/issues/2#issuecomment-7085143

robinkraft commented 12 years ago

Are you using s3fs in conjunction with rsync?

http://superuser.com/a/433329

On Jul 18, 2012, at 4:02 PM, Sam Ritchie wrote:

Try rsync once, then judge for yourself. I'm totally convinced.

On Wed, Jul 18, 2012 at 3:59 PM, Robin Kraft < reply@reply.github.com

wrote:

Here 'tis

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for quickly switching between branches, etc. I'm not convinced rsync is the way to go.

Reply to this email directly or view it on GitHub: https://github.com/reddmetrics/forma-deploy/issues/2#issuecomment-7085061

Sam Ritchie, Twitter Inc 703.662.1337 @sritchie09

(Too brief? Here's why! http://emailcharter.org)

Reply to this email directly or view it on GitHub: https://github.com/reddmetrics/forma-deploy/issues/2#issuecomment-7085143

robinkraft commented 12 years ago

Oh, right, missed that detail.

On Jul 18, 2012, at 4:21 PM, Sam Ritchie wrote:

What? no, rsync the jar up to the jobtracker machine from your machine after the deploy finishes and the cluster's ready. @eightysteele knows his shit, he can help.

On Wed, Jul 18, 2012 at 4:15 PM, Robin Kraft < reply@reply.github.com

wrote:

Are you using s3fs in conjunction with rsync?

http://superuser.com/a/433329

On Jul 18, 2012, at 4:02 PM, Sam Ritchie wrote:

Try rsync once, then judge for yourself. I'm totally convinced.

On Wed, Jul 18, 2012 at 3:59 PM, Robin Kraft < reply@reply.github.com

wrote:

[Here 'tis](

https://github.com/reddmetrics/forma-clj/issues/130#issuecomment-7077913)

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for quickly switching between branches, etc. I'm not convinced rsync is the way to go.

Reply to this email directly or view it on GitHub:

https://github.com/reddmetrics/forma-deploy/issues/2#issuecomment-7085061

Sam Ritchie, Twitter Inc 703.662.1337 @sritchie09

(Too brief? Here's why! http://emailcharter.org)

Reply to this email directly or view it on GitHub:

https://github.com/reddmetrics/forma-deploy/issues/2#issuecomment-7085143

Reply to this email directly or view it on GitHub: https://github.com/reddmetrics/forma-deploy/issues/2#issuecomment-7085393

Sam Ritchie, Twitter Inc 703.662.1337 @sritchie09

(Too brief? Here's why! http://emailcharter.org)

Reply to this email directly or view it on GitHub: https://github.com/reddmetrics/forma-deploy/issues/2#issuecomment-7085516

robinkraft commented 12 years ago

For reference, Sam's suggested solution, from here

  1. Put a jar on S3 and download that as part of the deploy.
  2. Build an uberjar locally and rsync it up to that location on the jobtracker.

The idea is that since rsync will only sync the changes, even if we're changing branches, etc., not that much is actually changing within the uberjar so it won't take long to sync.