Open zeke opened 10 years ago
@zeke: FYI, Pull https://github.com/timshadel/heroku-buildpack-github-netrc/pull/6 removes the user-env-compile
dependency.
Hi Gents,
Some good usage docs would be great here. Particularly, I'm assuming:
bundle install
for an app with private repo dependencies?You can imagine I'm here because this is not working as expected :)
-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/timshadel/heroku-buildpack-github-netrc.git
=====> Detected Framework: github-netrc
Generated .netrc & .curlrc files (available only at build-time)
Github User: rwdaigle
Organizations: spreedly
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-ruby.git
=====> Detected Framework: Ruby
-----> Compiling Ruby/Rack
-----> Using Ruby version: ruby-2.1.1
-----> Installing dependencies using 1.5.2
Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
Fetching gem metadata from https://rubygems.org/.........
Fetching additional metadata from https://rubygems.org/..
Fetching git@github.com:spreedly/caymans.git
Host key verification failed.
fatal: The remote end hung up unexpectedly
My issue turned out to be that you can't use git@git://
style URLs in your Gemfile (since that forces the user to be git
and not the one in your .netrc
) You have to use https://
without any embedded auth info. This has some other ramifications as well, mostly for local dev workflow. Once I nail down a process I'm happy with I'll see what I can do about updating the README.
The local dev flow is pretty awesome with https:// URLs with no authentication. You simply create a local ~/.netrc file, and everything works great.
This buys you several things.
Hope that helps! — Sent from Mailbox
On Thu, May 1, 2014 at 6:58 PM, Ryan Daigle notifications@github.com wrote:
My issue turned out to be that you can't use
git@git://
style URLs in your Gemfile (since that forces the user to begit
and not the one in your.netrc
) You have to usehttps://
without any embedded auth info. This has some other ramifications as well, mostly for local dev workflow. Once I nail down a process I'm happy with I'll see what I can do about updating the README.Reply to this email directly or view it on GitHub: https://github.com/timshadel/heroku-buildpack-github-netrc/issues/5#issuecomment-41973663
Hi @rwdaigle! Thanks for jumping in on this.
Hi Tim,
The local dev flow is pretty awesome with https:// URLs with no authentication. You simply create a local ~/.netrc file, and everything works great.
Unfortunately, this is quite a departure from the normal local development flow. Especially since you can't just populate your .netrc with your normal GH username/password – you have to use a token. Same goes for just logging in when prompted on git fetch
.
Username for 'https://github.com': rwdaigle
Password for 'https://rwdaigle@github.com': <normal password>
fatal: Authentication failed
That's the biggest hiccup to me in this workflow. Not only do you have to modify all your private repo git URLs (from git@
to https
), you also have to setup token-based credentials outside the normal git workflow.
I'd be curious to know if this has been your experience as well? And if so, does it being the Right Way:tm: to do things justify the difficulties?
Hey Ryan,
Awesome questions. Let me tackle them out of order.
I'd be curious to know if this has been your experience as well? And if so, does it being the Right Way to do things justify the difficulties?
Nope. You probably have something wrong with your regular Github password, or used the wrong URL or something. You might try:
GIT_CURL_VERBOSE=1 git fetch
What happens when git fetch
asks for creds is that it uses them directly as the Basic auth sent with curl, and never stores them unless you've configured a credential helper. That failure has nothing to do with ~/.netrc
. Sorry.
Especially since you can't just populate your .netrc with your normal GH username/password – you have to use a token.
Actually, that's not true. It's precisely because the ~/.netrc
file is intended to simply use your username and password that it was chosen as an easy way to give the build process private repo access.
If I put this in my ~/.netrc
, my git fetch
commands work just perfectly.
machine github.com login timshadel password <my-gh-pass>
Unfortunately, this is quite a departure from the normal local development flow.
So because this is just a way to supply usernames & passwords to git, you can use any of the typical ways to do that locally. I use the osxkeychain
credential-helper. Again, once all your private repo references are set with https
and have no usernames in them, at that point you can use pretty much any of the various credential approaches to using git, and any of the different kinds of credential available (credential-helper, netrc, etc) with Github in particular (user/pass, token). Nicely factored config.
Why not stick to user/pass in .netrc?
Now, the question becomes, Do I really want my password lying around in plain text? That's where the whole token piece comes in. Tokens are only used because they're easily revoked, and they can be used anywhere in Github's infrastructure that Basic auth can be used.
That means even when you have no ~/.netrc
file, and git pops up and asks you for your auth information, you can use the token directly, if you really want. The password is always x-oauth-basic
(see links in the README.md).
Username for 'https://github.com': <my-gh-TOKEN>
Password for 'https://<my-gh-TOKEN>@github.com': x-oauth-basic
HTTPS is the default URL protocol for Github repos
Authenticating to Github and storing the credentials for https
repos is already well documented in Github Help.
Notice that both in Github's docs and in their UI (clone URL), SSH is an afterthought. You have to go digging to find it. They've clearly shifted emphasis to https
URLs (likely because of how much simpler it is to use those through corporate firewalls, I'd guess).
So unlike you and I who have used Github since the beta, most newer folks will only see https
URLs, so this is the new normal workflow.
So, the main motivation for using ~/.netrc
at all in a buildpack is to provide a username & password to git during the build process as it clones repos with https
URLs with no embedded authentecation, presumably for private dependencies. That's it.
you also have to setup token-based credentials outside the normal git workflow
Hopefully by now it's obvious that you don't need to. You can, yes, but you don't need to do this for local work—only to use this buildpack.
you have to modify all your private repo git URLs (from git@ to https)
Yeah, this is the one-time pain point. There's a day or so of git remote set-url
stuff, and editing a pile of Gemfile
or package.json
or other files.
But the amazing thing is that npm
, bundle
, and a bunch of other tools will play nice with this approach.
One More Thing
If create a token using the curl command in the README, and then you want to play with Github's API, using curl and jq for example, then you'll want to make sure you've got two lines in your ~/.netrc
.
machine github.com login <my-gh-token> password x-oauth-basic
machine api.github.com login <my-gh-token> password x-oauth-basic
Then you can use curl
to see your repos.
curl -n https://api.github.com/user/repos
To get rid of the -n
, open the curl config file ~/.curlrc
. That'll also help out tools that don't automatically ask curl to look at the ~/.netrc
file while also letting those tools specify alternate creds.
--netrc-optional
This thread is amazing. @timshadel would you be interested in collaborating on a Heroku blog post or Dev Center article?
Wow. Sure, I guess. Is it used widely enough that you get questions like this?? But sure. What's the best way to get in touch?
We don't necessarily get direct questions about this, but users are always searching for new and elegant ways to access private dependencies in the build process. This .netrc
technique is more elegant and secure than other approaches I've seen, and fits well with Heroku's way of solving problems. It's also a perfect example of what composable buildpacks are all about.
Email me at zeke@heroku.com
Tim,
I believe I know why I was experiencing this anomaly:
you can't just populate your .netrc with your normal GH username/password – you have to use a token
I have Two-factor auth turned on. I suspect with that setting, you can't just use your normal user/pass in the .netrc
. Two-factor auth is quickly becoming the default for most orgs, so it's unfortunate that it causes this wrinkle for local dev using a .netrc
.
:wave: everyone
Just stumbled upon this repo/issue almost two years later. :)
I'm hoping to get this buildpack updated for compatibility with the Heroku successor to the
user-env-compile
labs feature.The README mentions using this buildpack to keep secrets out of
package.json
, but it's not clear to me how this works in practice. I did a little investigating and it doesn't appear thatnpm
uses.netrc
or environment variables when fetching dependencies.@timshadel, can you shed some light on this?
cc @friism