First git push in rrtools::use_github seems to be broken

benmarwick / rrtools

rrtools: Tools for Writing Reproducible Research in R

Other

670 stars 85 forks source link

First git push in rrtools::use_github seems to be broken #71

Closed nevrome closed 5 years ago

nevrome commented 5 years ago

In a new repository with git enabled:

> rrtools::use_github(auth_token = "1234", private = FALSE)
✔ Git is already initialized
✔ Checking title and description
✔   Title: What the Package Does (One Line, Title Case)
✔   Description: What the package does (one paragraph)
Are title and description ok?
1: Yup
2: No way
3: Uhhhh... Maybe?

Selection: 1
* Creating GitHub repository
✔ Adding GitHub remote
✔ Adding GitHub links to DESCRIPTION
* Pushing to GitHub and setting remote tracking branch
Error in git2r::push(r, "origin", "refs/heads/master", credentials = credentials) : 
  Error in 'git2r_push': error authenticating:

The problem seems to be how git2r::push() acquires ssh credentials. Maybe it's just a user error or a problem with my system. But my ssh-agent ist running and I see no problem here.

> sessioninfo::session_info()
─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.5.1 (2018-07-02)
 os       Manjaro Linux               
 system   x86_64, linux-gnu           
 ui       RStudio                     
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       Europe/Berlin               
 date     2018-11-02                  

─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version date       lib source                          
 assertthat    0.2.0   2017-04-11 [1] CRAN (R 3.5.1)                  
 backports     1.1.2   2017-12-13 [1] CRAN (R 3.5.1)                  
 cli           1.0.1   2018-09-25 [1] CRAN (R 3.5.1)                  
 clisymbols    1.2.0   2017-05-21 [1] CRAN (R 3.5.1)                  
 crayon        1.3.4   2017-09-16 [1] CRAN (R 3.5.1)                  
 curl          3.2     2018-03-28 [1] CRAN (R 3.5.1)                  
 digest        0.6.18  2018-10-10 [1] CRAN (R 3.5.1)                  
 evaluate      0.12    2018-10-09 [1] CRAN (R 3.5.1)                  
 fs            1.2.6   2018-08-23 [1] CRAN (R 3.5.1)                  
 git2r         0.23.0  2018-07-17 [1] CRAN (R 3.5.1)                  
 glue          1.3.0   2018-07-17 [1] CRAN (R 3.5.1)                  
 htmltools     0.3.6   2017-04-28 [1] CRAN (R 3.5.1)                  
 knitr         1.20    2018-02-20 [1] CRAN (R 3.5.1)                  
 Rcpp          0.12.19 2018-10-01 [1] CRAN (R 3.5.1)                  
 rmarkdown     1.10    2018-06-11 [1] CRAN (R 3.5.1)                  
 rprojroot     1.3-2   2018-01-03 [1] CRAN (R 3.5.1)                  
 rrtools     * 0.1.0   2018-11-02 [1] Github (nevrome/rrtools@06d2c2f)
 rstudioapi    0.8     2018-10-02 [1] CRAN (R 3.5.1)                  
 sessioninfo   1.1.0   2018-09-25 [1] CRAN (R 3.5.1)                  
 usethis       1.4.0   2018-08-14 [1] CRAN (R 3.5.1)                  
 withr         2.1.2   2018-03-15 [1] CRAN (R 3.5.1)                  
 yaml          2.2.0   2018-07-25 [1] CRAN (R 3.5.1)                  

[1] /home/clemens/R/x86_64-pc-linux-gnu-library/3.5
[2] /usr/lib/R/library

benmarwick commented 5 years ago

Yes, I can reproduce this, and encountered it recently in a workshop. I guess it's not exactly that same, since the function that fails is devtools::github_POST, but the overall result of failing to contact GitHub is...

> rrtools::use_github(auth_token = "GITHUB_PAT", private = FALSE)
✔ Git is already initialized
✔ Checking title and description
✔   Title: What the Package Does (One Line, Title Case)
✔   Description: What the package does (one paragraph)
Are title and description ok?
1: Nope
2: I forget
3: Yes

Selection: 3
* Creating GitHub repository
Error in stop(github_error(req)) : Bad credentials (401)
> traceback()
4: stop(github_error(req))
3: github_response(req)
2: github_POST("user/repos", pat = auth_token, body = list(name = jsonlite::unbox(pkg$package), 
       description = jsonlite::unbox(gsub("\n", " ", pkg$title)), 
       private = jsonlite::unbox(private)), host = host)
1: rrtools::use_github(auth_token = "GITHUB_PAT", private = FALSE)

> sessioninfo::session_info()
─ Session info ───────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.5.1 (2018-07-02)
 os       macOS  10.14                
 system   x86_64, darwin15.6.0        
 ui       RStudio                     
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       America/Los_Angeles         
 date     2018-11-02                  

─ Packages ───────────────────────────────────────────────────────────
 package     * version     date       lib
 assertthat    0.2.0       2017-04-11 [1]
 backports     1.1.2       2017-12-13 [1]
 cli           1.0.1       2018-09-25 [1]
 clisymbols    1.2.0       2017-05-21 [1]
 crayon        1.3.4       2018-11-01 [1]
 curl          3.2         2018-03-28 [1]
 digest        0.6.18      2018-10-10 [1]
 evaluate      0.12        2018-10-09 [1]
 fs            1.2.6       2018-08-23 [1]
 git2r         0.23.0.9000 2018-10-22 [1]
 glue          1.3.0       2018-10-31 [1]
 htmltools     0.3.6       2017-04-28 [1]
 httr          1.3.1       2017-08-20 [1]
 jsonlite      1.5         2017-06-01 [1]
 knitr         1.20        2018-02-20 [1]
 R6            2.3.0       2018-10-04 [1]
 Rcpp          0.12.19     2018-10-01 [1]
 rmarkdown     1.10        2018-06-11 [1]
 rprojroot     1.3-2       2018-01-03 [1]
 rrtools       0.1.0       2018-10-23 [1]
 rstudioapi    0.8         2018-10-02 [1]
 sessioninfo   1.1.0       2018-09-25 [1]
 usethis       1.4.0       2018-08-14 [1]
 withr         2.1.2       2018-03-15 [1]
 yaml          2.2.0       2018-07-25 [1]
 source                             
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 Github (gaborcsardi/crayon@0398b12)
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 Github (ropensci/git2r@e18e8f7)    
 Github (tidyverse/glue@35c61e9)    
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 Github (benmarwick/rrtools@872c839)
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     
 CRAN (R 3.5.0)                     

[1] /Library/Frameworks/R.framework/Versions/3.5/Resources/library

nevrome commented 5 years ago

Ok - I solved my issue. And it only took me four hours! I have to reconsider my life choices.

For me it works with rrtools::use_github(..., credentials = git2r::cred_ssh_key()). Would you mind testing this solution in your environment? Maybe this is a better default for credentials than NULL. At least the documentation should mention this possibility explicitly. Error in 'git2r_push': error authenticating: didn't exactly point me to the solution.

Now about why this took me four hours: The two problems -- your's and mine -- are not related. My problem was based on the authentication necessary for git2r::push(). This works via ssh keys and is direct communication of git client and git remote server. The problem you experienced is caused by problems in the interaction with the github API. To create a repository on github you need API access. That's not necessary for pulling/pushing/etc in a already created repository. The authentication with the github API requires an API token (GITHUB_PAT).

This GITHUB_PAT needs

public_repo scope or repo scope to create a public repository
repo scope to create a private repository

I suggest to also add this information to the documentation of use_github because: If you forget to enable the correct scope you get an Error 404 without further information. In search for the reason of this API request error I dismantled the interaction between rrtools->httr->curl. This was completely useless but kind of interesting. There are some small differences now between the copied functions in rrtools and the versions here. Maybe an update makes sense now.

I can offer a PR with my suggestions if you like them.

Concerning your error: How did you call this function? Just to make sure: You replaced the placeholder GITHUB_PAT with an actual token when you ran it, ja? You didn't actually ran

rrtools::use_github(auth_token = "GITHUB_PAT", private = FALSE)

benmarwick commented 5 years ago

Thanks for taking a look. Maybe you are right that we need to update the code we pasted in from devtools.

I'm getting a repo made on GitHub, but the push doesn't work:

> rrtools::use_github(auth_token = "...", private = FALSE, credentials = git2r::cred_ssh_key())
✔ Git is already initialized
✔ Checking title and description
✔   Title: What the Package Does (One Line, Title Case)
✔   Description: What the package does (one paragraph)
Are title and description ok?
1: No
2: Definitely
3: No way

Selection: 2
* Creating GitHub repository
✔ Adding GitHub remote
✔ Adding GitHub links to DESCRIPTION
* Pushing to GitHub and setting remote tracking branch
Error in git2r::push(r, "origin", "refs/heads/master", credentials = credentials) : 
  Error in 'git2r_push': Failed to authenticate SSH session: Unable to send userauth-publickey request

Here's what I need to in advance:

Need to create SSH keys: http://happygitwithr.com/ssh-keys.html
Need to put GITHUB_PAT in env: https://usethis.r-lib.org/articles/articles/usethis-setup.html#get-and-store-a-github-personal-access-token. Although the GITHUB_PAT method that I previously used doesn't work at the moment. I need to Sys.getenv("GITHUB_PAT") to get it, and paste it into the use_github fn, Yes, I did actually run rrtools::use_github(auth_token = "GITHUB_PAT", private = FALSE)

nevrome commented 5 years ago

I mean - the new devtools code doesn't solve this problems, but maybe it prevents future ones.

SSH Keys for git2r::push():

Interesting that git2r::cred_ssh_key() doesn't work for you. This function uses git2r::ssh_path() to find the key files and

provides a consistent means across OS-types to access the .ssh directory.

Do you store your keys at a non standard location? Or does ssh key management work completely different on MacOS? In the end it doesn't matter for me, but it would be nice to have a default setting that works in pretty much all default environments. On Unix-alike systems ssh keys are usually stored in ~/.ssh and git2r::cred_ssh_key() should work. On the other hand an ssh agent IS a nice tool if your keys have a passphrase. I don't know why the interaction of git2r and the agent doesn't automatically work if I call rrtools::use_github(..., credentials = NULL). Probably something wrong in libgit2 or a problem with my individual setup.

GITHUB_PAT for devtools::github_POST():

So this is solved with some more documentation, ja?

benmarwick commented 5 years ago

I got a new GitHub token and it all works fine for me now... But not if I use GITHUB_PAT as a system env var, so that's a bit odd. So should update the README a little to mention the need for SSH keys?

No luck with GITHUB_PAT:

> usethis::use_github(auth_token = "GITHUB_PAT", protocol = "https", private = FALSE)
✔ Setting active project to '/Users/bmarwick/pkgname'
● Check title and description
  Name:        pkgname
  Description: What the Package Does (One Line, Title Case)
Are title and description ok?
1: No
2: Yes
3: Not now

Selection: 2
✔ Creating GitHub repository
Error in gh::gh("POST /user/repos", name = repo_name, description = repo_desc,  : 
  GitHub API error (401): 401 Unauthorized
  Bad credentials
> Sys.getenv("GITHUB_PAT")
[1] "..."

But if I use the PAT as-is, all is well:

> usethis::use_github(auth_token = "...", protocol = "https", private = FALSE)
● Check title and description
  Name:        pkgname
  Description: What the Package Does (One Line, Title Case)
Are title and description ok?
1: Not now
2: Negative
3: Absolutely

Selection: 3
✔ Creating GitHub repository
✔ Adding GitHub remote
✔ Adding GitHub links to DESCRIPTION
✔ Setting URL field in DESCRIPTION to 'https://github.com/benmarwick/pkgname'
✔ Setting BugReports field in DESCRIPTION to 'https://github.com/benmarwick/pkgname/issues'
✔ Pushing to GitHub and setting remote tracking branch
✔ Opening URL https://github.com/benmarwick/pkgname

> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS  10.14.1

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets 
[6] methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0        clisymbols_1.2.0 
 [3] assertthat_0.2.0  crayon_1.3.4     
 [5] rprojroot_1.3-2   R6_2.3.0         
 [7] jsonlite_1.5      backports_1.1.2  
 [9] git2r_0.23.0.9000 httr_1.3.1       
[11] curl_3.2          rstudioapi_0.8   
[13] fs_1.2.6          ini_0.3.1        
[15] gh_1.0.1          desc_1.2.0       
[17] tools_3.5.1       glue_1.3.0       
[19] yaml_2.2.0        compiler_3.5.1   
[21] usethis_1.4.0

benmarwick commented 5 years ago

At today's workshop in Oxford we saw several computers with this error:

usethis::use_github(auth_token = "...", protocol = "https", private = FALSE)
[...]
✔ Creating GitHub repository
Error in gh::gh("POST /user/repos", name = repo_name, description = repo_desc,  : 
  GitHub API error (401): 401 Unauthorized
  Bad credentials

nevrome commented 5 years ago

rrtools is suffering from API fever attacks. This is yet another error.

I mean... It's clear that usethis::use_github(auth_token = "GITHUB_PAT", protocol = "https", private = FALSE) does not work: In this case you sent the actual string GITHUB_PAT as the PAT.

If usethis::use_github(auth_token = "...", protocol = "https", private = FALSE) fails I would first of all check if the users copied the PATs correctly to replace "..." . An additional whitespace could probably already cause problems. This especially makes sense if only some users experience the issue.

benmarwick commented 5 years ago

I'm still finding this step unreliable. Related, perhaps: https://cran.r-project.org/web/packages/credentials/vignettes/intro.html

nevrome commented 5 years ago

I believe this issue can me closed, although it remains technically unsolved. Hear me out:

I suggeste to remove rrtools::use_github() with #83. The rrtools workflow will completely rely on usethis::use_github(), if this PR is merged.
Therefore the problems described here should be discussed in the usethis issue forum.
The latest version of usethis::use_github() seems to deal better with these kind of credentials related errors. The user gets more info and more useful hints when something goes wrong.

In the rrtools README section about git/github I would change two small things:

Add credentials = git2r::cred_ssh_key() to the usethis call in the title. I believe that's a good default.
To See Happy Git With R for details on how to do this. I would add Make sure to set up a SSH key to connect to your Github account conveniently.

benmarwick commented 5 years ago

Thanks for these suggestions, those are great observations. I've added the text to the README as your propose in https://github.com/benmarwick/rrtools/commit/7aea08f9c97b6f571e7f5acd62675847085753c2. Afte adding my ssh key, following the instructions in Happy Git With R I got this, which although not perfect, was enough that I could easily go the last mile by myself. So that's a big improvement, thanks again!

> usethis::use_github(credentials = git2r::cred_ssh_key(), protocol = "https", private = FALSE)
✔ Setting active project to '/Users/bmarwick/pkgname2'
✔ Checking that current branch is 'master'
● Check title and description
  Name:        pkgname2
  Description: What the Package Does (One Line, Title Case)
Are title and description ok?

1: Absolutely not
2: Definitely
3: Negative

Selection: 2
✔ Creating GitHub repository
✔ Setting remote 'origin' to 'https://github.com/benmarwick/pkgname2.git'
✔ Adding GitHub links to DESCRIPTION
✔ Setting URL field in DESCRIPTION to 'https://github.com/benmarwick/pkgname2'
✔ Setting BugReports field in DESCRIPTION to 'https://github.com/benmarwick/pkgname2/issues'
✔ Pushing 'master' branch to GitHub and setting remote tracking branch
● Failed to push and set tracking branch.
  This often indicates a problem with git2r and the credentials.
  Try this in the shell, to complete the set up:
  `git push --set-upstream origin master`
✔ Opening URL 'https://github.com/benmarwick/pkgname2'