Blank creds file despite success reported in the UI

[fperez]

This week, since updating to your branch after the merge of my PR, I've seen a few times a failure I'd never encountered before. I run through the standard sequence, it reports success (green box, etc), but the creds file is empty:

(base) jupyter-fernando-2eperez[~]> cat /tmp/github-app-git-credentials 
(base) jupyter-fernando-2eperez[~]> d /tmp/github-app-git-credentials 
/home/jovyan
-rw------- 1 jovyan 0 Apr 23 06:01 /tmp/github-app-git-credentials

and obviously it doesn't actually work.

I had to run it again, and interestingly, when I looked, the creds file was actually empty at first, though a few seconds later it appeared with data:

(base) jupyter-fernando-2eperez[~]> cat /tmp/github-app-git-credentials 
(base) jupyter-fernando-2eperez[~]> cat /tmp/github-app-git-credentials 
https://x-access-token:ghu_[==============ELIDED===============]@github.com

Sorry the above aren't timestamped, but both cat cmds were run after having executed the notebook with:

and within a few seconds of one another. The first cat gave an empty file, then I tried again and it showed content.

Any ideas of what could be going on?

[yuvipanda]

@fperez the credentials file is autogenerated now, doesn't use a static filename. Look at ~/.gitconfig, it should tell you where the files are going.

[fperez]

Perhaps my report wasn't clear enough? The file is there, I can see it, but it's empty. On one occasion it was left empty, on the other it was empty at first, and then it showed content after a few seconds. In both cases, this was all after the UI declaring success with the green badge.

I first noticed the problem b/c it simply didn't work: I ran it, it said success, and when I tried to use Github, it asked me for username/pwd. I hadn't seen this failure before, so I'm wondering if you have... I'd used it daily for several months without issue, and now it's failed 2-3 times in the last week, so it seems something changed (or perhaps it's a coincidence and something else changed on the hub?).

[yuvipanda]

@fperez so with https://github.com/yuvipanda/gh-scoped-creds/pull/19, the file name is no longer hardcoded to /tmp/github-app-git-credentials, but automatically securely generated each time you run the script. Then we add an entry in ~/.gitconfig to tell git where to look for it (https://github.com/yuvipanda/gh-scoped-creds/pull/14). This makes the package safe to use from HPC systems (as otherwise /tmp is shared between users).

So I'm curious how /tmp/github-app-git-credentials is being populated at all! Can you check your ~/.gitconfig file to see what is being written there?

Sorry my earlier comment wasn't clear.

[yuvipanda]

I also have a pretty new (4.0) version of gh-scoped-creds installed in the stat159 hub (https://github.com/berkeley-dsep-infra/datahub/pull/3357) - can you run pip list to see if you're getting that package?

[fperez]

Got it - this is what I see right now, in ˜/.gitconfig, there's:

[credential "https://github.com"]
        helper = store --file=/tmp/tmpjm_ke_mc

and interestingly, that file is identical to /tmp/github-app-git-credentials:

(base) jupyter-fernando-2eperez[tmp]> ls -l
total 100
-rw------- 1 jovyan jovyan    75 Apr 23 06:53 github-app-git-credentials
-rw------- 1 jovyan jovyan     0 Apr 21 04:54 tmp5bebrhc8
-rw------- 1 jovyan jovyan    75 Apr 22 01:37 tmp8si3lji9
-rw------- 1 jovyan jovyan     0 Apr 21 18:39 tmpdzok7udh
-rw------- 1 jovyan jovyan    75 Apr 21 04:54 tmphkz1goy0
-rw------- 1 jovyan jovyan     0 Apr 23 06:01 tmpiftvk2_l
-rw------- 1 jovyan jovyan    75 Apr 23 06:53 tmpjm_ke_mc
-rw------- 1 jovyan jovyan    75 Apr 22 01:44 tmpk_y7bn4c
drwx------ 2 jovyan jovyan  4096 Apr 20 18:43 tmpmfrbl1vj/
-rw------- 1 jovyan jovyan    75 Apr 22 01:35 tmpyfdmdll2
-rw------- 1 jovyan jovyan 66463 Apr 19 21:24 tmpzgm45fdy
-rw------- 1 jovyan jovyan    75 Apr 21 18:41 tmpzq1nvl_e

(base) jupyter-fernando-2eperez[tmp]> diff github-app-git-credentials tmpjm_ke_mc 
(base) jupyter-fernando-2eperez[tmp]> 

BTW - if it makes debugging easier, feel free to hop in the hub as me, no worries. You have admin credentials on it to do that.

[fperez]

Oh, as for versions, I see this:

(notebook) (main)jupyter-fernando-2eperez[homework-dev]> pip list | grep scope
gh-scoped-creds                   4.0

[yuvipanda]

Very interesting that github-app-git-credentials exists at all! I'll dig in to see why that is happening

[fperez]

It just happened again - I ran the import/magic and it reported success, yet it doesn't work b/c it created the tmp file and github-app-git-credentials together, but both empty:

In git config:

[credential "https://github.com"]
        helper = store --file=/tmp/tmp8kt1wyjj

In /tmp:

-rw------- 1 jovyan     0 Apr 27 05:43 github-app-git-credentials
-rw------- 1 jovyan     0 Apr 27 05:43 tmp8kt1wyjj

And when I try to use it, no surprise:

(notebook) (main)jupyter-fernando-2eperez[site]> git push
Username for 'https://github.com': 

These failures started after the recent update, before it had never broken. Now it only works sometimes, though typically I can get it to work after trying over and over. Could we perhaps revert to the old version for now? I'd like this to work reliably, it's a key element of our daily workflow.

[yuvipanda]

@fperez I couldn't reproduce it unfortunately. I can revert to an older version, but given the fact that the words 'github-app-git-credentials' do not even exist in 4.0 (including the version installed in the container), my suspicion is that this is coming from a leftover install somewhere in your customizations. Are your students also reporting this problem, or is it just you?

In particular, I see you have a checkout in /home/jovyan/dev/github-app-user-auth - perhaps there's a remnant of that installed somewhere? I also see you have . in your $PATH which can often result in difficult to debug errors where some code gets read in in ways you don't expect. Maybe something like that is happening?

I'm also bumping the default version to 4.1 in https://github.com/berkeley-dsep-infra/datahub/pull/3364, which outputs an additional 'Tip' about using https URLs - this will make it easier to see which version of the app is being invoked.

However, if this is a widespread problem (and not just affecting you), I can try revert to an older version as well.

[yuvipanda]

I also dug through your .gitconfigs and found .gitconfig.local with

(base) jupyter-fernando-2eperez[~]> cat .gitconfig.local 
[credential]
        helper = store --file=/tmp/github-app-git-credentials

Maybe this is what is part of what is causing the issue? Can you remove that?

[fperez]

Thx @yuvipanda - I think just wiping that stale .gitconfig.local file did the trick. I'll test some more, but it looks like that was enough...

[consideRatio]

Ahhh very clear discussion to follow retrospectively! It sounds plausible to me that the empty file was created by this:

(base) jupyter-fernando-2eperez[~]> cat .gitconfig.local 
[credential]
        helper = store --file=/tmp/github-app-git-credentials

I'll go for a close of this issue as there is no action point remaining to take as I understand it!