more ghsetup issues

[navidcy]

I'm still struggling with payu ghsetup

I made a clean git repo and also made sure that I have collate: True in my config.yaml.

But still I get this:

Initialized empty Git repository in /home/552/nc3020/SOchanBcBtEddySat/layer2/testrepo2/.git/
[nc3020@raijin4 testrepo2]$ git add *
[nc3020@raijin4 testrepo2]$ git commit -m "first commit"
[master (root-commit) 76f9945] first commit
 8 files changed, 1510 insertions(+), 0 deletions(-)
 create mode 100644 MOM_input
 create mode 100644 MOM_override
 create mode 100644 config.yaml
 create mode 100644 diag_table
 create mode 100644 input.nml
 create mode 100644 la2_t6e-1ms.e1500038
 create mode 100644 la2_t6e-1ms.o1500038
 create mode 100755 sync_output_to_gdata.sh
[nc3020@raijin4 testrepo2]$ payu ghsetup
Enter github username: navidcy
Enter navidcy@github password: *********
[nc3020@raijin4 testrepo2]$ git push --set-upstream payu master
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[navidcy]

OK, this was resolved when I set up my github SSH key in raijin. (I thought that ghsetup was supposed to do that for me... no?)

[marshallward]

You shouldn't have to add the key manually, that seems like a payu error.

My best guess is that there is an old key which could be causing the problem? You may want to try and check if its using the wrong key in $HOME/.ssh/payu.

Currently we are not doing a very good job of managing the keys, and I could imagine many potential problems of mixups or misplaced keys.

This one might be hard to sort out without looking through your .ssh directory and your public keys, which we probably don't want to do over GitHub issues :). I won't be in until next week, but if you notice anything suspicious that would help.

I'll also try to stress test it a bit more, thanks for letting me know about the problem.

[marshallward]

Also, the place to look is in the repository-specific keys (under repository settings), not your account-wide user keys.

[navidcy]

For example, for the testrepo folder I created with ghsetup I see (i) a key in ~/.ssh/payu/id_rsa_payu_testrepo and ~/.ssh/payu/id_rsa_payu_testrepo.pub and (ii) also in GitHub in the repo settings I see a deploy key with name payu and read/write access. However, git push -v payu master was giving me the

Permission denied (publickey).
fatal: Could not read from remote repository. 

(Now that I've set up my own GitHub public key it works.)

[marshallward]

Can you try to replicate your steps and then confirm that the *.pub key in your .ssh/payu matches the public key in the repo?

[navidcy]

Shall I delete the other GitHub key that I've set up manually first?

On Oct 25, 2018, at 8:02 PM, Marshall Ward notifications@github.com wrote:

Can you try to replicate your steps and then confirm that the *.pub key in your .ssh/payu matches the public key in the repo?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/marshallward/payu/issues/135#issuecomment-432971942, or mute the thread https://github.com/notifications/unsubscribe-auth/AGyIQGG9-8FgjyxUssYJwdTjxItgSntjks5uoX4NgaJpZM4X5t7j.

[marshallward]

Right now I'm just wondering if it is using the wrong public key. If the keys don't match then it could be a problem.

[marshallward]

You can try this: remove your manual key and the key in the repo, delete the local keys in .ssh, and just try payu ghsetup again and see if it works.

It is more of a "reset" and won't explain the problem, but it's a baseline to make sure the basics are working.

[navidcy]

[nc3020@raijin4 testrepo]$ ls
MOM_input  MOM_override  config.yaml  diag_table  input.nml  sync_output_to_gdata.sh
[nc3020@raijin4 testrepo]$ ls ~/.ssh/
[nc3020@raijin4 testrepo]$ ls ~/.ssh/
[nc3020@raijin4 testrepo]$ payu ghsetup
Enter github username: navidcy
Enter navidcy@github password:*****
[nc3020@raijin4 testrepo]$ git push payu
Warning: Permanently added 'github.com,192.30.255.112' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
[nc3020@raijin4 testrepo]$

[marshallward]

oh you may also need to remove your remotes in the repository: git remote rm payu

(Actually, maybe just wipe the .git directory altogether if you don't need it?)

(Also, I just meant to delete the specific keys in ~/.ssh/payu/testrepo*, I hope you did not need any of those!)

[navidcy]

I ended up deleting all of myssh keys and now I can't even login to raijin or to some other machines I used to do without entering password! perhaps we should continue these experiments in person. :)

bottom line though is that the ghsetup functionality is currently not working...

[marshallward]

So sorry about that :( that ended up being very disruptive

Just retracing things, did you initialise your own repo with git init and then manually add the files? That's the only difference I can see here.

[navidcy]

I did that at some point but it didn't work either.

On Oct 25, 2018, at 8:41 PM, Marshall Ward notifications@github.com wrote:

So sorry about that :( that ended up being very disruptive

Just retracing things, did you initialise your own repo with git init and then manually add the files? That's the only difference I can see here.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/marshallward/payu/issues/135#issuecomment-432985208, or mute the thread https://github.com/notifications/unsubscribe-auth/AGyIQHvK5hDJRANO_Q7M9NS5-l2qujxOks5uoYdZgaJpZM4X5t7j.

[navidcy]

(Do you know how do I setup login to ssh without passwords? I always forget how to do it...)

On Oct 25, 2018, at 8:41 PM, Marshall Ward notifications@github.com wrote:

So sorry about that :( that ended up being very disruptive

Just retracing things, did you initialise your own repo with git init and then manually add the files? That's the only difference I can see here.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/marshallward/payu/issues/135#issuecomment-432985208, or mute the thread https://github.com/notifications/unsubscribe-auth/AGyIQHvK5hDJRANO_Q7M9NS5-l2qujxOks5uoYdZgaJpZM4X5t7j.

[marshallward]

I think the issue is you did git push --set-upstream payu master rather than payu push, which does some additional key management.

I think the biggest lesson here is that the internal git repo is not very resilient to explicit git commands.

ssh-keygen will generate new keys on your local machine, and you can add the public key to .ssh/authorized_keys (ed: on the remote machine).

[marshallward]

I have reproduced this, it's because a manual git push does not use the key.

Running payu ghsetup does add your key to ~/.ssh/payu, but you must run git push ~the command~ through the ssh agent to use the key. This is what payu push does.

ssh-agent bash -c "ssh-add {key}; git push --all payu"

A basic git push, or even in your case of setting the payu remote as your upstream, is not sufficient, since a key is required to interact with the repository without login.

---

Now having said that...

It's easy for me to say "Don't do git push - or if you do, then don't expect it to interact with the repository created by payu ghsetup. The issue is that the github repo was not envisioned as a fully interactive repository that one would work from. Rather, it was seen as more of an archive to track a user's runs and local changes.

This is also why the keys are restricted to a single repository, and are not easily accessible.

But if people are expecting the repository to behave as if they had created it themselves, and can easily push and pull to it, then we need to redesign things, and probably talk it over as a group.

One possibly solution would be to add an upstream ~report~ remote ("origin"), and just use HTTPS for password prompts. That works easily in your case, but could be problematic for people who cloned another person's experiment.

---

I am not really sure how to proceed here. But I think that we probably need more discussion about how to handle this.

Also, I am really sorry about your keys, I hope that it is not too hard to recover them.

[navidcy]

I don't disagree with the approach.

The only reason I did git push is simply because I'm so used to do that. I didn't remember that in the tutorial you said that we should do payu push...