Auth issue in demo

[yadudoc]

I've updated ~/Virtualenvs/AIMES-DEMO-SC2014/bin/demo_SC2014_env_setup.sh and commented out every site besides blacklight in ~/Virtualenvs/AIMES-DEMO-SC2014/etc/bundle_demo_SC2014.conf. But the run seems to fail when it tries to connect to resources as "mturilli".

env_setup.sh : https://gist.github.com/yadudoc/5411b773def8ee988510 Bundle conf : https://gist.github.com/yadudoc/adc07adc7c80e03d7fcb I have ssh keys setup for access to blacklight,

Here's the output I see: https://gist.github.com/yadudoc/8f620a6bebea4dc32c1c

[mturilli]

Hi yadudoc,

I see three problems:

1. Blacklight is designed to fail with this demo, feel free to choose any resource but blacklight. I will try to give more emphases to the following in the README, apologies: Note that the pilot on blacklight is supposed to fail. This illustrates the fault tolerant properties of the scheduler used to late-bind the tasks of the given skeleton on a dynamic number of pilots.
2. You need to change the permissions of your private key located at /home/yadu/.ssh/myCert.pem to 600. This is what the message you linked is asking for: Permissions 0644 for '/home/yadu/.ssh/myCert.pem' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. bad permissions: ignore key: /home/yadu/.ssh/myCert.pem
3. There is a bug in the demo script. It uses always mturilli for every other user than merzky. I will push a patch for this bug shortly.

Thank you for opening a ticket and uncovering the bug, Matteo

[mturilli]

I have patched the bug mentioned above and run a successful test. Please let me know whether it works for you too.

[yadudoc]

Hi Matteo, thanks for the quick fix. I've pulled your changes and added gordon, trestles, stampede in addition to blacklight to the setup but I'm still running into issues with the keys. I've specified my key using key_filename in bundle.conf, but the errors I see are complaining about a different key. Here's the error message I'm seeing : https://gist.github.com/yadudoc/37b8ccdebb118d3a2096

I have ssh-agents so the key should be unlocked. I'm worried if I'm specifying keys as the demo scripts expects it. -Yadu

[mturilli]

Hi Yadu,

Thank you for testing the patch I submitted.

Looking at the logs you linked in your messages, I see key_filename=/home/yadu/.ssh/id_rsa specified in your bundle configuration file but the script seems to be trying to use /home/yadu/.ssh/myCert.pem. It should be noted that the bundle authentication process bundle is independent from the one used by the pilot system and, therefore, by the demo script.

I can patch the demo script so for you to be able to specify the key you want to use for the pilot session, namely the key that will be used to access the remote resources in order to submit a pilotjob, but before doing that, would you mind trying from the same shell from which you are attempting to run the demo to run the following command:

ssh yadunand@stampede.tacc.xsede.org

and see whether you can log into stampede without being asked for a password? If so, could you also run the following:

ssh -vv yadunand@stampede.tacc.xsede.org

And see what key was used for the authentication?

Many thanks, Matteo

[yadudoc]

Hi Matteo,

I was using the id_rsa key specifically because that's the one I've setup on all the compute resources I have access to. So ssh yadunand@stampede.tacc.xsede.org , gets me logged in right away. I tried to specify the key file explicitly so that the scripts would avoid using the other keys in the .ssh folder. If there is any additional logs/debug info I can get you, or a skype/im session to debug live, let me know.

Plus I checked the verbose logs

debug1: Offering RSA public key: /home/yadu/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp ba:8d:d9:ec:f5:05:54:60:f4:0a:4e:7b:ea:78:b4:1a
debug1: Authentication succeeded (publickey).

Thanks! Yadu

[mturilli]

Hi Yadu,

Thank you for running these tests. Does the file /home/yadu/.ssh/myCert.pem exists at all? I am trying to understand why it gets picked up by default.

Many thanks, Matteo

[yadudoc]

Yes, /home/yadu/.ssh/myCert.pem exists. There are bunch of different keys in the .ssh folder, I do not know why that file is being picked up.

[mturilli]

Thank you. I will open a ticket with the pilot/SAGA developers and see what are the criteria used to pick up the default key. Meanwhile, I will implement a patch for the current demo script. This will require a bit of work, including updates to the documentation. I will update this ticket once the patch will be pushed.

On Thu, Feb 5, 2015 at 11:25 PM, Yadu Nand B notifications@github.com wrote:

Yes, /home/yadu/.ssh/myCert.pem exists. There are bunch of different keys in the .ssh folder, I do not know why that file is being picked up.

— Reply to this email directly or view it on GitHub https://github.com/mturilli/aimes.emanager/issues/2#issuecomment-73143136 .

Dr Matteo Turilli Department of Electrical and Computer Engineering Rutgers University

[andre-merzky]

The log attached to the first message contains:

Permissions 0644 for '/home/yadu/.ssh/myCert.pem' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /home/yadu/.ssh/myCert.pem

does changing the permissions change anything?

chmod 0600 /home/yadu/.ssh/myCert.pem

Thanks, Andre.

[mturilli]

Hi Andre,

Thank you for stepping in. Looking at the last log linked in the ticket, it seems the message has gone. What are the criteria used by RP/SAGA to choose a key among those available in the user .ssh/ directory?

The patch I am going to work on will allow for the user of the script to specify the designated ssh key with an environment variable. That path+key will then be added explicitly to the RP session.

[andre-merzky]

Matteo,

the demo script creates an ssh session context, at https://github.com/mturilli/aimes.emanager/blob/master/bin/demo_SC2014_script.py#L671 -- that contains a username, but does not point to a specific ssh key. At that point SAGA will more or less randomly pick a key, and seems to stumble over the .pem one. (BTW: pem is a different key format than usually used, I would guess this comes from an gsissh setup or something?)

Anyway, you may want to try to convince SAGA to use a specific key, by adding it to the context, as

context.user_key = '/home/user/.ssh/id_rsa'

That should translate into the right ssh settings, see https://github.com/radical-cybertools/saga-python/blob/master/saga/utils/pty_shell_factory.py#L558

HtH, Andre.

[andre-merzky]

PS.: I'll open a SAGA ticket anyway -- I think we should ignore pem keys for ssh, or at least should not pick them up as default keys...

[andre-merzky]

Hi Matteo -- sorry for the message spam, now our messages crossed :P

Re key selection: you see the mechanism here: https://github.com/radical-cybertools/saga-python/blob/master/saga/adaptors/context/ssh.py#L106

We explicitly include .pem keys, IIRC for gsissh setups, but if that makes trouble, I don't mind removing them from the default set...

[mturilli]

Hi Andre,

No worries about the messages, always very much appreciated :) Re the default selection criteria, I would go with the default naming used by ssh. I would patch RP/SAGA session so to pick id_rsa for ssh. For gsissh I would use instead usercert.pem as that is more or less the accepted default name in the gsi* world. What do you think?

Meanwhile, I will proceed with the described patching.

Thanks! Matteo

[andre-merzky]

Hi Matteo,

Re the default selection criteria, I would go with the default naming used by ssh. I would patch RP/SAGA session so to pick id_rsa for ssh. For gsissh I would use instead usercert.pem as that is more or less the accepted default name in the gsi* world. What do you think?

I think I would find that cumbersome. I use different ssh keys (usually not the default key) for different resources, and right now this just works out of the box. Changing that to only pick up the default ssh key would mean I'd have to create an ssh context for each ssh key I want to potentially use, for each application/test/example etc. So I am hesitant :)

I'll try to find a way to detect if the pem is passwd protected, that should avoid troubles like the one @yadudoc had...

[mturilli]

Hi Yadu,

I have committed and tested a patch for this issue. Please have a look at the configuration section of the README. You will have to edit demo_SC2014_env_setup.sh and set the variable AIMES_USER_KEY to the path of the public ssh key you want to use for the demo.

Please let me know whether it works for you too.

Best, Matteo

[yadudoc]

Hi Matteo,

I updated the repo and ran again with AIMES_USER_KEY set, but I still ran into the same issue as earlier: Caught exception: prompted for unknown key password (/home/yadu/.ssh/myCert.pem)

So, I decided to try moving the non-relevant keys to a separate folder. When I run, I get new errors : http://pastebin.com/rRGdUXJ6

I do not understand where this prompt for a password is coming from, I have ssh agents setup, so I can login to all the target resources without being prompted for a password.

Thanks, Yadu

[andre-merzky]

Hi @yadudoc -- would you please run again with 'SAGA_VERBOSE=DEBUG' set in your environment, and point us to the debug output?

Many thanks, Andre.

[andre-merzky]

@yadudoc, could you please try again with the changes committed by Matteo, specifically with the changed URL for the bundle database? A fresh virtualenv might also be great.

I hope the bundle DB url fix will solve your auth problem - if not, please ping us again! :)

[mturilli]

Hi Yadu, can we consider this issue addressed? I know the demo still fails for you but, as far as I can see, the auth issue seems to have been addressed.