sassoftware / saspy

A Python interface module to the SAS System. It works with Linux, Windows, and Mainframe SAS as well as with SAS in Viya.
https://sassoftware.github.io/saspy
Other
375 stars 149 forks source link

SAS Connection failed. No connection established. #176

Closed bikashdube closed 6 years ago

bikashdube commented 6 years ago

@tomweber-sas

So I resurrected an old environment where things used to work fine. Did a pipdate to upgrade all packages in the environment first.

When I do a simple SAS code submit in a SAS Kernel cell, I am having to terminate the session as no response comes back.

Code I am submitting:

ods _all_ close;
proc setinit;
run;

Here is the log after I terminate and confirm termination.

Exception caught!
SAS attention handling not supported over ssh. Please enter (T) to terminate SAS or (C) to continue.T
SAS Connection failed. No connection established. Double check you settings in sascfg.py file.

Attempted to run program /usr/bin/ssh with the following parameters:['/usr/bin/ssh', '-t', 'sashost.domain.com', '/opt/sas', '-WORK', '/opt/Default_saswork_v9', '-noterminal', '-nodms', '-nosyntaxcheck', '-pagesize', 'MAX', '-fullstimer', '-nofmterr', '-memsize', '3072M', '-nodms', '-stdio', '-terminal', '-nosyntaxcheck', '-pagesize', 'MAX', '']
Try running the following command (where saspy is running) manually to see if you can get more information on what went wrong:
/usr/bin/ssh -t sashost.domain.com /opt/sas -WORK /opt/Default_saswork_v9 -noterminal -nodms -nosyntaxcheck -pagesize MAX -fullstimer -nofmterr -memsize 3072M -nodms -stdio -terminal -nosyntaxcheck -pagesize MAX  
No SAS process attached. SAS process has terminated unexpectedly.
No SAS process attached. SAS process has terminated unexpectedly.
No SAS process attached. SAS process has terminated unexpectedly.
No SAS process attached. SAS process has terminated unexpectedly.
No SAS process attached. SAS process has terminated unexpectedly.
No SAS process attached. SAS process has terminated unexpectedly. 
Restarting SAS session on your behalf
in shutdown function

I am not sure where the duplicate options are coming after the 3072M option. Here is the full sascfg_personal.py

SAS_config_names=['ssh']

SAS_config_options = {'lock_down': False,
                      'verbose'  : True
                     }
SAS_output_options = {'output' : 'html5'}

ssh      = {'saspath' : '/opt/sas',
            'ssh'     : '/usr/bin/ssh',
            'host'    : 'sashost.domain.com', 
            'encoding': 'latin1',
            'options' : ["-WORK", "/opt/Default_saswork_v9", "-noterminal", "-nodms", "-nosyntaxcheck", "-pagesize","MAX","-fullstimer", "-nofmterr", "-memsize", "3072M"]
            }

When I run the ssh command in terminal, I get a successful response. So the ssh keys etc are all setup correctly.

Here is a sample response (partial shown)


NOTE: SAS initialization used:
      real time           0.03 seconds
      user cpu time       0.01 seconds
      system cpu time     0.02 seconds
      memory              3567.21k
      OS Memory           4504.00k
      Timestamp           10/16/2018 08:42:34 AM
      Step Count                        0  Switch Count  6
      Page Faults                       0
      Page Reclaims                     5119
      Page Swaps                        0
      Voluntary Context Switches        124
      Involuntary Context Switches      7
      Block Input Operations            88
      Block Output Operations           88

Similar error occurs when I try this in a Python3 kernel notebook as well.

 SAS (r) Proprietary Software 9.4 (TS1M2)
jupyter                         1.0.0
jupyter-client                  5.2.3
jupyter-console                 6.0.0
jupyter-contrib-core            0.3.3
jupyter-core                    4.4.0
jupyter-highlight-selected-word 0.2.0
jupyter-latex-envs              1.4.4
jupyterhub                      0.9.4
jupyterlab                      0.35.1
jupyterlab-launcher             0.13.1
jupyterlab-server               0.2.0
SAS-kernel                      2.1.7
saspy                           2.2.9
tomweber-sas commented 6 years ago

@bikashdube Well, I've tried this out myself and I'm not getting any problems. I initially suspected the

ods _all_ close;

you were submitting, as that would interfere with how saspy works with regard to getting output; it uses specific ODS statements to get and return output. However, proc setinit doesn't generate any output, it just writes to the log, which is simply text, so that isn't happening to be a problem for this code you submitted. I wouldn't submit that when running any code you expect to get output from though, as that would keep you from getting any ouput. Also, the 'duplicate' options, saspy's STDIO access method requires specific SAS system startup options to be able to work. Those options are submitted by saspy to run SAS with this access methods. You happen to put them into your options key also, which isn't needed. The other options like saswork and memsize, fullstimer... things like that are what the options: key is for. The specific startup options

-nodms -stdio -terminal -nosyntaxcheck -pagesize MAX

will be applied by saspy.

So, that gets us back to what is the problem here? One thing I'm noticing trying to run the sas_kernel is that it's taking a long time to start. I don't know why, but could that be what's happening? How long are you waiting for it to run?

But, let's do something proactive to see what's happening. We can't get much diagnostics from the sas_kernel, so let's try the python kernel and saspy directly. So 2 thisgs to start. Did you run

/usr/bin/ssh -t sashost.domain.com /opt/sas -WORK /opt/Default_saswork_v9 -noterminal -nodms -nosyntaxcheck -pagesize MAX -fullstimer -nofmterr -memsize 3072M -nodms -stdio -terminal -nosyntaxcheck -pagesize MAX

from the command line to see if there's actually something keeping this from working?

The other thing to try is to just run these same things using saspy and then we can look at the whole saslog and see if there's anything suspicious, assuming you are successfully connecting to SAS; the run from the command like would tell us that.

I've uploaded 2 notebooks, one sas_kernel and the other saspy, to show both of these doing the proc setinit and a proc print, with and without the ods all close to show that that isn't causing a problem, but it will keep you from getting output.

These are in the saspy-example repo in the issues folder: https://github.com/sassoftware/saspy-examples/blob/master/Issue_examples/Issue176_SAS_kernel.ipynb https://github.com/sassoftware/saspy-examples/blob/master/Issue_examples/Issue176_saspy.ipynb

I'm not having much luck with github rendering these notebooks. Sometime it does, but sometimes is doesn't. You can download any of these notebooks on that site and then upload them into your jupyter and they should resolve to see them, and you can also then run them then also.

The saspy version is simply running the following code, in case you can't get it to render either:

import saspy
sas = saspy.SASsession()
sas
ll = sas.submit('''
ods _all_ close;
proc setinit;
run;
''')
print(ll['LOG'])

ll = sas.submit('''
ods _all_ close;
proc print data=sashelp.cars(obs=5);run;
''')
print(ll['LST'])
len(ll['LST'])

ll = sas.submit('''
proc print data=sashelp.cars(obs=5);run;
''')
print(ll['LOG'])
from IPython.display import HTML 
HTML(ll['LST'])

To see the full log in saspy, you use the saslog() method of the SASsession object:

sas = saspy.SASsession()

print(sas.saslog())

Can you try out these things and see what that shows?

Thanks, Tom

bikashdube commented 6 years ago

@tomweber-sas

Yes the following works well for the user that spawns the notebook. A different non-root user owns the jupyterhub and configurable-http-proxy linux processes(sudospawner). The -noterminal option is important just FYI as -terminal tries to launch the X-session and it is not something that we need via remote Jupyter submits. But thanks for explaining that it is coming from saspy code defaults.

/usr/bin/ssh -t sashost.domain.com /opt/sas -WORK /opt/Default_saswork_v9 -noterminal -nodms -nosyntaxcheck -pagesize MAX -fullstimer -nofmterr -memsize 3072M -nodms -stdio -terminal -nosyntaxcheck -pagesize MAX
bikashdube commented 6 years ago

The two example notebooks rendered fine for me as well. I agree if we can get this working in a Py3 kernel it will work in SAS Kernel as well, so I am trying to run all the python examples for debugging. So the first one itself just sits waiting...

import saspy
sas = saspy.SASsession()
sas

After I run this all I see is

Using SAS Config named: ssh

There is no subprocess ID getting reflected in output. On a terminal I can see that there is a valid process ID but the notebook is not getting that back.

hubuser  20396 20347 17 12:07 ?        00:07:18 /home/hub/.pyenv/versions/3.6.3/bin/python3.6 -m ipykernel_launcher -f /home/hubuser/.local/share/jupyter/runtime/kernel-48ea3649-2082-468a-aa72-fe198b95d31d.json

I interrupted the kernel and this is what I get after confirming to Terminate the session within the notebook.

Using SAS Config named: ssh
Exception caught!
SAS attention handling not supported over ssh. Please enter (T) to terminate SAS or (C) to continue.T
SAS Connection failed. No connection established. Double check you settings in sascfg.py file.

Attempted to run program /usr/bin/ssh with the following parameters:['/usr/bin/ssh', '-t', 'sashost.domain.com', '/opt/sas', '-WORK', '/opt/Default_saswork_v9', '-fullstimer', '-nofmterr', '-memsize', '3072M', '-nodms', '-stdio', '-terminal', '-nosyntaxcheck', '-pagesize', 'MAX', '']

Try running the following command (where saspy is running) manually to see if you can get more information on what went wrong:
/usr/bin/ssh -t sashost.domain.com /opt/sas -WORK /opt/Default_saswork_v9 -fullstimer -nofmterr -memsize 3072M -nodms -stdio -terminal -nosyntaxcheck -pagesize MAX  

No SAS process attached. SAS process has terminated unexpectedly.
No SAS process attached. SAS process has terminated unexpectedly.
No SAS process attached. SAS process has terminated unexpectedly.
Access Method         = SSH
SAS Config name       = ssh
WORK Path             = No SAS process attached. SAS process has terminated unexpectedly./
SAS Version           = No SAS process attached. SAS process has terminated unexpectedly.
SASPy Version         = 2.2.9
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = No
Python Encoding value = latin1
tomweber-sas commented 6 years ago

Ok, so that seems like the connection to SAS isn't being established. Can you run the command from the command line and see what it does?

Try running the following command (where saspy is running) manually to see if you can get more information on what went wrong:
/usr/bin/ssh -t sashost.domain.com /opt/sas -WORK /opt/Default_saswork_v9 -fullstimer -nofmterr -memsize 3072M -nodms -stdio -terminal -nosyntaxcheck -pagesize MAX  

If there's something happening, like it's prompting you to verify the host (which happens w. ssh the first time you try to connect to a host, even with passwordless), Or there's some other issue with SAS coming up or something you would see that running the command directly but not necessarily via saspy.

To try this I spun up a second VM, to ssh to, and I saw a similar problem as this when I tried to connect. So I ran the command, as suggested, and it turned out it was prompting me for some other stuff (unrelated), but some prompts that only happen the first time I connect after it boots up. I didn't see that in saspy, but saw it when I ran this myself from the command line. That's why I write out that suggestion in the error messages :)

After I dealt w/ that first time prompt, saspy then connected as usual and there were no more issues. And that's then where I ran those two notebooks.

Let's see what happens when you run that from the command line.

Thanks, Tom

bikashdube commented 6 years ago

I get the following output when running the command as hubuser or hub(jupyterhub owner)

Unauthorized access to or use of this system is prohibited.
All access and use may be monitored and recorded.

1                                                          The SAS System                            13:15 Tuesday, October 16, 2018

NOTE: Copyright (c) 2002-2012 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software 9.4 (TS1M2)
      Licensed to xx
NOTE: This session is executing on the Linux 2.6.32-696.18.7.el6.x86_64 (LIN X64) platform.

NOTE: Updated analytical products:

      SAS/STAT 13.2

NOTE: Additional host information:

 Linux LIN X64 2.6.32-696.18.7.el6.x86_64 #1 SMP Thu Dec 28 20:15:47 EST 2017 x86_64 Red Hat Enterprise Linux Server release 6.9
      (Santiago)

You are running SAS 9. Some SAS 8 files will be automatically converted
by the V9 engine; others are incompatible.  Please see
http://support.sas.com/rnd/migration/planning/platform/64bit.html

PROC MIGRATE will preserve current SAS file attributes and is
recommended for converting all your SAS libraries from any
SAS 8 release to SAS 9.  For details and examples, please see
http://support.sas.com/rnd/migration/index.html

This message is contained in the SAS news file, and is presented upon
initialization.  Edit the file "news" in the "misc/base" directory to
display site-specific news and information in the program log.
The command line option "-nonews" will prevent this display.

NOTE: SAS initialization used:
      real time           0.04 seconds
      user cpu time       0.01 seconds
      system cpu time     0.01 seconds
      memory              3566.59k
      OS Memory           4504.00k
      Timestamp           10/16/2018 01:15:09 PM
      Step Count                        0  Switch Count  6
      Page Faults                       0
      Page Reclaims                     4922
      Page Swaps                        0
      Voluntary Context Switches        102
      Involuntary Context Switches      7
      Block Input Operations            88
      Block Output Operations           88

I do remember those debugging suggestions in the troubleshooting wiki and I had found them to be useful when this was being done the first time around on this host.

tomweber-sas commented 6 years ago

Is that the user that saspy is running as? That connected. Can you run the same python as the notbook is using from the command line and try to connect via saspy that way? Is there anything in the saslog() when you print it after terminating?

bikashdube commented 6 years ago

Great idea. I think last year you had me try the terminal also for another similar issue. So it looks like things are working fine in a terminal.

Python 3.6.3 (default, Oct  8 2017, 11:48:17)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import saspy
>>> sas = saspy.SASsession()
Using SAS Config named: ssh
SAS Connection established. Subprocess id is 24189

>>> sas
Access Method         = SSH
SAS Config name       = ssh
WORK Path             = /opt/Default_saswork_v9/SAS_work44C30012EB6B_sashost/
SAS Version           = 9.04.01M2P07232014
SASPy Version         = 2.2.9
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = LATIN1
Python Encoding value = latin1

And here is the saslog.


>>> print(sas.saslog())
Pseudo-terminal will not be allocated because stdin is not a terminal.

Unauthorized access to or use of this system is prohibited.
All access and use may be monitored and recorded.

1                                                          The SAS System                            13:45 Tuesday, October 16, 2018

NOTE: Copyright (c) 2002-2012 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software 9.4 (TS1M2)
      Licensed to xxxxxxxxxx
NOTE: This session is executing on the Linux 2.6.32-696.18.7.el6.x86_64 (LIN X64) platform.

NOTE: Updated analytical products:

      SAS/STAT 13.2

NOTE: Additional host information:

 Linux LIN X64 2.6.32-696.18.7.el6.x86_64 #1 SMP Thu Dec 28 20:15:47 EST 2017 x86_64 Red Hat Enterprise Linux Server release 6.9
      (Santiago)

You are running SAS 9. Some SAS 8 files will be automatically converted
by the V9 engine; others are incompatible.  Please see
http://support.sas.com/rnd/migration/planning/platform/64bit.html

PROC MIGRATE will preserve current SAS file attributes and is
recommended for converting all your SAS libraries from any
SAS 8 release to SAS 9.  For details and examples, please see
http://support.sas.com/rnd/migration/index.html

This message is contained in the SAS news file, and is presented upon
initialization.  Edit the file "news" in the "misc/base" directory to
display site-specific news and information in the program log.
The command line option "-nonews" will prevent this display.

NOTE: SAS initialization used:
      real time           0.02 seconds
      user cpu time       0.01 seconds
      system cpu time     0.01 seconds
      memory              3609.37k
      OS Memory           4504.00k
      Timestamp           10/16/2018 01:45:14 PM
      Step Count                        0  Switch Count  6
      Page Faults                       0
      Page Reclaims                     4941
      Page Swaps                        0
      Voluntary Context Switches        87
      Involuntary Context Switches      4
      Block Input Operations            160
      Block Output Operations           120

1    ;*';*";*/;
2    options svgtitle='svgtitle'; options validvarname=any; ods graphics on;
3
4    ;*';*";*/;
5    %put E3969440A681A2408885998500000001;
E3969440A681A2408885998500000001
6    ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
6  ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
7    ;*';*";*/;
8    %put WORKpath=%sysfunc(pathname(work));
WORKpath=/opt/Default_saswork_v9/SAS_work44C30012EB6B_sashost
9
10   ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;

11
12   %put E3969440A681A2408885998500000002;
E3969440A681A2408885998500000002
13   ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
13 ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
14   ;*';*";*/;
15   %put SYSV=&sysvlong4;
SYSV=9.04.01M2P07232014
16
17   ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;

18
19   %put E3969440A681A2408885998500000003;
E3969440A681A2408885998500000003
20   ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
20 ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
21   ;*';*";*/;
22   proc options option=encoding;run;
    SAS (r) Proprietary Software Release 9.4  TS1M2

 ENCODING=LATIN1   Specifies the default character-set encoding for the SAS session.
NOTE: PROCEDURE OPTIONS used (Total process time):
      real time           0.00 seconds
      user cpu time       0.00 seconds
      system cpu time     0.00 seconds
      memory              19.46k
      OS Memory           9872.00k
      Timestamp           10/16/2018 01:45:15 PM
      Step Count                        1  Switch Count  0
      Page Faults                       0
      Page Reclaims                     14
      Page Swaps                        0
      Voluntary Context Switches        2
      Involuntary Context Switches      0
      Block Input Operations            632
      Block Output Operations           0

23
24   ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;

25
26   %put E3969440A681A2408885998500000004;
E3969440A681A2408885998500000004
>>>

All of this I ran as the user that is running jupyterhub (a user called just hub)

The hubuser ID (a sudo spawned notebook for a different ID called hubuser doesn't have the ability to run a terminal directly as it is not configured with python. I will try to run a terminal within a notebook to see if it works that way.

bikashdube commented 6 years ago

Yes so when I start a terminal from a notebook with the userid hubuser to get access to Python 3.6.3, I see that the commands get stuck, just like it would in a cell inside a notebook.

Python 3.6.3 (default, Oct  8 2017, 11:48:17)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import saspy
>>> sas = saspy.SASsession()
Using SAS Config named: ssh
tomweber-sas commented 6 years ago

ok, I see, looking back in this thread, you had run from the command line; only saw stimer output so that didn't jump out at me. Also, you say this all worked before, but you updated all of the various modules. Many jupyter modules, saspy, sas_kernel, and others? So, that's where I'm curious about running saspy outside of jupyter. Just firing up python and submitting saspy:

# you use the python that jupyter is running, I'm showing w/ python I have, 3.5
> /home/hub/.pyenv/versions/3.6.3/bin/python3.6

> python3.5
Python 3.5.5 (default, Feb  6 2018, 10:56:47)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import saspy
>>> sas = saspy.SASsession(cfgname='issue176')
SAS Connection established. Subprocess id is 27360

>>> sas
Access Method         = SSH
SAS Config name       = issue176
WORK Path             = /sastmp/SAS_work218700000F1D_tom64-2/
SAS Version           = 9.04.01M2D07232014
SASPy Version         = 2.2.9
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = LATIN1
Python Encoding value = latin1

>>> ll = sas.submit('''
... ods _all_ close;
... proc setinit;
... run;
... ''')
>>> print(ll['LOG'][:1000])

27   ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
27 ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
28
29
30   ods _all_ close;
1                                                          The SAS System                            13:50 Tuesday, October 16, 2018

NOTE: html5 Memory Usage (267 events)
      Total: 2.62MB     High: 1332.89KB     Max: 32768B     Allocs: 15917   Frees: 15917
31   proc setinit;
32   run;
Original site validation data
Site name:    'SAS Institute Inc.'.
Site number:  1.
Expiration:   22NOV2018.
Grace Period:  62 days (ending 23JAN2019).
Warning Period: 31 days (ending 23FEB2019).
System birthday:   01JAN1997.
Operating System:   <ANY>   .
Product expiration dates:
---Base SAS Software                                                                                    22NOV2018
---SAS/STAT
>>>

I'm not seeing anything so far from the saspy side of the fence that's not working or looks problematic to me. I don't want to say that this is some problem due to updating random modules to various versions which may not be compatible. BUT, I happened to try to update various jupyter modules to newer versions once and couldn't get things to work and had to go back to a clean working original install before. So, I'm suspicious of that, but we need to actually prove something, not just speculate. So, if you can run saspy ok from python w/out jupyter in the mix, then there may be something to look at regarding that.

Thanks! Tom

tomweber-sas commented 6 years ago

Ah, you were ahead of me and already did this! :)

Ok, so now I am going to speculate that it's something with perhaps mismatched (don't happen to be compatible) jupyter modules. I didn't have luck the one time I tried updating various jupyter modules. Again, that doesn't prove anything, but w/ jupyter out of the mix, and you running this only from python it works. Trying from jupyter, even in jupyter terminal, it doesn't. Is there any way you can get back to an environment where these are all compatible versions of the modules? And I don't know really how to tell. I suspect you need a clean install of jupyter that has everything matched and compatible. Clearly the versions of the various modules don't all match each other, so I couldn't tell which ones work with which ones.

jupyter                         1.0.0
jupyter-client                  5.2.3
jupyter-console                 6.0.0
jupyter-contrib-core            0.3.3
jupyter-core                    4.4.0
jupyter-highlight-selected-word 0.2.0
jupyter-latex-envs              1.4.4
jupyterhub                      0.9.4
jupyterlab                      0.35.1
jupyterlab-launcher             0.13.1
jupyterlab-server               0.2.0

Tom

bikashdube commented 6 years ago

Yes this was a good set of debugging data points. I think something is broken during the upgrade and I don't know which module may be the cause here. I will keep trying some more. I was trying to also upgrade jupyterlab-hubextension and that one keeps failing.

Anyways with the hub ID (this is kind of the super ID of JupyterHub install) , I am able to run saspy in a native terminal. It is a domain ID and I am unable to login to Jupyter notebook with this ID as it requires 2 steps to login in a shell.

With the hubuser ID, I cannot run saspy in any terminal (via notebook). Natively this ID has only Python 2.7 available. This ID can login to Jupyter notebook spawned via the hub process.

It is a strange pickle :)

tomweber-sas commented 6 years ago

Yes, and I just don't know that much about Jupyter itself, myself, to give you much advice about how to upgrade it. Obviously, I'm glad that saspy is working for you, other then with the upgraded jupyter and with the issue of which user id you can use. I hate that I can't really help with either of those though. I'd like to have it all just work for you.

bikashdube commented 6 years ago

No worries Tom. I appreciate the help. Pretty sure I can get this working and I will let you know how it goes. I am happy to have a solution as of now. Will try to run a single user notebook for additional debugging (skipping JupyterHub).

Thanks again. Bikash

bikashdube commented 6 years ago

Well I have figured out the issue and luckily nothing was smashed on the desk when I found out what is going on.

Added -vvv to the ssh option in sascfg_personal.py to see if the terminal session(python 3.6.3) inside the notebook will print more verbose connectivity information for hubuser. Unfortunately it just prints the same information at first

>> import saspy
>>> sas = saspy.SASsession()
Using SAS Config named: ssh
SAS Connection failed. No connection established. Double check you settings in sascfg.py file.

Attempted to run program /usr/bin/ssh -vvv with the following parameters:['/usr/bin/ssh -vvv', '-t', 'sashost.domain.com', '/opt/sas', '-WORK', '/opt/Default_saswork_v9', '-fullstimer', '-nofmterr', '-memsize', '3072M', '-nodms', '-stdio', '-terminal', '-nosyntaxcheck', '-pagesize', 'MAX', '']

Try running the following command (where saspy is running) manually to see if you can get more information on what went wrong:
/usr/bin/ssh -vvv -t sashost.domain.com /opt/sas -WORK /opt/Default_saswork_v9 -fullstimer -nofmterr -memsize 3072M -nodms -stdio -terminal -nosyntaxcheck -pagesize MAX

No SAS process attached. SAS process has terminated unexpectedly.
No SAS process attached. SAS process has terminated unexpectedly.
No SAS process attached. SAS process has terminated unexpectedly.

And how was I checking this yesterday? By running this in a shell/terminal, but outside the notebook (despite your clear message to run this where "saspy is running", meaning I should have run the manual test within the notebook terminal). Outside the notebook in a normal shell that I login via Putty, it was succeeding fine because I have passwordless ssh working for ages for the hubuser to the SAS host from the Jupyter VM. In fact I have passwordless logins to 10s of hosts all setup ages ago via PuttyGen/Pageant.

But this morning when I started back again, I realized I should do the check from within the notebook terminal. And here is what I saw (relevant portion out of the very verbose output)

debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/hubuser/.ssh/id_rsa
debug3: no such identity: /home/hubuser/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/hubuser/.ssh/id_dsa
debug3: no such identity: /home/hubuser/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/hubuser/.ssh/id_ecdsa
debug3: no such identity: /home/hubuser/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/hubuser/.ssh/id_ed25519
debug3: no such identity: /home/hubuser/.ssh/id_ed25519: No such file or directory

Well of course it doesn't exist. I use the Pageant generated/converted rsh-ssa keys which has worked fine for me for years in logins to from windows/linux and other combinations as long as I had Pageant running locally. But the notebook terminal was specifically looking for the ssh-keygen generated keys so the pageant forwarding was not coming into effect.

So I setup new ssh-keygen keys and now BOB = UNCLE :)

bikashdube commented 6 years ago

@tomweber-sas

What is source of the empty element at the end in this list?

Attempted to run program /usr/bin/ssh -vvv with the following parameters:['/usr/bin/ssh -vvv', '-t', 'sashost.domain.com', '/opt/sas', '-WORK', '/opt/Default_saswork_v9', '-fullstimer', '-nofmterr', '-memsize', '3072M', '-nodms', '-stdio', '-terminal', '-nosyntaxcheck', '-pagesize', 'MAX', '']

Just also a suggestion to add this to the current verbose output on failure although anyone who has started developing SAS programs using saspy knows this very well by now.

"Double check you settings in sascfg.py file. or sascfg_personal.py"

Also based on the experience in this one, I think we should make the potential future users aware that passwordless logins should be setup using ssh-keygen only.

tomweber-sas commented 6 years ago

@bikashdube great digging! Actually, I was asking you to run it completely outside of jupyter (notebook) to take that out of the picture. But, both ways needed to be done to compare, so that's perfect. Yes, I think that's a great idea to add the -vvv in there, I'll do that. Oh, the trailing ,''] is just required for the fork/exec of python so that the parms for exec are built correctly. That's the argv/argc of a main() so that the argv list is null terminated, per convention.

I just happened, yesterday, to provide a coworker with a new option for STDIO over SSH to supply an identity file, as her own credentials weren't set up on the system she was connecting to, but had an identity file (.pem file) and needed for saspy to supply the '-i creds.pem' on the command line. I'm going to add that to master also when I get a chance. I'm not familiar with Pageant, but just googled it, so I see it's tied in w/ putty, so jupyter wasn't seeing/using those keys. I wonder if the -i would work with the .ppk files Pagent seems to create?

I'll see about getting both the -vvv and the -i support in there when I have a chance. Good job figuring this the rest of the way out! Tom

bikashdube commented 6 years ago

I can test the -i switch with the putty private key once you have it in master or any other branch, please let me know. Also I have not had to look at all this in a long time, but it seems that it is possible to also create the OpenSSH format private/public keys from the PuttyGen created private ppk/public keys and vice versa Discussion on Stack Overflow. I prefer not to do this however as I like my private key to be available only in my personal workstation and not float around in unix filesystems.

tomweber-sas commented 6 years ago

Ok, I just pushed that to master, Just add a line like the following to your config definition

'identity'  : '/fully/qualified/path.pem',   # or in your case .ppk
bikashdube commented 6 years ago

@tomweber-sas

Updated to master and gave it a shot. It turns out that ppk files have to be converted into pem format on *nix as ppk is compatible only with Pageant running on windows. ssh-add/keychain don't understand the format. Some more discussion here

tomweber-sas commented 6 years ago

Trying to catch back up. Seems like this is fixed for you. You did have to have keys in the format ssh needed them in, And you were able to convert from the Pageant generated format of keys from the .ppk to the .pem version that's supported by the -i on ssh as well. Is this all correct? Is there anything else on this issue I can do? trying to clean up issues today.

Thanks! Tom

bikashdube commented 6 years ago

Yes the main issue was that I did not have the SSH keys in what is called the OpenSSH format. The debugging step that helped to identify that was adding the -vvv option while connecting. We can close this issue with perhaps a takeaway to add -vvv in either documentation or actual implementation of debugging params.

Thank you very much for the support.

Thanks Bikash

tomweber-sas commented 6 years ago

Cool, thanks. Yes, I need to update the doc for the new identity key, and there's already a section in the troubleshooting guide for STDIO over SSH. I'll add in info on the keys and using -vvv (yes, lots of output), to help diagnose problems with this.

Thanks a lot! Tom

tomweber-sas commented 6 years ago

I've updated both the configuration doc for STDIO over SSH and the section in the troubleshooting guide with identity and -vvv diagnostics. Thanks for your help on this. I'll close this now. Let me know if you have any other issues!

Thanks, Tom