Cannot exchange data with SAS

[BoPeng]

I am running into issues while trying to exchange data between the sos and the sas kernel on a juoyterlab notebook. I create a panda dataframe on sos and when I try to pass and access that dataframe on the sas kernel with : %get df , I get the exception :

“Magic %get failed because the language module for SAS is not properly installed . Please install it according to language specific instructions on the
Running SoS section of the SoS homepage and restart Jupiter server.”

I have already verified : all the modules on my sos installation have been done properly , all the packages I need are installed included sos-sas

Do you have an idea what might be happening ? I heard that sas has itself a %get command and I am suspecting that the sos and sas exchange might not properly happen because of it . Is there a workaround that you know of ?

[BoPeng]

Let us check things one by one

1. Can you use SAS in a notebook with SAS kernel? If no, please check if you have saspy etc installed and properly configured.
2. Can you use sos with other kernels? You can try the Python3 kernel and check if sos notebook is properly installed.
3. Do you have multiple conda environment so the kernels might be using different interpreter than you thought. Check the output of jupyter kernelspec list --json to see if everything looks normal.

If all these are yes, please let me know the OS and version of sos, sos-notebook, sos-sas, SAS.... of course it would be helpful for you to do conda update sos sos-notebook jupyterlab-sos to update everything.

[ambamanyloic]

Yes for all 3 points : I can exchange data fine from sos to R , but when it comes to sos to sas there appears to be a disconnect. On the "jupyter kernelspec list" everything looks normal , I have quite some few kernels installed - maybe I should remove some of them ? The versions OS : RedHat 7.8 sos : 0.21.13 sos-notebook : 0.21.13 sos-sas : 0.19.0 sas-kernel : 2.2.0 saspy : 3.3.5

The conda update did not help but note that I installed sos-sas using pip since it was not available in my conda channel

Thanks

[ambamanyloic]

I omitted earlier to include the version for jupyterlab-sos : 0.6.8 and jupyter_kernel_gateway : 2.0.0

[ambamanyloic]

jupyter_kernel_gateway : 2.4.0 :)

[BoPeng]

SAS 9.4 or later?

[BoPeng]

Cannot see anything wrong. Could you run

export SOS_DEBUG=ALL

before running jupyter lab and send me (PM if needed) ~/.sos/sos_debug.log?

[loicskipper]

Yes I am using sas 9.4

[loicskipper]

This is what I get below : it looks like an issue with the sos-sas connection details : I just started the jupyterlab notebook , i have not even tried to connect to sas from sos and I am seeing the exception below

2020-08-29 13:52:37,693 - DEBUG - Using selector: EpollSelector

2020-08-29 13:52:37,697 - DEBUG - Using selector: EpollSelector

2020-08-29 13:52:37,811 - DEBUG - KERNEL - Starting SoS Kernel version 0.21.13 with SoS 0.21.13

2020-08-29 13:52:37,814 - DEBUG - VARIABLE - Set SOS_VERSION to '0.21.13' of type str

2020-08-29 13:52:37,814 - DEBUG - VARIABLE - Set master_id to '' of type str

2020-08-29 13:52:37,814 - DEBUG - VARIABLE - Set workflow_id to '' of type str

2020-08-29 13:52:37,814 - DEBUG - VARIABLE - Set step_name to '' of type str

2020-08-29 13:52:37,814 - DEBUG - VARIABLE - Set step_id to '' of type str

2020-08-29 13:52:37,815 - DEBUG - VARIABLE - Set CONFIG to "'user_name'":"'XXXXXX'" of type dict

2020-08-29 13:52:37,930 - DEBUG - VARIABLE - Set __host__ to 'localhost' of type str

2020-08-29 13:52:37,939 - DEBUG - VARIABLE - Set __host__ to 'localhost' of type str

2020-08-29 13:52:37,939 - DEBUG - VARIABLE - Set __host__ to 'localhost' of type str

2020-08-29 13:52:37,959 - DEBUG - VARIABLE - Set __host__ to 'localhost' of type str

2020-08-29 13:52:38,176 - DEBUG - VARIABLE - Set __host__ to 'localhost' of type str

2020-08-29 13:52:38,718 - DEBUG - KERNEL - Found registered language SAS

2020-08-29 13:52:38,945 - DEBUG - KERNEL - Failed to load registered language SAS: EOF when reading a line

2020-08-29 13:52:38,945 - DEBUG - KERNEL - Found registered language R

2020-08-29 13:52:38,964 - DEBUG - KERNEL - Found registered language Python2

2020-08-29 13:52:38,977 - DEBUG - KERNEL - Found registered language Python3

2020-08-29 13:52:38,982 - DEBUG - KERNEL - Found registered language Bash

2020-08-29 13:52:38,995 - DEBUG - KERNEL - Found registered language Zsh

2020-08-29 13:52:39,024 - WARNING - Failed to locate subkernel SAS with kernerl "sas" and language "sas": EOF when reading a line

2020-08-29 13:52:39,238 - DEBUG - KERNEL - Found registered language SAS

2020-08-29 13:52:39,465 - DEBUG - KERNEL - Failed to load registered language SAS: EOF when reading a line

2020-08-29 13:52:39,465 - DEBUG - KERNEL - Found registered language R

2020-08-29 13:52:39,481 - DEBUG - KERNEL - Found registered language Python2

2020-08-29 13:52:39,493 - DEBUG - KERNEL - Found registered language Python3

2020-08-29 13:52:39,498 - DEBUG - KERNEL - Found registered language Bash

2020-08-29 13:52:39,510 - DEBUG - KERNEL - Found registered language Zsh

2020-08-29 13:52:39,537 - WARNING - Failed to locate subkernel SAS with kernerl "sas" and language "sas": EOF when reading a line

2020-08-29 13:52:39,756 - DEBUG - KERNEL - Found registered language SAS

2020-08-29 13:52:39,965 - DEBUG - KERNEL - Failed to load registered language SAS: EOF when reading a line

2020-08-29 13:52:39,965 - DEBUG - KERNEL - Found registered language R

2020-08-29 13:52:39,983 - DEBUG - KERNEL - Found registered language Python2

2020-08-29 13:52:39,994 - DEBUG - KERNEL - Found registered language Python3

2020-08-29 13:52:39,999 - DEBUG - KERNEL - Found registered language Bash

2020-08-29 13:52:40,012 - DEBUG - KERNEL - Found registered language Zsh

2020-08-29 13:52:40,041 - WARNING - Failed to locate subkernel SAS with kernerl "sas" and language "sas": EOF when reading a line

2020-08-29 13:52:41,179 - DEBUG - KERNEL - Found registered language SAS

2020-08-29 13:52:41,397 - DEBUG - KERNEL - Failed to load registered language SAS: EOF when reading a line

2020-08-29 13:52:41,397 - DEBUG - KERNEL - Found registered language R

2020-08-29 13:52:41,415 - DEBUG - KERNEL - Found registered language Python2

2020-08-29 13:52:41,426 - DEBUG - KERNEL - Found registered language Python3

2020-08-29 13:52:41,431 - DEBUG - KERNEL - Found registered language Bash

2020-08-29 13:52:41,443 - DEBUG - KERNEL - Found registered language Zsh

2020-08-29 13:52:41,470 - WARNING - Failed to locate subkernel SAS with kernerl "sas" and language "sas": EOF when reading a line

2020-08-29 13:52:42,238 - DEBUG - KERNEL - Found registered language SAS

2020-08-29 13:52:42,472 - DEBUG - KERNEL - Failed to load registered language SAS: EOF when reading a line

2020-08-29 13:52:42,472 - DEBUG - KERNEL - Found registered language R

2020-08-29 13:52:42,486 - DEBUG - KERNEL - Found registered language Python2

2020-08-29 13:52:42,497 - DEBUG - KERNEL - Found registered language Python3

2020-08-29 13:52:42,502 - DEBUG - KERNEL - Found registered language Bash

2020-08-29 13:52:42,513 - DEBUG - KERNEL - Found registered language Zsh

[loicskipper]

It looks like even before I am trying to go from sos to sas , the sas kernel is not loading up the right way

[BoPeng]

SoS Notebook failed to locate sas, the exception is at

https://github.com/vatlab/sos-notebook/blob/e8e0743aa217e288155f44ddbdef0f03c43629e3/src/sos_notebook/kernel.py#L550

The error message

 EOF when reading a line

is not from sos notebook and I have to check where things could go wrong.

I can have a deeper look later today, but if you can, maybe you could edit lib/python3.7/site-packages/sos_notebook/kernel.py directly, add a few env.log_to_file('KERNEL', 'here') inside

https://github.com/vatlab/sos-notebook/blob/e8e0743aa217e288155f44ddbdef0f03c43629e3/src/sos_notebook/kernel.py#L271

to pinpoint it.

[ambamanyloic]

Ok , in L550 I see the exception being thrown "f'Failed to locate subkernel {kinfo[0]} with kernerl "{kinfo[1]}" and language "{kinfo[2]}": {e}' " but I dont see your calls to the sas kernel module itself , indicate if you can where it is , it looks like there is something extra there

[BoPeng]

Yes, line 550 is where the exception is, which is throw from the find function defined in line 271. In your case kernel is not None, language is also not None, we are looking at line 351, and I fail to see how EOF is triggered anywhere here. This was why I suggested that you add some env.log_to_file around that area to see where exactly the exception was triggered (if you know enough Python).

We have an identical system here (RHEL 7.8, JupyterLab, SAS etc) so it looks like that there is something different in our configurations.

[ambamanyloic]

Thank you , let me add the env.log_to_file

[BoPeng]

Thanks for your help. I could also help if your server is open for remote login.

[ambamanyloic]

Unfortunately I can not do the screen share , yes that is weird , I tried with the logs and they re not showing much ....can you try to replicate this by connecting to a sas server from your sos kernel on jupyterlab ? For some reason everything keeps on going back to that : "Failed to locate subkernel SAS with kernerl "sas" and language "sas": EOF when reading a line" ....

[BoPeng]

As I have said we have an almost identical set up here (jupyterhub + jupyterlab). SAS on a remote server connected through SSH. .. I am using a little newer sos-notebook (I updated conda-forge yesterday and you could try to update) but it should not matter since our sos-sas was set up before 0.21.13....

Are you running headless through jupyter_kernel_gateway? This could be a problem.

[ambamanyloic]

Ok , I will keep on looking . Thanks

[BoPeng]

Did you use an old sos notebook with SAS kernel used before sos-sas was installed? If so could you start a new sos notebook and try? I remember something remotely similar when that happens.

[BoPeng]

I think we barked at a wrong tree.

2020-08-29 13:52:38,945 - DEBUG - KERNEL - Failed to load registered language SAS: EOF when reading a line

is the message happened before the one we were looking:

https://github.com/vatlab/sos-notebook/blob/ff190379be39e4e8967063e9f5abb6387d5053f1/src/sos_notebook/kernel.py#L634-L644

It calls

entrypoint.load()

which indeed can cause something like EOF failed.

This seems to indicate that the entry point is broken. It never happened to me before but perhaps you can reinstall sos-sas?

[BoPeng]

Please create a small python file with content

import pkg_resources

for entrypoint in pkg_resources.iter_entry_points(group='sos_languages'):
    name = entrypoint.name
    print(f'found {name}')
    if name != 'SAS':
        continue
    print('loading SAS entry point')
    entrypoint.load()

and see if an exception would be triggered.

[ambamanyloic]

Ok I will try the reinstall and the script , I will keep you posted , thanks !

[ambamanyloic]

Ok I am reinstalling sos-sas now but it looks like so far your script can find sas , but it also tries to load the sas configs right away : python entry_point.py:

found SAS loading SAS entry point Please enter the name of the SAS Config you wish to run. Available Configs are: ['.....'] found R found Python2 found Python3 found Bash found Zsh

[BoPeng]

But your /path/to/lib/python3.7/site-packages/saspy/sascfg_personal.py should be accessible, right?

[ambamanyloic]

This is what is being returned below , I am wondering if the EOF is not being thrown at the end of the ['sasuat', 'sasprod']

found SAS loading SAS entry point Please enter the name of the SAS Config you wish to run. Available Configs are: ['sasuat', 'sasprod'] found R found Python2 found Python3 found Bash found Zsh

[BoPeng]

In that file, you should have something like

SAS_config_names=['ssh']

which points to a single configuration.

[ambamanyloic]

I give the option to go to either uat or prod SAS_config_names=['sasuat','sasprod']

[BoPeng]

Just for testing, could you limit to one configuration?

[BoPeng]

I think what happened was that when the entry point was loaded, it asked users to choose one configuration, and EOF error was reported due to no input device to answer the question.

[ambamanyloic]

interesting ..... ok I am still reinstalling sos-sas now , then I will try the single configuration

[BoPeng]

Then, after you confirm the problem, could you

1. restore to two options
2. find site-packages/sos_sas/kernel.py

Modify line 26 from

sas_config = SASconfig()

to

sas_config = SASconfig(prompt=False)

and let me know if this fixes the problem?

This change should fix the EOF error. However, I do not know what configuration will be used by sos-sas if you provide two options.

[ambamanyloic]

Yes still not good when going back to the 2 configs and adding the "sas_config = SASconfig(prompt=False)" : I am still getting the EOF in the logs : "Failed to locate subkernel SAS with kernerl "sas" and language "sas": EOF when reading a line"

By the way , when only using the one config either prod or uat , the sos logs show : "WARNING - Overriding language SAS used by subkernel SAS with language sas "

[ambamanyloic]

In either cases , the "Magic %get failed because ...." exception which is my original issue shows up ... it looks like the %get can not get translated still

[BoPeng]

The overriding thing should not happen if you start a new notebook.

[BoPeng]

Yes still not good when going back to the 2 configs and adding the "sas_config = SASconfig(prompt=False)" : I am still getting the EOF in the logs : "Failed to locate subkernel SAS with kernerl "sas" and language "sas": EOF when reading a line"

At least this is something I can try to reproduce. I will let you know.

[ambamanyloic]

actually my mistake , the single config works when I start a new notebook and the Magic %get gets picked up , I need to keep on testing so we can assert that the only issue is with the two configs , at least it can get me rolling on the single config for now , thanks a bunch I will keep you posted

[BoPeng]

How did you get two config working with a SAS notebook?

[ambamanyloic]

With the SAS notebook everything is going smoothly , I can call either uat or prod without any issues but I have to stay within the sas domain all the time , when in sos-sas I "can" exchange data between kernels. In sos-sas now it looks like it is working for me when I specify only one config env , I might have to create two different separate config files for uat and prod for now if there is no resolution for the dual SAS_config_names=['sasuat','sasprod']

[BoPeng]

I was wondering how you specify and switch between two configurations for notebooks with an SAS kernel.

[ambamanyloic]

Actually I dont really switch between configurations in the same session , I really wanted to have the option to go to uat or prod once within the sas notebook

[BoPeng]

I understand that part. My question is, when you use a normal Jupyter notebook with a SAS kernel, how did you specify which configuration to use. Did you make a selection when the notebook is created?

[BoPeng]

Figured out

[ambamanyloic]

Yes : when in the SAS kernel from a jupyter notebook , I get prompted to choose one of the two configs , then I can choose either one

[BoPeng]

OK, I have fixed the bug and released sos-sas 0.19.1. Please update it with pip install sos-sas --upgrade and try again.

You should be prompt for the use of configuration when you use SAS inside sos notebooks, however, this will make your notebook not papermill friendly, namely you cannot batch execute your notebook with sos-papermill.

[ambamanyloic]

Ok thanks a bunch , I ll run the upgrade tomorrow and let you know how it goes

[BoPeng]

Cool. Thanks for using sos and reporting the bug. Let me know if you notice any more problem.