search

sassoftware / saspy

A Python interface module to the SAS System. It works with Linux, Windows, and Mainframe SAS as well as with SAS in Viya.
https://sassoftware.github.io/saspy
Other
366 stars 149 forks source link

Get lineTypes as LanguageService.FlushLogLines or getJobLog for rest api #603

Closed ShuguangSun closed 2 weeks ago

ShuguangSun commented 1 month ago

Is your feature request related to a problem? Please describe. The line types (normal, source, error, warning, etc) can help identify issue location in the original code and can help throw usefull error message.

Describe the solution you'd like Return an array with item like 

    {
      "line": " 1    %put %sysget(SASROOT);",
      "type": "source"
    },
tomweber-sas commented 1 month ago

Ok, I now understand what you're asking; took a little research. It's gonna take a little more to see about a prototype, but I'm thinking I may be able to do this. The HTTP Access Method would be the simplest, as it looks straight forward. The IOM AM will take more to see, but it seems doable; gotta try and see (have to rework both Java and python sides of that to even see what I can get. The STDIO/SSH Access Methods have no API so this isn't something that could be supported from that Access Method.

My current thought would be adding an option on the submit method to ask for this to be returned instead of the 'normal' LOG that's returned. Would that be sufficient if this all works out?

Tom

ShuguangSun commented 1 month ago

Yes. Great!

tomweber-sas commented 1 month ago

So, do you need this in IOM? Or just HTTP? I have this working for HTTP already, as this is what's returned from the server, so it' s pretty easy. For IOM, it's gonna take a while to try to prototype and see if I can even get it to behave similarly. It looks doable, but it's gonna take a lot of work and code changes to prove it out. Just checking :)

I've prototyped HTTP and pushed it to a branch called loglines. Go give it a try and let me know what you think. I added a keyword loglines to submit() so you get back what you want if you se it to True. The default, of course, is False so things work as they have.

Here's a simple test showing how it works:

tom64-7> python3
Python 3.9.12 (main, Apr  5 2022, 06:56:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import saspy; sas = saspy.SASsession(cfgname='dotat'); sas
SAS server started using Context Data Mining compute context with SESSION_ID=c9a48d55-982c-4163-a0ef-747a840c5615-ses0000
Access Method         = HTTP
SAS Config name       = dotat
SAS Config file       = /opt/tom/github/saspy/saspy/sascfg_personal.py
WORK Path             = /opt/sas/viya/config/var/tmp/compsrv/default/c9a48d55-982c-4163-a0ef-747a840c5615/SAS_work452E000001E5_sas-compute-server-d4e302ec-a772-4421-85db-2ea9a6db5b24-600/
SAS Version           = V.04.00M0P05222024
SASPy Version         = 5.13.0
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = utf-8
Python Encoding value = utf_8
SAS process Pid value = 485

>>> x = sas.submit('data a; x=1; proc print data=NoExist; run;', results='text')
>>>
>>>
>>> x
{'LOG': '12   data a; x=1;\n\nNOTE: The data set WORK.A has 1 observations and 1 variables.\nNOTE: DATA statement used (Total process time):\n      real time           0.00 seconds\n      cpu time            0.00 seconds\n      \n\n12 !              proc print data=NoExist; run;\nERROR: File WORK.NOEXIST.DATA does not exist.\n\nNOTE: The SAS System stopped processing this step because of errors.\nNOTE: PROCEDURE PRINT used (Total process time):\n      real time           0.02 seconds\n      cpu time            0.03 seconds\n      \n\n', 'LST': ''}
>>>
>>>
>>> print(x['LOG'])
12   data a; x=1;

NOTE: The data set WORK.A has 1 observations and 1 variables.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds

12 !              proc print data=NoExist; run;
ERROR: File WORK.NOEXIST.DATA does not exist.

NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE PRINT used (Total process time):
      real time           0.02 seconds
      cpu time            0.03 seconds

>>> y = sas.submit('data a; x=1; proc print data=NoExist; run;', results='text', loglines=True)
>>>
>>>
>>> y
{'LOG': [{'line': '', 'type': 'note', 'version': 1}, {'line': '', 'type': 'note', 'version': 1}, {'line': '13   data a; x=1;', 'type': 'source', 'version': 1}, {'line': '', 'type': 'note', 'version': 1}, {'line': 'NOTE: The data set WORK.A has 1 observations and 1 variables.', 'type': 'note', 'version': 1}, {'line': 'NOTE: DATA statement used (Total process time):', 'type': 'note', 'version': 1}, {'line': '      real time           0.00 seconds', 'type': 'note', 'version': 1}, {'line': '      cpu time            0.01 seconds', 'type': 'note', 'version': 1}, {'line': '      ', 'type': 'note', 'version': 1}, {'line': '', 'type': 'note', 'version': 1}, {'line': '13 !              proc print data=NoExist; run;', 'type': 'source', 'version': 1}, {'line': 'ERROR: File WORK.NOEXIST.DATA does not exist.', 'type': 'error', 'version': 1}, {'line': '', 'type': 'note', 'version': 1}, {'line': 'NOTE: The SAS System stopped processing this step because of errors.', 'type': 'note', 'version': 1}, {'line': 'NOTE: PROCEDURE PRINT used (Total process time):', 'type': 'note', 'version': 1}, {'line': '      real time           0.00 seconds', 'type': 'note', 'version': 1}, {'line': '      cpu time            0.00 seconds', 'type': 'note', 'version': 1}, {'line': '      ', 'type': 'note', 'version': 1}, {'line': '', 'type': 'note', 'version': 1}], 'LST': ''}
>>>
>>>
>>> for line in y['LOG']:
...    print("TYPE=",line['type'], "LINE=",line['line'])
...
TYPE= note LINE=
TYPE= note LINE=
TYPE= source LINE= 13   data a; x=1;
TYPE= note LINE=
TYPE= note LINE= NOTE: The data set WORK.A has 1 observations and 1 variables.
TYPE= note LINE= NOTE: DATA statement used (Total process time):
TYPE= note LINE=       real time           0.00 seconds
TYPE= note LINE=       cpu time            0.01 seconds
TYPE= note LINE=
TYPE= note LINE=
TYPE= source LINE= 13 !              proc print data=NoExist; run;
TYPE= error LINE= ERROR: File WORK.NOEXIST.DATA does not exist.
TYPE= note LINE=
TYPE= note LINE= NOTE: The SAS System stopped processing this step because of errors.
TYPE= note LINE= NOTE: PROCEDURE PRINT used (Total process time):
TYPE= note LINE=       real time           0.00 seconds
TYPE= note LINE=       cpu time            0.00 seconds
TYPE= note LINE=
TYPE= note LINE=
>>>

Thanks, Tom

ShuguangSun commented 1 month ago

I need the IOM. It will be very appreciated!

tomweber-sas commented 1 month ago

Yes, ok, I'm not surprised. I have a prototype working for that today already. It seems to be working, but it needs more testing for sure. When I get it ready built (just been debugging it so it's not 'clean' yet), I'll push it out to the branch so you can try it. Hopefully before the end of the day today; I have a 4 day weekend.

tomweber-sas commented 1 month ago

OK, I've pushed what I have for IOM to the branch. So, if you can try it out before I get back next Wed., that'd be great. I still have more to do to vet all of this, especially the IOM as it's a lot of code changes. Also, the TYPE in IOM comes back as a number, so I have to try to find a mapping of those to the strings I get back in the HTTP am. For the moment, here's what I get with the same test case as above:

tom64-7> python3
Python 3.9.12 (main, Apr  5 2022, 06:56:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import saspy; sas = saspy.SASsession(cfgname='iomj'); sas
SAS Connection established. Subprocess id is 138188

Access Method         = IOM
SAS Config name       = iomj
SAS Config file       = /opt/tom/github/saspy/saspy/sascfg_personal.py
WORK Path             = /sastmp/SAS_workB79900015CA8_tom64-7/SAS_work21A700015CA8_tom64-7/
SAS Version           = 9.04.01M8P01182023
SASPy Version         = 5.13.0
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = utf-8
Python Encoding value = utf_8
SAS process Pid value = 89256

>>> x = sas.submit('data a; x=1; proc print data=NoExist; run;', results='text')
/opt/tom/github/saspy/saspy/sasioiom.py:1076: UserWarning: Noticed 'ERROR:' in LOG, you ought to take a look and see if there was a problem
  warnings.warn("Noticed 'ERROR:' in LOG, you ought to take a look and see if there was a problem")
>>> x
{'LOG': '\x147                                                          The SAS System                           Friday, May 24, 2024 03:21:00 PM\n\n34         \n35         data a; x=1;\n\nNOTE: The data set WORK.A has 1 observations and 1 variables.\nNOTE: DATA statement used (Total process time):\n      real time           0.10 seconds\n      cpu time            0.01 seconds\n      \n\n35       !              proc print data=NoExist; run;\nERROR: File WORK.NOEXIST.DATA does not exist.\n\nNOTE: The SAS System stopped processing this step because of errors.\nNOTE: PROCEDURE PRINT used (Total process time):\n      real time           0.00 seconds\n      cpu time            0.00 seconds\n      \n36         \n37         \n38         \n\x148                                                          The SAS System                           Friday, May 24, 2024 03:21:00 PM\n\n39         ', 'LST': ''}
>>> print(x['LOG'])
7                                                          The SAS System                           Friday, May 24, 2024 03:21:00 PM

34
35         data a; x=1;

NOTE: The data set WORK.A has 1 observations and 1 variables.
NOTE: DATA statement used (Total process time):
      real time           0.10 seconds
      cpu time            0.01 seconds

35       !              proc print data=NoExist; run;
ERROR: File WORK.NOEXIST.DATA does not exist.

NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE PRINT used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds

36
37
38
8                                                          The SAS System                           Friday, May 24, 2024 03:21:00 PM

39
>>>
>>>
>>>
>>>
>>> y = sas.submit('data a; x=1; proc print data=NoExist; run;', results='text', loglines=True)
>>> y
{'LOG': [{'line': '9                                                          The SAS System                           Friday, May 24, 2024 03:21:00 PM', 'type': '3'}, {'line': '', 'type': '3'}, {'line': '42         ', 'type': '2'}, {'line': '', 'type': '8'}, {'line': '', 'type': '8'}, {'line': '43         data a; x=1;', 'type': '2'}, {'line': '', 'type': '8'}, {'line': 'NOTE: The data set WORK.A has 1 observations and 1 variables.', 'type': '8'}, {'line': 'NOTE: DATA statement used (Total process time):', 'type': '8'}, {'line': '      real time           0.20 seconds', 'type': '8'}, {'line': '      cpu time            0.01 seconds', 'type': '8'}, {'line': '      ', 'type': '8'}, {'line': '', 'type': '8'}, {'line': '43       !              proc print data=NoExist; run;', 'type': '2'}, {'line': 'ERROR: File WORK.NOEXIST.DATA does not exist.', 'type': '6'}, {'line': '', 'type': '8'}, {'line': 'NOTE: The SAS System stopped processing this step because of errors.', 'type': '8'}, {'line': 'NOTE: PROCEDURE PRINT used (Total process time):', 'type': '8'}, {'line': '      real time           0.00 seconds', 'type': '8'}, {'line': '      cpu time            0.00 seconds', 'type': '8'}, {'line': '      ', 'type': '8'}, {'line': '44         ', 'type': '2'}, {'line': '45         ', 'type': '2'}, {'line': '46         ', 'type': '2'}, {'line': '10                                                         The SAS System                           Friday, May 24, 2024 03:21:00 PM', 'type': '3'}, {'line': '', 'type': '3'}, {'line': '47         ', 'type': '2'}], 'LST': ''}
>>> for line in y['LOG']:
...    print("TYPE=",line['type'], "LINE=",line['line'])
...
TYPE= 3 LINE= 9                                                          The SAS System                           Friday, May 24, 2024 03:21:00 PM
TYPE= 3 LINE=
TYPE= 2 LINE= 42
TYPE= 8 LINE=
TYPE= 8 LINE=
TYPE= 2 LINE= 43         data a; x=1;
TYPE= 8 LINE=
TYPE= 8 LINE= NOTE: The data set WORK.A has 1 observations and 1 variables.
TYPE= 8 LINE= NOTE: DATA statement used (Total process time):
TYPE= 8 LINE=       real time           0.20 seconds
TYPE= 8 LINE=       cpu time            0.01 seconds
TYPE= 8 LINE=
TYPE= 8 LINE=
TYPE= 2 LINE= 43       !              proc print data=NoExist; run;
TYPE= 6 LINE= ERROR: File WORK.NOEXIST.DATA does not exist.
TYPE= 8 LINE=
TYPE= 8 LINE= NOTE: The SAS System stopped processing this step because of errors.
TYPE= 8 LINE= NOTE: PROCEDURE PRINT used (Total process time):
TYPE= 8 LINE=       real time           0.00 seconds
TYPE= 8 LINE=       cpu time            0.00 seconds
TYPE= 8 LINE=
TYPE= 2 LINE= 44
TYPE= 2 LINE= 45
TYPE= 2 LINE= 46
TYPE= 3 LINE= 10                                                         The SAS System                           Friday, May 24, 2024 03:21:00 PM
TYPE= 3 LINE=
TYPE= 2 LINE= 47
>>>
>>>
tomweber-sas commented 1 month ago

HA! I found it! Now I'm good for a long weekend!

>>> y = sas.submit('data a; x=1; proc print data=NoExist; run;', results='text', loglines=True)
/opt/tom/github/saspy/saspy/sasioiom.py:1076: UserWarning: Noticed 'ERROR:' in LOG, you ought to take a look and see if there was a problem
  warnings.warn("Noticed 'ERROR:' in LOG, you ought to take a look and see if there was a problem")
>>> y
{'LOG': [{'line': '7                                                          The SAS System                           Friday, May 24, 2024 04:36:00 PM', 'type': 'Title'}, {'line': '', 'type': 'Title'}, {'line': '34         ', 'type': 'Source'}, {'line': '35         data a; x=1;', 'type': 'Source'}, {'line': '', 'type': 'Note'}, {'line': 'NOTE: The data set WORK.A has 1 observations and 1 variables.', 'type': 'Note'}, {'line': 'NOTE: DATA statement used (Total process time):', 'type': 'Note'}, {'line': '      real time           0.05 seconds', 'type': 'Note'}, {'line': '      cpu time            0.02 seconds', 'type': 'Note'}, {'line': '      ', 'type': 'Note'}, {'line': '', 'type': 'Note'}, {'line': '35       !              proc print data=NoExist; run;', 'type': 'Source'}, {'line': 'ERROR: File WORK.NOEXIST.DATA does not exist.', 'type': 'Error'}, {'line': '', 'type': 'Note'}, {'line': 'NOTE: The SAS System stopped processing this step because of errors.', 'type': 'Note'}, {'line': 'NOTE: PROCEDURE PRINT used (Total process time):', 'type': 'Note'}, {'line': '      real time           0.00 seconds', 'type': 'Note'}, {'line': '      cpu time            0.00 seconds', 'type': 'Note'}, {'line': '      ', 'type': 'Note'}, {'line': '36         ', 'type': 'Source'}, {'line': '37         ', 'type': 'Source'}, {'line': '38         ', 'type': 'Source'}, {'line': '8                                                          The SAS System                           Friday, May 24, 2024 04:36:00 PM', 'type': 'Title'}, {'line': '', 'type': 'Title'}, {'line': '39         ', 'type': 'Source'}], 'LST': ''}
>>> for line in y['LOG']:
...    print("TYPE=",line['type'], "LINE=",line['line'])
...
TYPE= Title LINE= 7                                                          The SAS System                           Friday, May 24, 2024 04:36:00 PM
TYPE= Title LINE=
TYPE= Source LINE= 34
TYPE= Source LINE= 35         data a; x=1;
TYPE= Note LINE=
TYPE= Note LINE= NOTE: The data set WORK.A has 1 observations and 1 variables.
TYPE= Note LINE= NOTE: DATA statement used (Total process time):
TYPE= Note LINE=       real time           0.05 seconds
TYPE= Note LINE=       cpu time            0.02 seconds
TYPE= Note LINE=
TYPE= Note LINE=
TYPE= Source LINE= 35       !              proc print data=NoExist; run;
TYPE= Error LINE= ERROR: File WORK.NOEXIST.DATA does not exist.
TYPE= Note LINE=
TYPE= Note LINE= NOTE: The SAS System stopped processing this step because of errors.
TYPE= Note LINE= NOTE: PROCEDURE PRINT used (Total process time):
TYPE= Note LINE=       real time           0.00 seconds
TYPE= Note LINE=       cpu time            0.00 seconds
TYPE= Note LINE=
TYPE= Source LINE= 36
TYPE= Source LINE= 37
TYPE= Source LINE= 38
TYPE= Title LINE= 8                                                          The SAS System                           Friday, May 24, 2024 04:36:00 PM
TYPE= Title LINE=
TYPE= Source LINE= 39
>>>
ShuguangSun commented 1 month ago

It helps me. Thanks.

tomweber-sas commented 3 weeks ago

Sorry, been off on other things, but since this seems to be what you're asking for, and I'm not seeing any issues with it, I 'm good to merge it in and build a new release with it. Does that work for you? Is it still working for you without issue?

Thanks, Tom

ShuguangSun commented 3 weeks ago

It works for me! please go ahead.

tomweber-sas commented 2 weeks ago

This is now in the current production release, V5.14.0. Go ahead and install that so you have the latest. I'll close this ticket. Let me know if you need anything else! Thanks, Tom