search

sassoftware / saspy

A Python interface module to the SAS System. It works with Linux, Windows, and Mainframe SAS as well as with SAS in Viya.
https://sassoftware.github.io/saspy
Other
373 stars 150 forks source link

enhance lastlog() for methods that submit multiple blocks of code #320

Closed tomweber-sas closed 4 years ago

tomweber-sas commented 4 years ago

the lastlog() method currently returns the log from the last submit() method that was run. That's great for methods that only do one code submission, as they lastlog is just the results from that code. This is better than using saslog() and having to scroll to the end to see what code was run; saslog can become long and this becomes tedious - that's why lastlog was added.

For methods that submit more than one block of code; like df2sd and sd2df, lastlog does not provide the whole set of log segments for all of the code that was run to do what the single method represents. Thus, it's not easy to look at, or analyse/parse the log for what all code this method ran.

This issue is to fix this for any of these methods so that lastlog() really provides the SAS LOG for all of the code that needed to be run for the method, as that's really what is needed by this lastlog method.

tomweber-sas commented 4 years ago

Here's an example of current behavior, and what I'm expecting to get when I fix this:

Current results:

>>> df = cars.to_df()
>>> print(sas.lastlog())

143  filename sock socket ':56729' recfm=S  lrecl=4096;
144  data _null_; set sashelp.'cars'n ;
145  format 'MSRP'n best32.; format 'Invoice'n best32.; format 'EngineSize'n best32.; format 'Cylinders'n best32.; format
145! 'Horsepower'n best32.; format 'MPG_City'n best32.;
146  format 'MPG_Highway'n best32.; format 'Weight'n best32.; format 'Wheelbase'n best32.; format 'Length'n best32.;
147  file sock;
148  'Make'n = translate('Make'n, '2020'x, '0102'x);
149  'Model'n = translate('Model'n, '2020'x, '0102'x); 'Type'n = translate('Type'n, '2020'x, '0102'x); 'Origin'n =
149! translate('Origin'n, '2020'x, '0102'x); 'DriveTrain'n = translate('DriveTrain'n, '2020'x, '0102'x);
150  put 'Make'n '02'x;
151  put 'Model'n '02'x; put 'Type'n '02'x; put 'Origin'n '02'x; put 'DriveTrain'n '02'x; put 'MSRP'n '02'x; put 'Invoice'n '02'x;
151! put 'EngineSize'n '02'x; put 'Cylinders'n '02'x; put 'Horsepower'n '02'x; put 'MPG_City'n '02'x;
152  put 'MPG_Highway'n '02'x; put 'Weight'n '02'x; put 'Wheelbase'n '02'x; put 'Length'n '01'x; run;
NOTE: The file SOCK is:
      Local Host Name=tom64-5,
      Local Host IP addr=127.0.0.1,
      Peer Hostname Name=localhost,
      Peer IP addr=127.0.0.1,Peer Name=N/A,
      Peer Portno=56729,Lrecl=4096,Recfm=Stream

NOTE: 6420 records were written to the file SOCK.
      The minimum record length was 3.
      The maximum record length was 41.
NOTE: There were 428 observations read from the data set SASHELP.CARS.
NOTE: DATA statement used (Total process time):
      real time           0.07 seconds
      cpu time            0.06 seconds

153
154
155
>>>

Expected results showing all of the log for all of the code actually run by this method. What's above is at the end of this.

>>> df = cars.to_df()
>>> print(sas.lastlog())

95   ;*';*";*/;
96   data _null_; e = exist("sashelp.'cars'n");
97   v = exist("sashelp.'cars'n", 'VIEW');
98    if e or v then e = 1;
99   put 'TABLE_EXISTS=' e 'TAB_EXTEND=';run;
TABLE_EXISTS=1 TAB_EXTEND=
NOTE: DATA statement used (Total process time):
      real time           0.05 seconds
      cpu time            0.02 seconds

100
101  ;*';*";*/;%put %upcase(e3969440a681a2408885998500000011);
E3969440A681A2408885998500000011
102  ;*';*";*/;
103  data _null_; e = exist("sashelp.'cars'n");
104  v = exist("sashelp.'cars'n", 'VIEW');
105   if e or v then e = 1;
106  put 'TABLE_EXISTS=' e 'TAB_EXTEND=';run;
TABLE_EXISTS=1 TAB_EXTEND=
NOTE: DATA statement used (Total process time):
      real time           0.02 seconds
      cpu time            0.01 seconds

107
108  ;*';*";*/;%put %upcase(e3969440a681a2408885998500000012);
E3969440A681A2408885998500000012
109  ;*';*";*/;
110  data sasdata2dataframe / view=sasdata2dataframe; set sashelp.'cars'n ;run;
NOTE: DATA STEP view saved on file WORK.SASDATA2DATAFRAME.
NOTE: A stored DATA STEP view cannot run under a different operating system.
NOTE: DATA statement used (Total process time):
      real time           0.04 seconds
      cpu time            0.01 seconds

111  data _null_; file STDERR;d = open('sasdata2dataframe');
112  lrecl = attrn(d, 'LRECL'); nvars = attrn(d, 'NVARS');
113  lr='LRECL='; vn='VARNUMS='; vl='VARLIST='; vt='VARTYPE=';
114  put lr lrecl; put vn nvars; put vl;
115  do i = 1 to nvars; var = varname(d, i); put var; end;
116  put vt;
117  do i = 1 to nvars; var = vartype(d, i); put var; end;
118  run;
NOTE: The file STDERR is:
      Pipe command="<standard error>"
LRECL= 152
VARNUMS= 15
VARLIST=
Make
Model
Type
Origin
DriveTrain
MSRP
Invoice
EngineSize
Cylinders
Horsepower
MPG_City
MPG_Highway
Weight
Wheelbase
Length
VARTYPE=
C
C
C
C
C
N
N
N
N
N
N
N
N
N
N

NOTE: 34 records were written to the file STDERR.
      The minimum record length was 1.
      The maximum record length was 11.
NOTE: View WORK.SASDATA2DATAFRAME.VIEW used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds

NOTE: DATA statement used (Total process time):
      real time           0.05 seconds
      cpu time            0.02 seconds

119
120  ;*';*";*/;%put %upcase(e3969440a681a2408885998500000013);
E3969440A681A2408885998500000013
121  ;*';*";*/;
122  data work._n_u_l_l_;output;run;
NOTE: The data set WORK._N_U_L_L_ has 1 observations and 0 variables.
NOTE: DATA statement used (Total process time):
      real time           0.02 seconds
      cpu time            0.01 seconds

123  data _null_; file STDERR; set work._n_u_l_l_ sashelp.'cars'n (obs=0 );put 'FMT_CATS=';
124  _tom = vformatn('Make'n);put _tom;
125  _tom = vformatn('Model'n);put _tom;
126  _tom = vformatn('Type'n);put _tom;
127  _tom = vformatn('Origin'n);put _tom;
128  _tom = vformatn('DriveTrain'n);put _tom;
129  _tom = vformatn('MSRP'n);put _tom;
130  _tom = vformatn('Invoice'n);put _tom;
131  _tom = vformatn('EngineSize'n);put _tom;
132  _tom = vformatn('Cylinders'n);put _tom;
133  _tom = vformatn('Horsepower'n);put _tom;
134  _tom = vformatn('MPG_City'n);put _tom;
135  _tom = vformatn('MPG_Highway'n);put _tom;
136  _tom = vformatn('Weight'n);put _tom;
137  _tom = vformatn('Wheelbase'n);put _tom;
138  _tom = vformatn('Length'n);put _tom;
139  run;
NOTE: The file STDERR is:
      Pipe command="<standard error>"
FMT_CATS=
$
$
$
$
$
DOLLAR
DOLLAR
BEST
BEST
BEST
BEST
BEST
BEST
BEST
BEST

NOTE: 16 records were written to the file STDERR.
      The minimum record length was 1.
      The maximum record length was 9.
NOTE: There were 1 observations read from the data set WORK._N_U_L_L_.
NOTE: There were 0 observations read from the data set SASHELP.CARS.
NOTE: DATA statement used (Total process time):
      real time           0.04 seconds
      cpu time            0.03 seconds

140  proc delete data=work._n_u_l_l_;run;
NOTE: Deleting WORK._N_U_L_L_ (memtype=DATA).
NOTE: PROCEDURE DELETE used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds

141
142  ;*';*";*/;%put %upcase(e3969440a681a2408885998500000014);
E3969440A681A2408885998500000014
143  filename sock socket ':56729' recfm=S  lrecl=4096;
144  data _null_; set sashelp.'cars'n ;
145  format 'MSRP'n best32.; format 'Invoice'n best32.; format 'EngineSize'n best32.; format 'Cylinders'n best32.; format
145! 'Horsepower'n best32.; format 'MPG_City'n best32.;
146  format 'MPG_Highway'n best32.; format 'Weight'n best32.; format 'Wheelbase'n best32.; format 'Length'n best32.;
147  file sock;
148  'Make'n = translate('Make'n, '2020'x, '0102'x);
149  'Model'n = translate('Model'n, '2020'x, '0102'x); 'Type'n = translate('Type'n, '2020'x, '0102'x); 'Origin'n =
149! translate('Origin'n, '2020'x, '0102'x); 'DriveTrain'n = translate('DriveTrain'n, '2020'x, '0102'x);
150  put 'Make'n '02'x;
151  put 'Model'n '02'x; put 'Type'n '02'x; put 'Origin'n '02'x; put 'DriveTrain'n '02'x; put 'MSRP'n '02'x; put 'Invoice'n '02'x;
151! put 'EngineSize'n '02'x; put 'Cylinders'n '02'x; put 'Horsepower'n '02'x; put 'MPG_City'n '02'x;
152  put 'MPG_Highway'n '02'x; put 'Weight'n '02'x; put 'Wheelbase'n '02'x; put 'Length'n '01'x; run;
NOTE: The file SOCK is:
      Local Host Name=tom64-5,
      Local Host IP addr=127.0.0.1,
      Peer Hostname Name=localhost,
      Peer IP addr=127.0.0.1,Peer Name=N/A,
      Peer Portno=56729,Lrecl=4096,Recfm=Stream

NOTE: 6420 records were written to the file SOCK.
      The minimum record length was 3.
      The maximum record length was 41.
NOTE: There were 428 observations read from the data set SASHELP.CARS.
NOTE: DATA statement used (Total process time):
      real time           0.07 seconds
      cpu time            0.06 seconds

153  ;*';*";*/;
154
155
156  ;*';*";*/;%put %upcase(e3969440a681a2408885998500000015);
E3969440A681A2408885998500000015
>>>
tomweber-sas commented 4 years ago

Another case when not getting pandas output, which has to do sd2df under the covers to get the results back a a data frame. With text results, the log is much shorter, as there weren't that many extra steps.

Current:

>>> cars.results='text'
>>>
>>> cars.means()
                                                           The SAS System                     10:24 Thursday, September 17, 2020   1

                                                        The MEANS Procedure

                                         N
Variable      Label               N   Miss         Median           Mean        Std Dev        Minimum      25th Pctl      50th Pctl
------------------------------------------------------------------------------------------------------------------------------------
MSRP                            428      0          27635          32775          19432          10280          20330          27635
Invoice                         428      0          25295          30015          17642    9875.000000          18851          25295
EngineSize    Engine Size (L)   428      0       3.000000       3.196729       1.108595       1.300000       2.350000       3.000000
Cylinders                       426      2       6.000000       5.807512       1.558443       3.000000       4.000000       6.000000
Horsepower                      428      0     210.000000     215.885514      71.836032      73.000000     165.000000     210.000000
MPG_City      MPG (City)        428      0      19.000000      20.060748       5.238218      10.000000      17.000000      19.000000
MPG_Highway   MPG (Highway)     428      0      26.000000      26.843458       5.741201      12.000000      24.000000      26.000000
Weight        Weight (LBS)      428      0    3474.500000    3577.953271     758.983215    1850.000000    3103.000000    3474.500000
Wheelbase     Wheelbase (IN)    428      0     107.000000     108.154206       8.311813      89.000000     103.000000     107.000000
Length        Length (IN)       428      0     187.000000     186.362150      14.357991     143.000000     178.000000     187.000000
------------------------------------------------------------------------------------------------------------------------------------

                                    Variable      Label                75th Pctl         Maximum
                                    ------------------------------------------------------------
                                    MSRP                                   39215          192465
                                    Invoice                                35733          173560
                                    EngineSize    Engine Size (L)       3.900000        8.300000
                                    Cylinders                           6.000000       12.000000
                                    Horsepower                        255.000000      500.000000
                                    MPG_City      MPG (City)           21.500000       60.000000
                                    MPG_Highway   MPG (Highway)        29.000000       66.000000
                                    Weight        Weight (LBS)       3978.500000     7190.000000
                                    Wheelbase     Wheelbase (IN)      112.000000      144.000000
                                    Length        Length (IN)         194.000000      238.000000
                                    ------------------------------------------------------------
>>> print(sas.lastlog())

235
236  proc means data=sashelp.'cars'n  stackodsoutput n nmiss median mean std min p25 p50 p75 max;run;
NOTE: There were 428 observations read from the data set SASHELP.CARS.
NOTE: The PROCEDURE MEANS printed page 1.
NOTE: PROCEDURE MEANS used (Total process time):
      real time           0.11 seconds
      cpu time            0.07 seconds

237
>>>

When fixed:

>>> cars.results='text'
>>>
>>> cars.means()
                                                           The SAS System                     10:24 Thursday, September 17, 2020   1

                                                        The MEANS Procedure

                                         N
Variable      Label               N   Miss         Median           Mean        Std Dev        Minimum      25th Pctl      50th Pctl
------------------------------------------------------------------------------------------------------------------------------------
MSRP                            428      0          27635          32775          19432          10280          20330          27635
Invoice                         428      0          25295          30015          17642    9875.000000          18851          25295
EngineSize    Engine Size (L)   428      0       3.000000       3.196729       1.108595       1.300000       2.350000       3.000000
Cylinders                       426      2       6.000000       5.807512       1.558443       3.000000       4.000000       6.000000
Horsepower                      428      0     210.000000     215.885514      71.836032      73.000000     165.000000     210.000000
MPG_City      MPG (City)        428      0      19.000000      20.060748       5.238218      10.000000      17.000000      19.000000
MPG_Highway   MPG (Highway)     428      0      26.000000      26.843458       5.741201      12.000000      24.000000      26.000000
Weight        Weight (LBS)      428      0    3474.500000    3577.953271     758.983215    1850.000000    3103.000000    3474.500000
Wheelbase     Wheelbase (IN)    428      0     107.000000     108.154206       8.311813      89.000000     103.000000     107.000000
Length        Length (IN)       428      0     187.000000     186.362150      14.357991     143.000000     178.000000     187.000000
------------------------------------------------------------------------------------------------------------------------------------

                                    Variable      Label                75th Pctl         Maximum
                                    ------------------------------------------------------------
                                    MSRP                                   39215          192465
                                    Invoice                                35733          173560
                                    EngineSize    Engine Size (L)       3.900000        8.300000
                                    Cylinders                           6.000000       12.000000
                                    Horsepower                        255.000000      500.000000
                                    MPG_City      MPG (City)           21.500000       60.000000
                                    MPG_Highway   MPG (Highway)        29.000000       66.000000
                                    Weight        Weight (LBS)       3978.500000     7190.000000
                                    Wheelbase     Wheelbase (IN)      112.000000      144.000000
                                    Length        Length (IN)         194.000000      238.000000
                                    ------------------------------------------------------------
>>> print(sas.lastlog())

228  ;*';*";*/;
229  data _null_; e = exist("sashelp.'cars'n");
230  v = exist("sashelp.'cars'n", 'VIEW');
231   if e or v then e = 1;
232  put 'TABLE_EXISTS=' e 'TAB_EXTEND=';run;
TABLE_EXISTS=1 TAB_EXTEND=
NOTE: DATA statement used (Total process time):
      real time           0.05 seconds
      cpu time            0.02 seconds

233
234  ;*';*";*/;%put %upcase(e3969440a681a2408885998500000023);
E3969440A681A2408885998500000023
235  ;*';*";*/;
236  proc means data=sashelp.'cars'n  stackodsoutput n nmiss median mean std min p25 p50 p75 max;run;
NOTE: There were 428 observations read from the data set SASHELP.CARS.
NOTE: The PROCEDURE MEANS printed page 1.
NOTE: PROCEDURE MEANS used (Total process time):
      real time           0.11 seconds
      cpu time            0.07 seconds

237
238  ;*';*";*/;%put %upcase(e3969440a681a2408885998500000024);
E3969440A681A2408885998500000024
>>>
tomweber-sas commented 4 years ago

initial push of this is in new branch 'lastlog'

tomweber-sas commented 4 years ago

This has been merged in to the main branch.

biojerm commented 4 years ago

checked out using lastlog() I think i can make this work! Thanks again for the suggestions and help.

tomweber-sas commented 4 years ago

Good deal! I'll close this and I suppose I'll push this out to a new release. It's much better then it was, so getting this into production is a good thing. As always, let me know if you need anything else! Tom