Running SAS program via SasPy as batch in OnDemand Academics Cloud

[kshank5]

@tomweber-sas I have a SAS program that will take about 4-6 hours to compute. In my desktop version I run these as batch but I am trying to do this via python to implement a "one-click" process where I run my python with SasPy, establish connection with sas, run the required file in sas as a background process (or batch) in the ODA Cloud version. Is this possible at all? Are there options to run sas programs as batch in the cloud version? Thank you!

[tomweber-sas]

Yes, you can run saspy in Python batch scripts. There's an option in saspy for this, so that it isn't trying to render output produced by methods (since there's no where to render them; not interactive). Instead all results are returned from the method, so your code can do with it what you want. See https://sassoftware.github.io/saspy/advanced-topics.html#using-batch-mode, And in the example notebook, you can see example of using this in cells 61-66: https://github.com/sassoftware/saspy-examples/blob/main/SAS_contrib/Ask_the_Expert_Demo.ipynb

[kshank5]

Thank you Tom for the details. I am able to now connect to a table and run some procs in python using SasPy. This will be most useful to me if I could just "call" a sas program, say "ComputeIntake.sas" in my studio and then bring back the results to python. I went through the notebook link but it only seems to have examples on how to access a table from SAS and how to "submit" a sas code block to SAS ODA. Is there a specific syntax where I can say submit("home/kshank5/SasPrograms/ComuputeIntake.sas" from python where ComputeIntake.sas is housed in the Studio and python kicks off the execution in SAS. The results can remain in Studio. At that point I know how to access the results tables from ODA to Python with your notebook. PS: My apologies for asking what may be documented some place. I am trying to get this done very quickly and dont have a whole lot of time. Hence checking to see if there is someone out there who already knows how to do this. Thank you for your patience.

[tomweber-sas]

Well, some of what you're asking doesn't make sense. Studio and saspy are both simply Clients which connect to SAS servers. So, there's no concept of saspy and studio interacting. Nothing is IN Studio, if there's something that persists, like the datasets you have in that library, or files in a persisted directory, you now can access from both clients, then that's on the sever tier. Each time you connect to ODA (a SAS Workspace server) from either studio or saspy, it's a different workspace server every time. So anything that persists across connections is stored on the server side. You can't submit code from one client and get the results from another (they aren't connected to the same server). You can create data that persists on the server side, from one client, and then access that from another client later, if it's in a library that persists for you across connections. So, I'm not really clear on the interaction between studio and saspy that you're asking about. In either case, you're connected to SAS, so you can do SAS things from either client. If you want to submit a SAS program, just do sas.submit("%include '/home/kshank5/SasPrograms/ComuputeIntake.sas';") as long as that file is on that server in that directory. You can submit that from either studio or saspy; either way it's just submitting SAS code to SAS.

[kshank5]

Sorry! Yes; I did muddy up the terminology. In my head I was calling the SAS server as studio to differentiate clearly between SAS and python (pardon me, that pic is in my head, not necessarily precise). I meant SAS server and Python (I use Pycharm IDE to be specific). the %include submit worked. However, I do not see the mylib or the output in the SAS server that the program is meant to create. The sas code file I "submit" creates the myLib with "stroke" table in it as below libname mylib "/home/kshank5/EPG1V2/1_CSCI 5663 - Stats Programming/1_Assignment#11_Final Project"; No errors from within SasPy in Pycharm. Is this the expected behavior? I am very new to SAS and it does not help that I am racing against time! Thanks for your support. Here is my code

import saspy
sas = saspy.SASsession(results='html')
sas.set_batch(True)
sas.submit("%include '/home/kshank5/EPG1V2/1_CSCI 5663 - Stats Programming/1_Assignment#11_Final Project/Assignment#11_FinalProject.sas';")

Here is the output in PyCharm

> Using SAS Config named: oda
> SAS Connection established. Subprocess id is 37016
> 
> SAS Connection terminated. Subprocess id was 37016
> 
> Process finished with exit code 0

[tomweber-sas]

Yeah, no problem. So I have no idea what SAS code is in that file. What are you executing? What output does it create? Running that code (which you are submitting in batch, FWIW) doesn't do anything with the output it gets. sas.submit() returns a dict with both the SASLOG and any output that was created by whatever you submitted( the LiSTing). You did nothing with that. didn't even assign it to a variable, so it's 'lost'. What are you expecting from whatever it is you're running? Again, I can't see what you're doing, so no info equals 'I don't know' on my end. What are you expecting to see, and how/where are you expecting to see it? What code did you execute in SAS? Also, how about issuing print(sas.saslog()) as the last statement, so you can see what actually happened on the SAS side? That's always helpful, especially when you don't really know what SAS does. If you show me the code in the file and the log after running it, I can actually provide some feedback, if I know what you are expecting to happen.

[kshank5]

Thanks Tom. I guess I understand the part where not assigning the output sends it into an abyss. However, the library I create with my code should be there on the SAS server, no? Attaching my code.

strokedata.txt

[tomweber-sas]

Ok, that helps a lot. Have you ever run any of this interactively to see what happens and see what you get? You're running a lot of code in one single submit. It's creating files on disk, and also generating a lot of output to both the LOG and the LST. This is exactly what the section of the example notebook I pointed you at, regarding batch mode is about. So, what are you imaging is supposed to happen with all of the output that goes to the LST. There's a lot that ends up in the LOG and way more that comes back in the LIST; dozens of which are plots and graphs and the such from the analytic procs, lots of which is just text output and data that's 'printed' out (text output not graphs). Where are expecting all of this to go, or how/where are you expecting to 'see' it. It's all gone as as soon as your batch program ends. What are you actually expecting/trying to do with all that output?

[kshank5]

first and foremost, I was expecting the code to create "mylib" under libraries on the SAS server since that is the first statement. But when that did not happen, I knew I was doing something wrong so did not bother to dig deeper into where the results of my program went. I needed to troubleshoot why the first statement in my sas code was not being executed. Does that make sense? As you had suggested, I printed the logs and it says, among other things,

43400 rows created in MYLIB.PRESTROKE from /home/kshank5/EPG1V2/1_CSCI 5663 - Stats Programming/1_Assignment#11_Final Project/Stroke.csv.

But I cannot see mylib or the data under it in my SAS IDE. Should I not be expecting to see that?

When I run the same program from the SAS studio on the server directly, it creates MyLib and I am able to see it. Should I not be expecting the same behavior when I invoke the same program from my Python IDE through a sas.submit().

[tomweber-sas]

I'm not completely clear on what you're looking at or what you're expecting. Not sure you understand what a libname staement means. When you assign a libref in one SAS session, that doesn't make that libref appear in some other client on some other SAS session. If you create a libref in both session, that's referencing the same file system directory (that both servers can access), then you would see the same tables in it from both clients (you may need to refresh one in a UI if you created a table in the other since you looked at the window showing what tables are in it). The librefs don't even need to be named the same; they are pointing at a filesystem location. If you create a libref in each client point to the same filesystem location, you will see the same tables regardless of the name of the librefs.

The one line you show from the log, above, is that from the second step; the proc import? If so, then the libref was assigned and the table was written to that filesystem location. So, again, I'm not sure you're clear on what's really going on. If there were errors in the log about those, then that's something else, though you didn't mention an error, and I can't see the log for myself to know what happened.

This may get a lot further if we can do a teams meeting where you can show me what you're actually doing and I can help you understand what it is you're trying todo from these two different clients. I really don't know what you're expecting to get or have happen with how you're describing all of this. Do you want to get on a teams meeting at some point so I can actually see what you're doing and help explain how things work for you?

[kshank5]

Hi Tom - TBH, I do NOT understand SAS at all - not just the libref, just about everything! And I have not done any justice to the reading part. I HAVE to use it for my research, so I am just learning on-the-go!

So, yes, a Teams call would be awesome! Please let me know a few days/times you are available and I will make something work. Best to skip tomorrow from 8 to 2PM CST. I really appreciate all your help and support.

[tomweber-sas]

OK, thanks for telling me that; I kinda guessed it, but didn't want to say anything :) Looks like anytime Thursday. I'm EST, so mostly tomorrow is gone and I have stuff all day on Wed. I'm free all day Thurs though, so any time then would work.

[tomweber-sas]

Also I have Friday afternoon open too. And, I get it, trying to learn SAS on the fly. It's not as obvious as learning Python on the fly. SAS is very powerful, but not instinctive or outright simple. I know.

[kshank5]

Thank you for understanding! I can do Thursday any time between 12PM and 3 00PM EST. Or Friday between 10 -11AM or after 1 30PM EST.

[tomweber-sas]

Cool, can you email me so I can reply with a meeting request. Send it tomorrow. Tom.Weber@sas.com

[tomweber-sas]

Just a catchup. We were able to get together today and walk through ways to accomplish what was needed. So, next is to code it up and see. More to come ...

[tomweber-sas]

Hey @kshank5 I hope you've been able to get this working like we discussed. I'm going to close this issue since it's not really still an open problem. However, if you need anything else, just reopen it, or open another or however you like. Still happy to help, of course! Thanks! Tom