search

IBMSpectrumComputing / lsf-faas

LSF Python Function service
Apache License 2.0
5 stars 5 forks source link

lsf-faas integration with jupyter notebook #4

Closed asm582 closed 4 years ago

asm582 commented 4 years ago

is there a documentation on lsf-faas integration with jupyter notebook?

asm582 commented 4 years ago

I ran the below snippet to get lsf-faas working with jupyter notebook: import os import sys module_path = os.path.abspath(os.path.join('/lsf-faas/src')) if module_path not in sys.path: sys.path.append(module_path)

I am already logged-in and jupyter notebook is running from the template that LSF admin provides. It seems logging is mandatory before submitting a job, can we make the logon functionality optional, or is another way to submit jobs?


Please logon before using this function.
xawyong commented 4 years ago

there is a bug when verify token ( if server return "there are no jobs submitted by user ", then it will cause verify token failed) so you get "Please logon before using this function." now we use other API to verify token, think it can solve your problem @asm582

now installation is mandatory, I think it will solve the integration with jupyter notebook.

Unit test in jupyter notebook

In [1]:from lsf_faas import lsf
         def myfun(*arg):
               return arg
         id1 = lsf.sub(myfun,1,2)
​
In [2]: lsf.get(id1)

​out [2]: (1, 2)
asm582 commented 4 years ago

@xawyong can you please point to the latest installation instructions? cannot submit from a jupyter running on the cluster.

I have installed the library as below in my anaconda environment on the login node on the cluster:

(base) [asmalvan@xxxx-login-01 lsf-faas]$ python setup.py installrunning install running bdist_egg running egg_info writing lsf_faas.egg-info/PKG-INFO writing dependency_links to lsf_faas.egg-info/dependency_links.txt writing top-level names to lsf_faas.egg-info/top_level.txt reading manifest file 'lsf_faas.egg-info/SOURCES.txt' writing manifest file 'lsf_faas.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/lsf_faas copying build/lib/lsf_faas/init.py -> build/bdist.linux-x86_64/egg/lsf_faas copying build/lib/lsf_faas/lsf.py -> build/bdist.linux-x86_64/egg/lsf_faas copying build/lib/lsf_faas/lsflib.py -> build/bdist.linux-x86_64/egg/lsf_faas byte-compiling build/bdist.linux-x86_64/egg/lsf_faas/init.py to init.cpython-38.pyc byte-compiling build/bdist.linux-x86_64/egg/lsf_faas/lsf.py to lsf.cpython-38.pyc byte-compiling build/bdist.linux-x86_64/egg/lsf_faas/lsflib.py to lsflib.cpython-38.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying lsf_faas.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying lsf_faas.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying lsf_faas.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying lsf_faas.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO zip_safe flag not set; analyzing archive contents... lsf_faas.pycache.lsf.cpython-38: module MAY be using inspect.getsource creating 'dist/lsf_faas-0.1-py3.8.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing lsf_faas-0.1-py3.8.egg creating /home/asmalvan/tools/anaconda3/lib/python3.8/site-packages/lsf_faas-0.1-py3.8.egg Extracting lsf_faas-0.1-py3.8.egg to /home/asmalvan/tools/anaconda3/lib/python3.8/site-packages Adding lsf-faas 0.1 to easy-install.pth file

Installed /home/asmalvan/tools/anaconda3/lib/python3.8/site-packages/lsf_faas-0.1-py3.8.egg Processing dependencies for lsf-faas==0.1 Finished processing dependencies for lsf-faas==0.1

When I run your test program I get the below error:

from lsf_faas import lsf def myfun(*arg): return arg id1 = lsf.sub(myfun,1,2)


Please logon before using this function.```
xawyong commented 4 years ago

@asm582 I do not understand your mentioned "on the login node on the cluster" and "I am already logged-in". is logged into PAC by PACClient.py???

currently, user must log on LSF(PAC) to get the token through lsf.logon API ,no need do it again until the token is expired. the token from PACClient.py and lsf_faas is not trust each other.

asm582 commented 4 years ago

I am not sure about PAC client. I launched Jupyter notebook in the cluster using Jupyter notebook templates from the application center. I am trying to submit an LSF job from the Jupyter notebook that is already running in the cluster, is it not a valid use case?

On Thu, Sep 10, 2020 at 9:35 PM xawyong notifications@github.com wrote:

@asm582 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_asm582&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=EoJzDBOIwVt7lOICrHk1lQ&m=RgdLrtB6UvBEbh_QiXsNBkQQ-w3rQwZUNvGtImxTcfU&s=iHxT-F9IeVfJkI_quXD2NEqnfmvGDQJMgOoW7cr9XAo&e= I do not understand your mentioned "on the login node on the cluster" and "I am already logged-in". is logged into PAC by PACClient.py???

currently, user must log on LSF(PAC) to get the token through lsf.logon API ,no need do it again until the token is expired. the token from PACClient.py and lsf_faas is not trust each other.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IBMSpectrumComputing_lsf-2Dfaas_issues_4-23issuecomment-2D690820802&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=EoJzDBOIwVt7lOICrHk1lQ&m=RgdLrtB6UvBEbh_QiXsNBkQQ-w3rQwZUNvGtImxTcfU&s=uupJHNFFLv-ESNO2QbIRnMhVWnrSKUZtUHzPsPNZg2s&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABNACPOBGGZXGVBBECSBVSDSFF5HBANCNFSM4RAKX3TA&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=EoJzDBOIwVt7lOICrHk1lQ&m=RgdLrtB6UvBEbh_QiXsNBkQQ-w3rQwZUNvGtImxTcfU&s=QI8vN73YlZ7RUrWL2wM9uauPkdmzcORo08T7rPHs1ws&e= .

-- Regards,

Abhishek Malvankar

xawyong commented 4 years ago

@asm582 thanks. now i got it. I regard it as SSO case and we can handle it in future. but for now, seems you must logon. below explain more about the details or solution:

asm582 commented 4 years ago

Ok thanks, can we keep the issue open, I will wait for a PR from you to test.

Thanks

On Fri, Sep 11, 2020 at 1:30 AM xawyong notifications@github.com wrote:

@asm582 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_asm582&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=EoJzDBOIwVt7lOICrHk1lQ&m=sKnQF2nznldiknzQKV3Gpes4UDE8Fdfkzg9WsImgoII&s=HY5QLfcyxt2NnsOZNS6vsn0Lq7UeE3iqh85UosLTrF8&e= thanks. now i got it. I regard it as SSO case and we can handle it in future. but for now, seems you must logon. below explain more about the details or solution:

  • user login Application center from browser, and the token is stored into local storage as cookie by browser for future used.
  • lsf_faas directly store/use the token into/from $HOME/.lsf_faas
  • so can not share the token to each other. In your case, if you open jupyter notebook template, since there is not found valid token, so you get the message "Please logon before using this function"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IBMSpectrumComputing_lsf-2Dfaas_issues_4-23issuecomment-2D690883834&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=EoJzDBOIwVt7lOICrHk1lQ&m=sKnQF2nznldiknzQKV3Gpes4UDE8Fdfkzg9WsImgoII&s=qEtSxG0r1SM440ircqqHUtn10z331mnNKpBBu4CHfTc&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABNACPJRZ6DYZT3SA5XFXLTSFGYV7ANCNFSM4RAKX3TA&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=EoJzDBOIwVt7lOICrHk1lQ&m=sKnQF2nznldiknzQKV3Gpes4UDE8Fdfkzg9WsImgoII&s=8k2M0CY9hrFxA1iU6EUlG6BuZhtAhksoNhiJFEZfIxY&e= .

-- Regards,

Abhishek Malvankar

asm582 commented 4 years ago

Hello, I cannot log in when the Jupyter notebook is launched through the application center. 1) Can you please confirm which host should I use to log in: login node? or application center host? 2) Which port should be used to login? 3) File name that should be used to store cookies?

If you can paste commands that would be very helpful, thanks.

xawyong commented 4 years ago

@asm582 the logon logon method (username, password ,host = Application center host, port=8080, isHttps) is used to logon Application center. the port, you can get from /opt/ibm/lsfsuite/ext/gui/conf/jvm.options. by default. for http it is 8080, https is 8443.

  grep "http" /opt/ibm/lsfsuite/ext/gui/conf/jvm.options
#define http/https port
-Ddefault.http.port=8080
-Ddefault.https.port=8443

lsf_faas will store the token into $HOME/.lsf_faas/.lsfpass