search

ucb-rit / brc_oodapps

5 stars 1 forks source link

Suggestions for Additional Interactive Apps to be added to OOD #16

Open markyashar opened 3 years ago

markyashar commented 3 years ago

Regarding the discussion in today's (6/22/21) dev meeting on additional apps to consider adding to the OOD deployment/implementation on Savio:

In addition to soliciting feedback from users on what additional interactive apps should be included in OOD, some additional considerations should be given to what apps Savio users are currently using on the Savio viz node and if the viz node is retired, could some of those apps be moved to OOD?

It would be useful to review the meeting notes from an earlier dev meeting on OOD that took place in December 2020 here: https://docs.google.com/document/d/1C-5zW6_x9s-0bRDH0XTHEIOX-AcRp1I95vg1SW_fTfk/edit?usp=sharing

Some suggestions that have been discussed in meetings and that have been brought up in ServiceNow tickets/consultations and/or that appear in FCA account request applications include (and some of these are already supported and enabled by OOD/OSC -- see https://openondemand.org/ and https://osc.github.io/ood-documentation/latest/install-ihpc-apps.html):

Quesiton: If software is not free and requires a license on Savio, can it still be included as an interactive app on OOD, or is licensed (non-free) software off-limits for access as an interactive app through OOD?

Maybe we can check on what software is currently being used on the Savio viz node with something like:

ssh viz [myashar@viz ~]$ top | grep -v root

%Cpu(s): 0.8 us, 0.9 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65923752 total, 46181740 free, 10853744 used, 8888268 buff/cache KiB Swap: 7312380 total, 2948688 free, 4363692 used. 54320444 avail Mem

19354 tasnuva 20 0 10.5g 981044 113564 S 12.5 1.5 5769:34 MATLAB
27891 oekenta 20 0 12.6g 2.7g 200984 S 4.3 4.2 431:01.78 MATLAB
14327 ann_sch+ 20 0 17.0g 3.8g 336796 S 3.0 6.1 679:53.72 MATLAB
27398 ann_sch+ 20 0 2781628 180712 27432 S 3.0 0.3 553:43.07 cef_helper
12074 azhou314 20 0 1955420 281580 66500 S 1.7 0.4 2:27.09 tensorboard
18771 tasnuva 20 0 253332 8980 7464 S 1.7 0.0 8:27.35 vncserverui
13936 myashar 20 0 58820 2620 1528 R 1.0 0.0 0:00.07 top
817 paciorek 20 0 283076 49416 1988 S 0.7 0.1 307:47.27 jupyter-noteboo
18493 cbachand 20 0 9887.2m 985.7m 118808 S 0.7 1.5 233:06.40 MATLAB
20782 cth 20 0 824296 299500 18700 S 0.7 0.5 230:17.83 firefox
12710 m_chint+ 20 0 359280 6896 4716 S 0.3 0.0 5:42.48 Xvnc-core
18739 tasnuva 20 0 354852 20732 7992 S 0.3 0.0 3:04.35 Xvnc-core

%Cpu(s): 1.3 us, 0.7 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65923752 total, 46182248 free, 10853372 used, 8888132 buff/cache KiB Swap: 7312380 total, 2948688 free, 4363692 used. 54320804 avail Mem

19354 tasnuva 20 0 10.5g 981864 113564 S 25.2 1.5 5769:35 MATLAB
27891 oekenta 20 0 12.6g 2.7g 200984 S 5.0 4.2 431:01.93 MATLAB
14327 ann_sch+ 20 0 17.0g 3.8g 336796 S 3.0 6.1 679:53.81 MATLAB
27398 ann_sch+ 20 0 2781628 180712 27432 S 3.0 0.3 553:43.16 cef_helper
18493 cbachand 20 0 9887.2m 985.7m 118808 S 0.7 1.5 233:06.42 MATLAB
817 paciorek 20 0 283076 49416 1988 S 0.3 0.1 307:47.28 jupyter-noteboo
5643 oekenta 20 0 2762772 22120 2116 S 0.3 0.0 74:27.65 MATLAB
13936 myashar 20 0 58820 2620 1528 R 0.3 0.0 0:00.08 top
20782 cth 20 0 824296 299500 18700 S 0.3 0.5 230:17.84 firefox
24317 oekenta 20 0 373592 35404 8004 S 0.3 0.1 50:15.87 Xvnc-core

%Cpu(s): 0.7 us, 0.7 sy, 0.0 ni, 98.4 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st

or

[myashar@viz ~]$ ps -ef | grep -v root | more

UID PID PPID C STIME TTY TIME CMD paciorek 812 1 0 May11 ? 00:00:00 /bin/sh -c module list; jupyter notebook --no-browser --ip="" paciorek 817 812 0 May11 ? 05:07:48 /global/software/sl-7.x86_64/modules/langs/python/3.6/bin/python /global/software/sl-7.x86_64/modules/langs/python/3.6/bin/jupyter-notebook --no-browse r --ip= oekenta 1144 24344 0 May02 ? 00:00:01 xterm oekenta 1146 1144 0 May02 pts/4 00:00:00 bash oekenta 1850 1146 0 May02 pts/4 00:00:08 emacs oekenta 1899 1850 0 May02 pts/5 00:00:09 /usr/bin/gdb /global/software/sl-7.x86_64/modules/tools/matlab/r2017b/bin/glnxa64/MATLAB jacktse+ 2460 1 0 May16 ? 00:00:00 vncserver-virtual -geometry 1600x900 jacktse+ 2461 2460 0 May16 ? 00:02:21 /usr/bin/Xvnc-core :11 -auth /global/home/users/jacktseng/.Xauthority -pn -geometry 1600x900 jacktse+ 2481 2460 0 May16 ? 00:00:00 /bin/sh /etc/vnc/xstartup.custom jacktse+ 2485 2481 0 May16 ? 00:00:00 xterm -geometry 80x24+10+10 -ls -title x11 Desktop jacktse+ 2486 2481 0 May16 ? 00:00:00 /global/software/sl-6.x86_64/modules/tools/openbox/3.6.1/bin/openbox jacktse+ 2490 2485 0 May16 pts/24 00:00:00 -bash jacktse+ 2525 1 0 May16 ? 00:05:25 /usr/bin/vncserverui virtual 16 jacktse+ 2540 2525 0 May16 ? 00:00:07 /usr/bin/vncserverui -statusicon 5 rpc 4097 1 0 Apr21 ? 00:00:09 /sbin/rpcbind -w dbus 4105 1 0 Apr21 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation ntp 4176 1 0 Apr21 ? 00:00:10 /usr/sbin/ntpd -u ntp:ntp -g munge 4895 1 0 Apr21 ? 00:08:10 /usr/sbin/munged rpcuser 5055 1 0 Apr21 ? 00:00:00 /usr/sbin/rpc.statd oekenta 5643 1899 0 May02 pts/5 01:14:27 /global/software/sl-7.x86_64/modules/tools/matlab/r2017b/bin/glnxa64/MATLAB -nojvm oekenta 10091 1 0 May01 ? 00:00:00 dbus-launch --autolaunch=0e568ebb0af34e44879710be3e46ae97 --binary-syntax --close-stderr oekenta 10092 1 0 May01 ? 00:00:00 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session ann_sch+ 10599 1 0 Jun10 ? 00:00:00 vncserver ann_sch+ 10600 10599 0 Jun10 ? 00:01:39 /usr/bin/Xvnc-core :7 -auth /global/home/users/ann_scheliga/.Xauthority -pn ann_sch+ 10692 10599 0 Jun10 ? 00:00:00 /bin/sh /etc/vnc/xstartup.custom ann_sch+ 10693 10692 0 Jun10 ? 00:00:00 xterm -geometry 80x24+10+10 -ls -title x11 Desktop ann_sch+ 10694 10692 0 Jun10 ? 00:00:00 /global/software/sl-6.x86_64/modules/tools/openbox/3.6.1/bin/openbox ann_sch+ 10696 10693 0 Jun10 pts/13 00:00:00 -bash ann_sch+ 10737 1 0 Jun10 ? 00:00:17 /usr/bin/vncserverui virtual 17 ann_sch+ 10752 10737 0 Jun10 ? 00:00:00 /usr/bin/vncserverui -statusicon 5 azhou314 11225 1 0 15:09 ? 00:00:00 SCREEN -S viz azhou314 11226 11225 0 15:09 pts/20 00:00:00 /bin/bash jacktse+ 12038 1 0 May14 ? 00:00:00 Xvfb :99 -screen 0 1024x768x24 +extension GLX +render -noreset azhou314 12074 11226 1 15:09 pts/20 00:02:29 /global/home/users/azhou314/.conda/envs/perception/bin/python /global/home/users/azhou314/.conda/envs/perception/bin/tensorboard --logdir logdirDIMEX/ --port 1234 ioannis 12318 1 0 May05 ? 00:00:00 vncserver ioannis 12319 12318 0 May05 ? 00:02:21 /usr/bin/Xvnc-core :6 -auth /global/home/users/ioannis/.Xauthority -pn ioannis 12340 12318 0 May05 ? 00:00:00 /bin/sh /etc/vnc/xstartup.custom ioannis 12345 12340 0 May05 ? 00:00:03 xterm -geometry 80x24+10+10 -ls -title x11 Desktop ioannis 12346 12340 0 May05 ? 00:00:00 /global/software/sl-6.x86_64/modules/tools/openbox/3.6.1/bin/openbox ioannis 12350 12345 0 May05 pts/12 00:00:00 -bash ioannis 12379 1 0 May05 ? 00:01:08 /usr/bin/vncserverui virtual 22 ioannis 12394 12379 0 May05 ? 00:00:01 /usr/bin/vncserverui -statusicon 5 m_chint+ 12709 1 0 May05 ? 00:00:00 vncserver m_chint+ 12710 12709 0 May05 ? 00:05:42 /usr/bin/Xvnc-core :5 -auth /global/home/users/m_chintalapati/.Xauthority -pn m_chint+ 12739 12709 0 May05 ? 00:00:00 /bin/sh /etc/vnc/xstartup.custom m_chint+ 12742 12739 0 May05 ? 00:00:01 /global/software/sl-6.x86_64/modules/tools/openbox/3.6.1/bin/openbox m_chint+ 12784 1 0 May05 ? 00:01:12 /usr/bin/vncserverui virtual 17 m_chint+ 12799 12784 0 May05 ? 00:00:01 /usr/bin/vncserverui -statusicon 5 jacktse+ 14050 1 0 May15 ? 00:00:00 vncserver -geometry 1600x900 jacktse+ 14051 14050 0 May15 ? 00:02:05 /usr/bin/Xvnc-core :10 -auth /global/home/users/jacktseng/.Xauthority -pn -geometry 1600x900 jacktse+ 14070 14050 0 May15 ? 00:00:00 /bin/sh /etc/vnc/xstartup.custom jacktse+ 14075 14070 0 May15 ? 00:00:00 xterm -geometry 80x24+10+10 -ls -title x11 Desktop

markyashar commented 3 years ago

Additional links to interactive apps that could be included in OOD:

This slide deck from an Open OnDemand presentation might be of interest: https://drive.google.com/file/d/1sj1Hjt_ywVtKfva2wDth-_HZz7sRL7D0/view?usp=sharing

markyashar commented 3 years ago

In regards to the issue of monitoring which modules and applications are being used on the Savio viz node to help determine which interactive apps to add to OOD, this Ask.CI forum thread ("Slurm Reports: Frequency of Application/Module Use") may be of interest: https://ask.cyberinfrastructure.org/t/slurm-reports-frequency-of-application-module-use/1947

markyashar commented 2 years ago

Some additional ideas and suggestions for interactive apps to add to OOD:

(1) TensorBoard -- see here also for examples of implementation of TensorBoard on OOD:

https://discourse.openondemand.org/t/tensorboard-for-open-ondemand/340

https://github.com/stanford-rc/sh_ood-apps/tree/main/sh_tensorboard

https://github.com/zPianodude/ood-apps

(2) It might also be worthwhile to explore photogrammetry workflows GUI on OOD (Chris H.) when the OOD desktop app is available, etc., i.e., Agisoft Metashape: https://www.agisoft.com/. For example, see:

https://docs.google.com/document/d/1OQtpA7SSCyWOiFov-aDjC4TPWrcmCtJD/edit?usp=sharing&ouid=115066181203367192083&rtpof=true&sd=true

and

https://drive.google.com/drive/folders/1JErpljAvnTE_0RwoTReIh02ft9awcQV0?usp=sharing

(3) It could be useful to consider adding Globus itself (if at all possible), and other data transfer tools such as Filezilla, Cyberduck, or the rclone GUI, as an interactive app to Open OnDemand, and/or via direct integration with the Open OnDemand Files app [1]. This might provide an even more convenient way for Savio users to carry out large data transfers to and from Savio via Globus, though I'm not sure if this capability currently exists within OOD at this time. (The same can be considered for any future SRDC OOD implementations).

[1] It looks like the OOD development team has been working on online integration of Globus with the Files app, but I'm not sure if it has actually been successfully implemented yet:

https://discourse.openondemand.org/t/scheduling-of-open-ondemand-2-0-release/1205

https://trello.com/c/rcnBbBUZ/51-globus-online-integration

https://trello.com/c/iPKFaOx1/45-files-app-replacement