chandra-mta / Ocat_Flask_App

USINT Ocat written in the Python Flask Web Framework
0 stars 0 forks source link
notification-system web-application

########################### Usint Ocat Related Scripts ###########################

========== Structure

app.db              ---- sql database
update_user_database.py --- update user database part in app.db
ocat.py             ---- probably we don't need this, but registering the fuctions
config.py           ---- config file
env/                ---- environment setting; currently set for python3.8
other_scripts       ---- a directory to keep related non-flask python scripts
    house_keeping   ---- a directory to keep files used by non-flask python scripts
cdo_warning_list    ---- a file to keep <obsid>.<rev> which needs cdo warining
                         probably we should move this to somewhere-else

app:
        email.py    ---- email related functions
        __init__.py ---- setting functions
        model.py    ---- setting models of users

        auth        ---- a directory to keep authentication related scripts
        chkupdata   ---- a directory to keep checkupdata related scripts
        errors      ---- a directory to keep error handler related scripts
        express     ---- a directory to keep express signoff related scripts
        main        ---- a directory to keep the main page related scripts
        ocatdatapage---- a directory to keep ocat data page related scripts
        orupdate    ---- a directory to keep orupdate related scripts
        scheduler   ---- a directory to keep scheduler related scripts
        supple      ---- a directory to keep supplemental python scripts

        static:
            color_list      ---- a list of color coding
            dir_list        ---- a list directories used by scripts
            js              ---- a directory to keep js scripts
            no_plots.pyg    ---- a png file saying no plot
            ocat_style.css  ---- ocat css style sheets
            param_list      ---- a list of parameters used in ocat related scripts
            ocatdatapage    ---- a directory to keep ocatdatapage related static files/html pages
            orupdate        ---- a directory to keep orupdate related static html page
            scheduler       ---- a directroy to keep a schedular related static html page

        templates:  
            base.html       ---- a base html template
            index.html      ---  a main index page

            <function dir>  ---- directories to keep templates of the functions

==== auth

login authetication process. user names/passwords are those of HEAD.

routes.py --- a main script forms.py --- keeps forms based on flask form init.py --- a script to setup the function

data: /data/mta4/CUS/www/Usint/Pass_dir/.htpasswd

templates: login.html --- the main page bad_login.html --- the page to show a wrong authentificaiton

==== main

display the top page before going to the indivisual functions

routes.py --- a main script init.py --- a script to setup the function

templates: index.html --- the index page (located at ../templates/ level)

===== error

error handler

handlers.py --- a main script init.py --- a script to setup the function

data: none

templete: 404.html --- 404 error page 500.html --- 500 error page

========= chkupdata

display all original/requested/current parameter values for a given ..

routes.py --- a main script init.py --- a script to setup the function

related scripts: supple/read_ocat_data.py

data: current_app.config['DATA_DIR']/updates/. AXAF sybase batabase (via read_ocat_data.py)

templates: index.html --- the main page display.html --- the page to display the parameter list try_agin.html --- the page to display the notice, when . is not found macros.html --- macro holder

======= express

express sign-off/approval page

routes.py --- a main script init.py --- a script to setup the function

related scripts: ocatdatapage/create_selection_dict.py ocatdatapage/update_data_record_file.py

data: current_app.config['DATA_DIR']/updates/. current_app.config['DATA_DIR']/updates/updates_table.db current_app.config['DATA_DIR']/updates/approved AXAF sybase batabase (via read_ocat_data.py)

templates: index.html --- the main page macros.html --- macro holder

============= ocatdatapage

ocat data page to udpate the parameter values

routes.py --- a main script init.py --- a script to setup the function check_value_range.pyA --- check whether the values are in the expected range create_selection_dict.py --- create a dict of p_id <--> [] send_notifications.py --- sending out notification submit_other_obsids.py --- update obsids on a list as the original obsid was update update_data_record_file.py --- create a data recorde file for a given obsid

related scripts: supple/read_ocat_data.py email.py

data: current_app.config['DATA_DIR']/updates/. current_app.config['DATA_DIR']/updates/updates_table.db current_app.config['DATA_DIR']/updates/approved AXAF sybase batabase (via read_ocat_data.py)

/mp_long_term --- planned roll angle from MP site /scheduled_obs_list --- scheduled obsids templates: index.html --- the main page/prameter value update page display_parameters.html --- updated parameter value check page finalize.html --- the page to display the job complete notification provide_obsid.html --- the page to display , if it was not found macros.html --- macro holder ======== orupdate ======== target parameter status page routes.py --- a main script __init__.py --- a script to setup the function related scripts: supple/get_value_from_sybase.py ocatdatapage/create_selection_dict.py ocatdatapage/update_data_record_file.py data: current_app.config['DATA_DIR']/updates/. current_app.config['DATA_DIR']/updates/updates_table.db current_app.config['DATA_DIR']/updates/approved AXAF sybase batabase (via read_ocat_data.py) templates: index.html --- the main page macros.html --- macro holder Note: html_base is hard coded in rountes.py: replace when it is moved the page is refreshed every 3 minites to display most recent data. this is done becasue multiple users can be updating the databases and someone else might updated them while a user tries to update the database. ============== rm_submission ============== remove an accidental submission routies.py --- a main script __init__.py --- a script to setup the function data: current_app.config['DATA_DIR']/updates/. current_app.config['DATA_DIR']/updates/updates_table.db current_app.config['DATA_DIR']/updates/approved template: index.html --- the main page macros.html --- macro holder ========= scheduler ========= POC duty sign-up sheet routies.py --- a main script read_poc_schedule.py --- a script to read the schedule database and create a data table __init__.py --- a script to setup the function related scripts: other_scripts/create_schedule_table.py other_scripts/write_this_week_too_poc.py data: current_app.config['INFO_DIR']/schedule template: index.html --- the main page macros.html --- macro holder ====== supple ====== provide supplemental scripts used by several groups get_value_from_sybase.py --- a script to access AXAF sybase database and read data values ocat_common_functions.py --- a collection of functions used by other scripts read_ocat_data.py --- a script to ectract all parameter values for a given obsid ============== other scripts ============== this directory keeps non-flask python scripts create_schedule_table.py --- a create a static html and send out notifications write_this_week_too_poc.py --- a write out the current POC at /home/mta/TOO-POC ================= related scripnts ================= scripts which create data used by this set of flask scripts in /data/mta4/obs_ss/ --------------------- find_planned_roll.py --- find a roll angles find_scheduled_obs.py --- find scheduled observations in /data/mta4/CUS/www/Usint/ ----------------------------- create_schedule_table.py --- create POC duety table html/send POC duty notification signoff_request.py --- send out sign-off request check_hrc_si_mode_sign_off.py --- check hrc si mode sign off status in /data/mta4/CUS/www/Usint/Too_Obs ------------------------------------- too_ddt_update.py --- check too/ddt observaitons and udpate related information ========= Live Data ========= Active databases are located in: /data/mta4/CUS/www/Usint/ocat/ * approved --- a list of approved obsid * updates_table.db --- a SQLite Database of sign-off status . NULL --- the column does not need to sign-off NA --- the column needs to be signed-off N/A --- the column did require sign-off but this set of changes is no longered desired by usint * updates/. --- a record of updated parameters for . * /data/mta4/CUS/www/Usint/ocat/Info_save/too_contact_info/schedule --- POC duty schedule /data/mta4/obs_ss/mp_long_term --- planned roll angle from MP site /data/mta4/obs_ss/scheduled_obs_list --- scheduled obsids Note: * replace dir_list with dir_list_live in /data/mta4/CUS/www/Usint/Ocat/ocat/app/static/ * html_base is hard coded in orupdate/rountes.py: replace when it is moved * please also edit folowing files to activate a live data access! config.py DEBUG app/__init__.py create_app * if you change the python path, you also need to update: ocat/env/pyvenv.cfg ========= Test Data ========= The test data are located in: /proj/web-cxc/cgi-gen/mta/Obscat/ocat in the similar configulation. extras: copy_data_to_test.py --- delete any changed made the day before, and copy a live data daily to the test site comp_list --- used by copy_data_to_test.py to check what were changed Note: replace dir_list with dir_list_test in /data/mta4/CUS/www/Usint/Ocat/ocat/app/static/ and do the reverse of the Live Data case (see "dir_list_live" and "dir_list_test"). /proj/web-cxc/cgi-gen/mta/Obscat/ocat/: ocat_dir <--- this must be changed for live data! /data/mta4/obs_ss/: obs_ss /data/mta4/www/CUS/Usint/: cus_dir /data/mta4/CUS/www/Usint/Pass_dir/: pass_dir /data/mta4/CUS/www/Usint/ocat/Info_save/too_contact_info/: info_dir ==================== Environment Setting ==================== setenv SKA /proj/sot/ska setenv PYTHONPATH "/data/mta/Script/Python3.8/envs/ska3-shiny/lib/python3.8/site-packages:/data/mta/Script/Python3.8/lib/python3.8/site-packages/" source /soft/SYBASE16.0/SYBASE.csh setenv PYTHONPATH "/soft/SYBASE16.0/OCS-16_0/python/python34_64r/lib:$PYTHONPATH" setenv FLASK_APP ocat setenv FLASK_ENV development ====================== To start a test server ====================== * set the environment above * go to "ocat" directory (e.g., /data/mta4/CUS/www/Usint/Ocat/ocat) * run: /data/mta/Script/Python3.8/bin/flask * open: http://127.0.0.1:5000/ on your browser =============== Flask libraries =============== flask-wtf flask-sqlalchemy flask-migrate flask-login flask-mail flask-bootstrap (css) flask-moment (time zone) email-validator pyjwt (json web tokens) =============== Other libraries =============== os sys re io string time datetime json sybpydb codecs pwd crypt getpass pathlib copy numpy random threading Chandra.Time ================ Flask DB update ================ /data/mta/Script/Python3.8/bin/flask db migrate -m 'followers' /data/mta/Script/Python3.8/bin/flask db upgrade ========= Cron jobs ========= cus on r2d2-v ------------- # #---TOO contact information # 3 0 * * * cd /data/mta4/CUS/www/Usint/; /data/mta4/CUS/www/Usint/create_schedule_table.py >> $HOME/Logs/too_contact.cron 2>&1 # 50 * * * * /data/mta4/CUS/www/Usint/TOO_Obs/Scripts/too_ddt_wrap_script >> $HOME/Logs/too_ddt_update.cron 2>&1 # #-- Mismatch Observation check # 5 5,8,10,12,14 * * * cd /data/mta4/CUS/www/Usint/; /data/mta4/CUS/www/Usint/find_mismatch.py >> $HOME/Logs/find_mismatch.cron 2>&1 # # #--- Sign Off Request (formally naggem.pl) # 0 4 * * 0-6 cd /data/mta4/CUS/www/Usint/; /data/mta4/CUS/www/Usint/signoff_request.py >> $HOME/Logs/signoff_request.cron 2 >&1 # #--- HRC SI Mode Signoff Checker # 34 5 * * * cd /data/mta4/CUS/www/Usint; /data/mta4/CUS/www/Usint/check_hrc_si_mode_sign_off.py >> $HOME/Logs/check_hrc_si_mode_sign_off.cron 2>&1 mta on c3po-v ------------- # #--- Read MP Long Term Web Page and Extract OBSID and Planned Roll Angle # 8 1 * * * cd /data/mta4/obs_ss; /data/mta4/obs_ss/find_planned_roll.py >> $HOME/Logs/find_planned_roll.cron 2>&1 # #--- Create an Access List for ocatdata2html.cgi # 6 12,18,23 * * 0-6 cd /data/mta4/obs_ss; /data/mta4/obs_ss/make_access_list.py >> $HOME/Logs/make_access.cron 2>&1 # #--- Scheduled Observation List # 35 * * * * cd /data/mta4/obs_ss/; /data/mta4/obs_ss/find_scheduled_obs.py >> $HOME/Logs/find_scheduled_obs.cron 2>&1 # #--- SOT Database # 30 * * * * cd /data/mta4/obs_ss/; /data/mta4/obs_ss/sot_data.ksh >> $HOME/Logs/sot_data.cron 2>&1 cus on r2d2-v ------------- # #--- update test data # 14 3 * * * cd /proj/web-cxc/cgi-gen/mta/Obscat/ocat; /proj/web-cxc/cgi-gen/mta/Obscat/ocat/copy_data_to_test.py > /dev/null 2>&1 # #--- Copying Data Set # 12 3,9,15,21 * * * cd /data/mta4/CUS/www/Usint/ocat; cp /data/mta4/CUS/www/Usint/ocat/updates_table.db /data/mta4/CUS/www/Usint/ocat/updates_table.db~ >> /dev/null 2>&1 ========== TRANSITION ========== READ: Live Data section!! Currently, Ocat Data can be written only by a user 'http', not cus. This creates a minor problem when you need to transit from the test phase to the actual operation. Here is my suggesiton. When you decide to activate the flask version of the Ocat Data Page, open: https://icxc.harvard.edu/cus/Usint/change_http_file.cgi then submit: chmod 777 /data/mta4/CUS/www/Usint/ocat/approved* chmod 777 /data/mta4/CUS/www/Usint/ocat/updates_table.db* chmod 777 -R /data/mta4/CUS/www/Usint/ocat/updates and chmod 777 /data/mta4/CUS/www/Usint/ocat/Info_save/too_contact_info/schedule* These commands makes all diretories/files accessible by anyone. I know that this is a major security problem, but the only way that I can think of during the transition. Once every users got used to the new setting, and all links are pointing to the new scripts, ask sys-help to change the owner of the diretory and all files under it to 'cus'. Then, change the permission to 755. I hope that the transition period is not more than a couple of weeks (if not shorter). Changing the html pointings. ----------------------------- Sub html pages of Updated Target List page must be updated when the transition is finished. https://cxc.cfa.harvard.edu/cus/Usint/updated.html In /data/mta4/CUS/www/Usint/Save_month_html: Use sed command: sed -i "s/https:\/\/cxc.harvard.edu\/mta\/CUS\/Usint\/chkupdata.cgi?/http:\/\/\/chkupdata\//g' *.html where is the new http address. In /data/mta4/CUS/www/Usint/updated_fill.py, comment out line 266, and update http address on 267 and un-comment that line.