search

sot / kadi

Chandra commands and events
https://sot.github.io/kadi
BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

Py3 server with + event/cmd cron jobs processing with django 3.0 #143

Closed taldcroft closed 4 years ago

taldcroft commented 4 years ago

This is based on the scm_version branch. Features:

Skare3 package requirements

In order to pass all documented tests these package versions or branches are required:

Testing

Create test environment

Roughly speaking, on kady:

$ which conda
/export/tom/miniconda3/envs/ska3-django3/bin/conda
$ conda --version
conda 4.3.21
$ conda create -n ska3-django3 --clone=ska3
$ source activate ska3-django3
$ conda uninstall --force ska3-flight ska3-core django kadi

# Original ska3 env doesn't have perl (by mistake) so fix that here.
# Without --force it wanted to re-install old versions of kadi, django, gcc and 
# other random stuff
$ conda install --force -c $ska3conda task_schedule
$ conda install --force -c $ska3conda -c $ska3conda/core-pkg-repo perl perl-core-deps perl-ska-classic

$ pip install django  # version 3.0.1
$ pip install kadi  # From git repo
Successfully installed kadi-4.18.2.dev45+g9478c71
$ pip install find_attitude  # From git repo; version 3.3.dev2+gbe4485b (py3-server branch)
$ pip install mica  # From git repo py3-server branch
Successfully installed mica-4.20.dev3+gf5136d6

$ cd ~/git/task_schedule  # modernize branch
$ make install
$ cd ~/git/watch_cron_logs  # modernize branch
$ make install 

# Data links
$ ls -l /export/tom/miniconda3/envs/ska3-django3/data
total 4
lrwxrwxrwx 1 aldcroft aldcroft 24 Dec 27 16:32 agasc -> /proj/sot/ska/data/agasc/
lrwxrwxrwx 1 aldcroft aldcroft 29 Dec 27 16:32 cmd_states -> /proj/sot/ska/data/cmd_states/
lrwxrwxrwx 1 aldcroft aldcroft 30 Dec 27 16:31 eng_archive -> /proj/sot/ska/data/eng_archive/
-rw-r--r-- 1 aldcroft aldcroft  0 Dec 27 16:33 events3.db3
lrwxrwxrwx 1 aldcroft aldcroft 32 Dec 27 16:31 find_attitude -> /proj/sot/ska/data/find_attitude/
lrwxrwxrwx 1 aldcroft aldcroft 23 Dec 27 16:31 mica -> /proj/sot/ska/data/mica/

# Edit kadi task_schedule config files to change alerts from aca to aldcroft
$ nano /export/tom/miniconda3/envs/ska3-django3/share/kadi/task_schedule_events.cfg 
$ nano /export/tom/miniconda3/envs/ska3-django3/share/kadi/task_schedule_cmds.cfg

Make events3.db3 and cmds.pkl, cmds.h5 files

Follow instructions in NOTES.build

Set up cron job

$ crontab -l | grep kadi
# py3-server branch of kadi processing in Ska3
00 09 * * * /export/tom/miniconda3/envs/ska3-django3/bin/skare task_schedule3.pl -config kadi/task_schedule_events.cfg
10 09 * * * /export/tom/miniconda3/envs/ska3-django3/bin/skare task_schedule3.pl -config kadi/task_schedule_cmds.cfg
...

$ cp events3.db3 cmds.h5 cmds.pkl /export/tom/miniconda3/envs/ska3-django3/data/kadi/

Web server

Linux

Start a local server that runs the installed version of kadi, mica, find_attitude

MacOSX:

Start a local server that runs the installed version of kadi, mica, find_attitude, sherpa (4.11.1 from sherpa channel)

Fixes #145

taldcroft commented 4 years ago

Note that this is based on current kadi master.

jeanconn commented 4 years ago

For these kind of changes to the dev ("test"?) server, what are thoughts on how we do promotion when done with testing?

taldcroft commented 4 years ago

The process for transition to the new server provides a good model.

jeanconn commented 4 years ago

The process for transition to the new server involved one going out of service and one being promoted by DNS swap. Was that your plan for the next promotion (promote test to flight by DNS change)? Would you then start a new test machine or swap flight to test and then update it in place?

taldcroft commented 4 years ago

My first choice is to promote this, along with Django 2.2 to Ska3 flight before doing work on the test server. That if course needs a different sort if testing. And there might be problems in that strategy with legacy ska access. Will need to think about that...

taldcroft commented 4 years ago

I was talking about the testing process. The promotion would be simply test on test then install the same update to production aka flight. Very simple, flight down only long enough to restart the server.

jeanconn commented 4 years ago

@taldcroft There's a bunch here. Do you want to update the PR top-text to talk about what you've already done for testing and anything specific you think reviewers should think about while reviewing?

taldcroft commented 4 years ago

Haven't requested review just yet, so stand by. :smile:

jeanconn commented 4 years ago

Sorry, got excited.

taldcroft commented 4 years ago

Yes, it's good stuff and I think almost there. 🤞 There is a fair bit of testing I have done that I will document in the top-matter.

taldcroft commented 4 years ago

And BTW django 3.0 was just release a couple weeks ago (and 3.0.1 I think). From my first test it looks to just work here at least on Mac so will go with that if possible.

taldcroft commented 4 years ago

For the record here is the script used for checking that events and commands are the same. This is a one-off modification of a similar script in ska_testr so not CM'ing it.

(ska3-django3) kady$ cat validate/write_events_cmds.py
#!/usr/bin/env python

import os
import argparse

import kadi.events
import kadi.cmds
import kadi.paths
from Chandra.Time import DateTime

def get_opt(args=None):
    parser = argparse.ArgumentParser(description='Write commands and events ')
    parser.add_argument("--stop",
                        help="Processing stop date")
    parser.add_argument("--start",
                        help=("Processing start date"))
    parser.add_argument("--data-root",
                        default='.',
                        help="Output data directory")
    opt = parser.parse_args(args)
    return opt

def write_events(start, stop):
    print('Using events file {}'.format(kadi.paths.EVENTS_DB_PATH()))

    for attr in dir(kadi.events):
        evt = getattr(kadi.events, attr)
        if type(evt) is not kadi.events.EventQuery:
            continue

        # These event types are frequently updated after ingest.  See
        # https://github.com/sot/kadi/issues/85.  For now just ignore.
        if attr in ('caps', 'dsn_comms', 'major_events'):
            continue

        dat = evt.filter(start, stop).table
        if attr != 'obsid' and 'obsid' in dat.colnames:
            del dat['obsid']

        filename = os.path.join(opt.data_root, attr + '.ecsv')
        print('Writing event {}'.format(filename))
        dat.write(filename, format='ascii.ecsv')

def write_cmds(start, stop):
    print('Using commands file {}'.format(kadi.paths.IDX_CMDS_PATH()))
    cmds = kadi.cmds.filter(start, stop)
    out = repr(cmds)
    filename = os.path.join(opt.data_root, 'cmds.txt')
    print('Writing commands {}'.format(filename))
    with open(filename, 'w') as fh:
        fh.write(out)

if __name__ == '__main__':
    opt = get_opt()
    if not os.path.exists(opt.data_root):
        os.makedirs(opt.data_root)
    start = DateTime(opt.start)
    stop = DateTime(opt.stop)

    write_events(start, stop)
    write_cmds(start, stop)
taldcroft commented 4 years ago

Diffs in events

Using write_events_cmds.py

taldcroft commented 4 years ago

Diffs in commands

Checked only from 2019:001 to 2020:020 (through the end of JAN0620A) due to file size (about 10 Mb per year).

Only diffs were:

taldcroft commented 4 years ago

@jeanconn - this has grown into a many tentacled beast, but I believe that everything is now ready for review. I'd like to get this into a skare3 release in the next week or two if all possible to avoid having it all slide off the radar and require yet another reboot.

For your part, I think most of the review can be process and testing, though you are welcome to comment on the code if you find time/energy.

jeanconn commented 4 years ago

Am I correct in understanding that the promotion plan for this would also be that @taldcroft would update cron jobs so that events3.db3 and cmds files would be made in the skare3 processing, but events.db3 will continue to be made by the ska2 process (also run by @taldcroft )?

Probably also need an update to the checkers to check status for events3.db3

jeanconn commented 4 years ago

Also, I think that skare3 will need to package kadi as noarch: generic instead of noarch:python to get the task_schedule.cfg with the 4.3.21 version of conda. I think we determined that noarch:generic doesn't work for windows but should be ok for linux and osx until we get to shiny and/or windows.

taldcroft commented 4 years ago

Am I correct in understanding that the promotion plan for this would also be that @taldcroft would update cron jobs so that events3.db3 and cmds files would be made in the skare3 processing, but events.db3 will continue to be made by the ska2 process (also run by @taldcroft )?

Yes, exactly that.

Probably also need an update to the checkers to check status for events3.db3

Makes sense.

taldcroft commented 4 years ago

I've evolved to see value in those dependencies. :smile: