Closed taldcroft closed 4 years ago
Note that this is based on current kadi master.
For these kind of changes to the dev ("test"?) server, what are thoughts on how we do promotion when done with testing?
The process for transition to the new server provides a good model.
The process for transition to the new server involved one going out of service and one being promoted by DNS swap. Was that your plan for the next promotion (promote test to flight by DNS change)? Would you then start a new test machine or swap flight to test and then update it in place?
My first choice is to promote this, along with Django 2.2 to Ska3 flight before doing work on the test server. That if course needs a different sort if testing. And there might be problems in that strategy with legacy ska access. Will need to think about that...
I was talking about the testing process. The promotion would be simply test on test then install the same update to production aka flight. Very simple, flight down only long enough to restart the server.
@taldcroft There's a bunch here. Do you want to update the PR top-text to talk about what you've already done for testing and anything specific you think reviewers should think about while reviewing?
Haven't requested review just yet, so stand by. :smile:
Sorry, got excited.
Yes, it's good stuff and I think almost there. 🤞 There is a fair bit of testing I have done that I will document in the top-matter.
And BTW django 3.0 was just release a couple weeks ago (and 3.0.1 I think). From my first test it looks to just work here at least on Mac so will go with that if possible.
For the record here is the script used for checking that events and commands are the same. This is a one-off modification of a similar script in ska_testr so not CM'ing it.
(ska3-django3) kady$ cat validate/write_events_cmds.py
#!/usr/bin/env python
import os
import argparse
import kadi.events
import kadi.cmds
import kadi.paths
from Chandra.Time import DateTime
def get_opt(args=None):
parser = argparse.ArgumentParser(description='Write commands and events ')
parser.add_argument("--stop",
help="Processing stop date")
parser.add_argument("--start",
help=("Processing start date"))
parser.add_argument("--data-root",
default='.',
help="Output data directory")
opt = parser.parse_args(args)
return opt
def write_events(start, stop):
print('Using events file {}'.format(kadi.paths.EVENTS_DB_PATH()))
for attr in dir(kadi.events):
evt = getattr(kadi.events, attr)
if type(evt) is not kadi.events.EventQuery:
continue
# These event types are frequently updated after ingest. See
# https://github.com/sot/kadi/issues/85. For now just ignore.
if attr in ('caps', 'dsn_comms', 'major_events'):
continue
dat = evt.filter(start, stop).table
if attr != 'obsid' and 'obsid' in dat.colnames:
del dat['obsid']
filename = os.path.join(opt.data_root, attr + '.ecsv')
print('Writing event {}'.format(filename))
dat.write(filename, format='ascii.ecsv')
def write_cmds(start, stop):
print('Using commands file {}'.format(kadi.paths.IDX_CMDS_PATH()))
cmds = kadi.cmds.filter(start, stop)
out = repr(cmds)
filename = os.path.join(opt.data_root, 'cmds.txt')
print('Writing commands {}'.format(filename))
with open(filename, 'w') as fh:
fh.write(out)
if __name__ == '__main__':
opt = get_opt()
if not os.path.exists(opt.data_root):
os.makedirs(opt.data_root)
start = DateTime(opt.start)
stop = DateTime(opt.stop)
write_events(start, stop)
write_cmds(start, stop)
Using write_events_cmds.py
Checked only from 2019:001 to 2020:020 (through the end of JAN0620A) due to file size (about 10 Mb per year).
Only diffs were:
@jeanconn - this has grown into a many tentacled beast, but I believe that everything is now ready for review. I'd like to get this into a skare3 release in the next week or two if all possible to avoid having it all slide off the radar and require yet another reboot.
For your part, I think most of the review can be process and testing, though you are welcome to comment on the code if you find time/energy.
Am I correct in understanding that the promotion plan for this would also be that @taldcroft would update cron jobs so that events3.db3 and cmds files would be made in the skare3 processing, but events.db3 will continue to be made by the ska2 process (also run by @taldcroft )?
Probably also need an update to the checkers to check status for events3.db3
Also, I think that skare3 will need to package kadi as noarch: generic instead of noarch:python to get the task_schedule.cfg with the 4.3.21 version of conda. I think we determined that noarch:generic doesn't work for windows but should be ok for linux and osx until we get to shiny and/or windows.
Am I correct in understanding that the promotion plan for this would also be that @taldcroft would update cron jobs so that events3.db3 and cmds files would be made in the skare3 processing, but events.db3 will continue to be made by the ska2 process (also run by @taldcroft )?
Yes, exactly that.
Probably also need an update to the checkers to check status for events3.db3
Makes sense.
I've evolved to see value in those dependencies. :smile:
This is based on the scm_version branch. Features:
obsid
as a standard field for telemetry-based eventsevents.db3
toevents3.db3
. New file and schema are not compatible with previous version.Skare3 package requirements
In order to pass all documented tests these package versions or branches are required:
run-info
branch (https://github.com/sot/ska_helpers/pull/6)py3-server
branch (https://github.com/sot/find_attitude/pull/7)py3-server
branch (https://github.com/sot/mica/pull/213)modernize
branch (https://github.com/sot/task_schedule/pull/15)modernize
branch (https://github.com/sot/watch_cron_logs/pull/6)Testing
Create test environment
Roughly speaking, on kady:
Make events3.db3 and cmds.pkl, cmds.h5 files
Follow instructions in
NOTES.build
Set up cron job
Web server
Linux
Start a local server that runs the installed version of kadi, mica, find_attitude
MacOSX:
Start a local server that runs the installed version of kadi, mica, find_attitude, sherpa (4.11.1 from sherpa channel)
Fixes #145