Added maude decom for assembling images

javierggt commented 5 years ago

This is still a preliminary work on implementing ACA image fetching from Maude.

Frame-based fetch This is the current implementation in this pull request. It fetches VCDU frames from maude, locates ACA fields and decommutes the ACA 225-byte frames.

We could consider refactoring the MSID collection

taldcroft commented 5 years ago

For reference, here are the MSIDs at the beginning of each minor frame. Using the minor frame counter CVCMNCTR which cycles 0..127 should be a good bet.

One caveat is that it is always possible that a particular readout of any MSID will simply be missing. That happens commonly in realtime telemetry, and very rarely (but still sometimes happens) in dump data. If the minor frame count is out of sequence then one can assume loss of the VCDU ("VCDU" is a 0.25625 sec chunk of 1025 bytes of telemetry corresponding to that minor frame) and just reset everything and wait for the next minor_frame % 4 == 0 to get going again.

There should be one option to require "perfect" data for a fully-reconstructed image (plus the other related ACA image telemetry values) in order to output a record. From there one can imagine more fault-tolerant behavior like putting in NaN for missing values in the image, but it can get complicated, e.g. if the scale factor is missing. For the most part the dump data is almost always perfect and for real-time you don't mind missing the entire image occasionally, so requiring "perfect" data might be a fine solution.

<Table length=18>
  MSID   START_MINOR_FRAME START_WORD START_BIT LENGTH      technical_name     
 str15         int64         int64      int64   int64           str24          
-------- ----------------- ---------- --------- ------ ------------------------
CCSDSVER                 0          1         0      2  CCSDSVER VERSION NUMBER
   XADD1                 0          1         0      8     VCDU HEADER 1ST OF 6
 CCSDSID                 0          1         2      8            SPACECRAFT ID
   XADD2                 0          2         0      8     VCDU HEADER 2ND OF 6
  CVCCTU                 0          2         2      1            CVCCTU FILLER
CCSDSFMT                 0          2         3      1                 ACA FLAG
CCSDSPBK                 0          2         4      1            PLAYBACK TYPE
CCSDSTMF                 0          2         5      3      TELEMETRY FORMAT ID
  XVCDU1                 0          3         0      8     VCDU HEADER 3RD OF 6
CCSDSVCD                 0          3         0     24 VCDU COUNTER -- CCSDSVCD
CVCDUCTR                 0          3         0     24 VCDU COUNTER -- CVCDUCTR
CVCMJCTR                 0          3         0     17 VCDU MAJOR FRAME COUNTER
  XVCDU2                 0          4         0      8     VCDU HEADER 4TH OF 6
  XVCDU3                 0          5         0      8     VCDU HEADER 5TH OF 6
CVCMNCTR                 0          5         1      7 VCDU MINOR FRAME COUNTER
XREPLAYF                 0          6         0      8     VCDU HEADER 6TH OF 6
CCSDSREP                 0          6         0      1              REPLAY FLAG
 CVCDSPR                 0          6         1      7           CVCDSPR FILLER

javierggt commented 5 years ago

I do not know how I should see if any MSID is missing, other than checking the time interval between them or by checking the consistency with other MSIDs.

In this function one does not see the stream. Basically, maude does the decom, so when I request an MSID, I get a sequence of values and times. I don't know what I'm supposed to see when an MSID is missing, but I am assuming it is just a missing entry in the sequence. I think that would be a question for someone who knows better.

Assuming what would happen is a missing value in the sequence, If I use CVCMNCTR as the reference, and a pixel MSID is missing, the pixel value returned by this function will be nan at that time. If CVCMNCTR is missing, then I would see two values for the pixel data between CVCMNCTR values. I have not considered this case.

taldcroft commented 5 years ago

I'm 98% sure that missing data will just look like no returned value during the interval. E.g. for AOPCADMD you expect one sample very 1.025 seconds, but instead during some of those intervals you might see zero returned values. Looking at the Interface docs (https://occweb.cfa.harvard.edu/twiki/bin/view/Software/MAUDE/WebHome) there is no mention of missing data or flagging data as bad.

jeanconn commented 5 years ago

I agree with @taldcroft that it seems to me that missing data will just look like no returned value, but I don't quite understand the docs with regard to the description of "flags" for the single msid case. I'm somewhat assuming that those are already fully parsed in maude for the items like "subset" and "tolerance".

javierggt commented 5 years ago

Yes, that is what I meant by "a missing entry in the sequence."

I think Maude does nothing for missing MSIDs. If they are missing, they just do not appear in my result. After a query, I then have to match the time of each MSID update to the corresponding minor frame header.

I do not think the 'tolerance' option helps me much. This is because when one specifies a tolerance, the result includes only cases when all secondary MSIDs are updated at the time of the primary MSID (within tolerance). I would then have to group MSIDs in the right way, according to whether they are expected together or not, and if one is missing then the others will be missing too. This gets complicated by the fact that differently sized images have segments that update at different rates. It would help if the tolerance was interpreted with any instead of all, but it would not solve all issues.

So... I am setting the time to be the time of the immediately preceding minor frame counter and:

if I see two entries at the same time, something went wrong. I will throw an exception (this would happen if there is a minor frame update missing). Maybe there are other errors that can arise from a minor frame update missing, but does this ever happen?
by default, and MSID will have a value of NaN if there is no value at a given time. This deals with the case of other missing MSIDs.

javierggt commented 5 years ago

to clarify a bit more: the time I am setting is the time of the minor frame when the image size of the first segment of the image is updated.

(I have updated this a couple of times to make sure it is right)

taldcroft commented 5 years ago

by default, an MSID will have a value of NaN if there is no value at a given time. This deals with the case of other missing MSIDs.

In general you won't be able to use NaN as a missing-value sentinel because the MSIDs can be int and str type. Since the container is an astropy Table you could you use a masked table (with masked column for everything) but this can get a bit ugly.

javierggt commented 5 years ago

well, strings could be set to the 'NaN' string. I don't know about integers.

taldcroft commented 5 years ago

if I see two entries at the same time, something went wrong. I will throw an exception (this would happen if there is a minor frame update missing).

We should think about the default behavior being fault tolerant, as in not intentionally raising exceptions. Basically we expect that telemetry may toss weirdness at us, and this module should protect the calling application from that messiness to the extent possible. One doesn't want to have to wrap every call to this decom in a try/except by default.

Of course one might imagine a strict mode that raises exceptions for the caller to handle. In practice I suspect that would be rarely used, and something slightly more useful would probably be something like a log of problems in the telemetry.

taldcroft commented 5 years ago

well, strings could be set to the 'NaN' string. I don't know about integers.

For int values you are pretty much stuck (esp. for ones that are like uint8).

taldcroft commented 5 years ago

One big picture comment is to keep in mind that we have two somewhat distinct requirements:

Support realtime streaming with image updates each 1.025 sec
Support archival telemetry query (i.e. anything not in realtime)

Realtime streaming should allow the following (more-or-less):

Provide the most recent complete image(s) for a slot. There may multiple images since the last query if the viewing app got paused. There may need to be some concept of popping images off the object, or just clearing the stack. Not totally clear though.
Optional but nice: provide the partially assembled image and associated telemetry for a slot, i.e. filling in values as they become available each 1.025 sec. Remember an 8x8 will take 4.1 sec to complete, but there is valid data before hand. Sometimes you would like to see image updates as a discrete update "here is the entire image and telemetry", and sometimes you want to see updates dribbling in the second they become available.
Indicate if data are stale / missing, e.g. one expects to have fresh data at this time but it is not available.

Another important thing to remember is that the timing of different slots is essentially asynchronous, so any slot can start at any 1.025 sec boundary (so for 6x6 or 8x8 they can be out of sync). This really drives the design here to focus on an object that is responsible for just one slot.

In addition to the slot-based telemetry there is also global (non-slot) telemetry that needs to be captured and displayed, so that is probably a separate class object.

javierggt commented 5 years ago

Yes, I have had most of those things in mind to some degree. The one exception is missing values, because that has yet to be specified. I think this need more thought.

The functions in this module are strictly for fetching and assembling the results. Some of the things you mention are handled downstream. For a real-time system, I thought there would be a separate class that repeatedly fetches data and caches it. In this way you can update when the image is complete, or with partial images. My idea then was to have a separate class in aca_view that does this asynchronous collection, and emits signals when new data is available.

For archival telemetry, I can already tell you that fetching is slow.

Only recently I started thinking that:

I might want to split everything by slot anyway. Note that the only 'synchronicity' imposed to different slots comes from the minor frame times. I have not seen data from different slots completely out of phase, but I will see if I find some to add some test.
I might need a separate container for non-slot MSIDs. There is another puzzling thing: There seem to be MSIDs for AC housing temperature, CCD temperature, primary/secondary temperature, all per slot. Isn't that a global number?

jeanconn commented 5 years ago

Regarding the temperatures, I think most of those are just copies across the slots, and in the viewer we may just want AACCCDPT from non-ACA telemetry instead.

There is some extended/non-copy data via some of the 8x8 data (see Appendix A in the ACA User Guide). Mostly that just means, for example, that a ccd temperature with smaller quantization is available if we are getting 8x8 data on slot 6.

jeanconn commented 5 years ago

Though as far as I know, the Appendix A data I mention is not available via msids in MAUDE.

javierggt commented 4 years ago

I just pushed the changes we discussed yesterday.

You can also see:

current docs here: http://cxc.cfa.harvard.edu/mta/ASPECT/tmp/chandra_aca_pr81/docs/
coverage report here: http://cxc.cfa.harvard.edu/mta/ASPECT/tmp/chandra_aca_pr81/coverage/

javierggt commented 4 years ago

I just want to point out, before this is approved, that the pixel arrangement in the docstring is the transpose of the one in the PIXEL_MAP variable.

The one in the docstring is what is shown in the ACA User Guide, figures 1-8 and 1-9 (rotated so columns increase to going the right and rows increase going up).

The one in PIXEL_MAP is the one actually used. The result agrees with images in level0 mica files. I do not see where I could be transposing it.

jeanconn commented 4 years ago

Thanks @javierggt . For our future selves, I think that both representations are "right" and the transposition is just a product of different ways of indexing into / describing the matrix.. but we should probably write that down in the docstring too (dig out or describe convention in the ACA User Guide?) and either explicitly note the mismatch in the docstring or capture both versions in the docstring.

javierggt commented 4 years ago

I added the glob to setup.py. This would be it for the files, right?

jeanconn commented 4 years ago

It looks like there is some code that starts to decom the image status bits, but I don't see the bits (or the combined IMGSTAT) in the maude table.

javierggt commented 4 years ago

I don't think I have added the decom of the status bits. I did add the decom of different headers.

jeanconn commented 4 years ago

OK. I added a new issue #84 to for the status bits.

sot / chandra_aca

Added maude decom for assembling images #81