Proper support for archiveFields in policy files.

[slacmshankar]

In the policy file, if you specify fields of a PV in the archiveFields but do not include these fields in the getFieldsArchivedAsPartOfStream then the archiver does not archive these fields correctly.

Note this does NOT apply to folks who use the polices file shipped with the archiver - (SLAC).

For example, if you archive many fields of the MOTOR record using

        elif pvRTYP=="motor":
            archiveFields=['DVAL', 'OFF', 'DIR', 'DHLM', 'DLLM', 'TWV', 'VBAS', 'VELO', 'ACCL', 'BDST', 'BVEL', 'BACC', 'RDBD', 'RTRY', 'UEIP', 'URIP', 'DLY', 'DISA', 'DISP', 'FOFF', 'FRAC', 'OMSL', 'JVEL', 'JAR', 'HVEL', 'ATHM', 'CDIR', 'DIFF', 'DMOV', 'DRBV', 'ERES', 'FOF', 'HLM', 'HLS', 'HOMF', 'HOMR', 'HOPR', 'JOGF', 'JOGR', 'LLM', 'LLS', 'LOPR', 'LVIO', 'MISS', 'MOVN', 'MRES', 'MSTA', 'NTM', 'NTMF', 'RBV', 'RCNT', 'RDIF', 'REP', 'RHLS', 'RLLS', 'RRBV', 'RRES', 'RVAL', 'RVEL', 'S', 'SBAK', 'SBAS', 'SET', 'SMAX', 'SPMG', 'SREV', 'STOP', 'SYNC', 'TDIR', 'TWF', 'TWR', 'UREV', 'VMAX', 'VOF']

but do not include the these fields are part of the getFieldsArchivedAsPartOfStream there is inconsistent behavior.

def getFieldsArchivedAsPartOfStream():
     return ['HIHI','HIGH','LOW','LOLO','LOPR','HOPR','DRVH','DRVL'];

The archiveFields was meant to record fields that change infrequently and as part of .VAL changes. If the field changes more frequently than the .VAL; then the archiver stores the field with the .VAL change and will miss storing many changes. This is especially true of the .RBV for a motor PV. At SLAC, we typically store these fields outside the main .VAL PV using separate PVTypeInfo's. However, the above policy snippet can be interpreted as 1) When archiving a MOTOR PV, archive all these other fields as well. 2) For all these other fields, those that are to be to stored as part of the .VAL stream, use our current archiveFields (parent->child relationship between the EPICS_V3_PV). 3) For all these other fields, those that are NOT stored as part of the .VAL stream, convert them into separate PVTypeInfos.

The correct solution is to treat the archiveFields from the policies files as Eric interpreted it. That is, for all MOTOR records, also archive these extra fields. However, only those fields that are specified in getFieldsArchivedAsPartOfStream should be part of the main PV and stored with the .VAL. All other fields should generate PVTypeInfo's of their own and be archived separately...

We do attach to these fields using camonitors but since we store them as part of the .VAL field, not all changes in these fields get stored in the db. For example, when setting a MOTOR's VAL, the RBV changes slowly upto the VAL. However, the archiver does not store these changes as it stores the one with the .VAL change.

So this is principally a archive PV workflow fix. During the workflow, only those archiveFields that are in getFieldsArchivedAsPartOfStream are added to the extraFields in the .VAL's PVTypeInfo. All others should be converted into PVTypeInfo's of their own/or added to the archive workflow as separate entities to be eventually converted into PVTypeInfo's of their own.

We should also have a migration path for existing installations.

One approach is to have two migration BPL methods. 1) Determine those PVs/fields that are affected. This is mainly walking thru all the PVTypeInfo's and looking at archiveFields that are not part of getFieldsArchivedAsPartOfStream. 2) The other takes a PV and then creates PVTypeInfo's for those fields that are not part of getFieldsArchivedAsPartOfStream /or adds them to the archive PV workflow to be eventually converted into PVTypeInfo's of their own.

The other approach is to do this on startup; perhaps if a archappl.properties property is set.

[berryma4]

Mentioning a simple migration approach; upgrade to new war files, delete affected PVs from archive (do not delete data), re-add PVs to archive.

[slacmshankar]

Generated a pre-release; please let me know if you run into any issues.