Open kleihan opened 1 week ago
I would propose:
Presence:
Supported:
And if deprecated, we can have some extra metadata informing about the removal and some elaboration text to describe the recommended alternative. Similar to what Java does: https://www.baeldung.com/java-deprecated
@JoanPuig, defining all non-documented tags as forbidden sounds like a great simplification for subsetting scenarios, especially in cases where the source scenario is very big and the subset is very small. It does require to fully declare the subset but I think that is in the interest .
In theory and for FIX messages, everybody should be using FIX Latest (or a company-wide spec derived from FIX Latest) as initial source "scenario" (could be seen as base scenario). Using that base as reference, one can define a subset for an element (e.g. Instrument component) with a minimal set of attributes. The reference attribute (e.g. id
) is always required, name
may be used for legibility, but then you only need to add attributes to override what is pulled in from base (Tablature works this way), e.g. a change of the presence attribute from optional
to required
.
@JoanPuig, I support the proposal to change the supported
attribute to being either supported
or deprecated
as it would remove the disambiguity and solve another problem. Today, deprecation is only implicitly given by the presence of the attributes deprecated
and deprecatedEP
that contain a version and an EP number. The problem is that you are required to use pedigree information to express deprecation. Making this explicit as supported=deprecated
decouples that from the pedigree.
We already have on attribute to point to the alternative, i.e. replacedByField
. This may not be enough for all cases (not always a 1:1 replacement) but is a good start (FIX Latest currently does not use it). The elaboration text could be another new purpose (e.g. DEPRECATED) for the documentation of an element (see #229).
@JoanPuig, I agree that the value constant
for the presence
attribute should be discussed and possibly removed. This is how it is today:
<fixr:fieldRef id="22" name="SecurityIDSource" presence="constant" value="1"/>
Would using a literal type require to override the type
attribute? We should keep the field datatype to allow validation of the literal defined in Orchestra. The field could also have a code set and the constant must match one of its values.
Using a rule is probably already possible with Orchestra v1.0 with assign
(not 100% if this is correct, spec only uses it to assign values from incoming messages to response messages).
<fixr:fieldRef id="22" name="SecurityIDSource">
<fixr:assign>1</fixr:assign>
</fixr:fieldRef>
How about removing presence=constant
and defining the use of presence=ignored
for constants (presence of value
implies a constant) instead? It would mean that the message may contain the field with a value but that the value will be ignored and overwritten with the value defined in the Orchestra XML file.
<fixr:fieldRef id="22" name="SecurityIDSource" presence="ignored" value="1"/>
It also seems better suited for automated generation of specs out of Orchestra XML files compared to the use of a rule.
My comments were not related to scenario inheritance. That is part of the problem I am trying to highlight, overloading those concepts for inheritance will be problematic down the road. The current definition is already a bit ambiguous.
On a given message scenario (whatever way we have defined the inheritance, not important for this) there will be fields with different types. For each field, I would like to be able to express the following: 1) is the field of type T or of type Optional[T] 2) What are the possible values of the field? Normally that can be solved with a validation rule, but, do we want a short hand notation for the "constant" case which will be common?
There are two other important concepts for a field: 1) Is the field tolerated in the message but it will be ignored by the system. Which can be described as a Optional[T] with no validation rules, but further, we tag it with this extra concept to let implementers know it will not be used. 2) Support / deprecation status of the field
Related to the supported status I would envision that from release to release a field would move through the following states: supported -> deprecated but still used -> deprecated and not used (tolerated in the message) -> not in the spec, therefore error if field in message
@JoanPuig can you help me understand your use of "deprecated" in your last comment. Is this standard FIX fields that have been deprecated by the standard, or fields marked as depreciated in your implementation spec? Just trying to understand your context.
@JoanPuig agree with the flavors of deprecation as many spec issuers support the old fields prior to actual deprecation. I think we need to take into account that there are incoming and outgoing messages from the perspective of the issuer of a spec. The concept of tolerating a field only applies to incoming messages, only the issuer can "tolerate". The outgoing message may use the term "deprecated", but only in the sense of "still used". When the issuer decides to no longer use a field in an outgoing message, it should simply be removed.
Here is a proposal to cover the state changes:
The presence
attribute is relevant except for "deprecated" where it should be "ignored" (incoming) or "optional" (outgoing). The presence attribute ignored
does not really apply to outgoing messages, or? presence=required
on outgoing messages simply means that it is always present. It is debatable whether fields that are to be deprecated could still be required or would always be optional. Is there a use case for this?
This is a table with the combinations that can occur (rows for supported
and columns for presence
):
Incoming | Optional | Required | Ignored |
---|---|---|---|
Supported | X | X | X |
ToBeDeprecated | X | X | X |
Deprecated | N/A | N/A | X |
Outgoing | Optional | Required | Ignored |
---|---|---|---|
Supported | X | X | N/A |
ToBeDeprecated | X | X | N/A |
Deprecated | X | N/A | N/A |
@ltaikit the concept applies both to the FIX standard and specs that a service provider would produce. The idea is to be able to warn consumers that something they are using will change in the future in a machine readable way. If as a consumer my implementation is generating messages with fields tagged as ToBeDeprecated (using @kleihan nomenclature), I would want the FIX engine to issue a warning. Similarly, if an implementation is reading a field from an incoming message that is tagged as ToBeDeprecated, I would want the FIX engine to issue a warning.
@kleihan I agree with the analysis of the different deprecation states and that they depend on direction.
I have checked the use case for presence=forbidden
in RC1 (no change from V1.0). This example is in the spec and uses presence=forbidden
as part of a rule for conditional presence and not to disallow a field entirely.
<fixr:fieldRef id="99" name="StopPx" presence="optional">
<fixr:rule name="StopOrderRequiresStopPx" presence="required">
<fixr:when>OrdType == ^Stop</fixr:when>
</fixr:rule>
<fixr:rule name="LimitOrderForbidsStopPx" presence="forbidden">
<fixr:when>OrdType \!= ^Stop</fixr:when>
</fixr:rule>
</fixr:fieldRef>
The Orchestra examples use supported=forbidden
to exclude codes from a code set.
<fixr:code value="6" sort="6" added="FIX.2.7" supported="forbidden" id="40006" name="WithOrWithout">
<fixr:annotation>
<fixr:documentation purpose="SYNOPSIS">With Or Without/>
</fixr:annotation>
</fixr:code>
How would we express these use cases if we remove forbidden
as value from both attributes? The second example is easier to solve as one would simply not define the code, which implies it is forbidden (see proposal from @JoanPuig above). The first case is harder as conditional rules are an important part of the semantics. Would it suffice to keep forbidden
for the presence
attribute and use the spec to limit its use to rules? It would then not
It seems that ambiguity can also come from things that are valid as per the schema but not intended to be used that way. The schema may become overly complex if we try to enforce the intended usages. I see this as a question to answer for the discussion #235. How far should the role of the schema go to enforce the validity of an Orchestra XML file? Do we need a compatibility checker that goes beyond the validation against the schema, e.g. to detect the usage of presence=forbidden
outside of a rule?
Such a validator sits between the one only looking at the schema and the one that is specific to an encoding such as tag=value where we want to ensure additional things like naming conventions and field ordering. The definition of the term "valid" requires a defined context (schema, Orchestra semantics, encoding,...). These thoughts should feed into the necessary tooling that can be developed as a community.
Orchestra provides two similar attributes (
presence
,supported
) with partially overlapping values.The attribute presence is part of the group, component and field references as well as the field definition (to define a constant value). The attribute
supported
is part ofentityAttribGrp
and not described in the specification.It is proposed to clarify the difference between the two and/or consolidate overlapping values. This is important to identify a potential use of one or the other to exclude specific elements from a scenario that is related to another scenario (see #212, #228).