search

FIXTradingCommunity / orchestrations

Service offerings expressed with Orchestra
Apache License 2.0
10 stars 7 forks source link

Format of Orchestra XML file has changed substantially between EP269 and EP272 #30

Open outofphase opened 2 years ago

outofphase commented 2 years ago

The format of the FIX Latest Orchestra XML file has changed substantially between EP269 and EP272, see below for snippet. This has caused various issues in my build process, and there is possibly an incompatibility of this new format with the tablature tools. Before I dig into this much further, can I ask whether this change was intentional?

$ git checkout f4a5205
HEAD is now at f4a5205 Create Known Issues.md

$ head -15 FIX\ Standard/OrchestraFIXLatest.xml 
<?xml version="1.0" encoding="UTF-8"?>
<fixr:repository xmlns:dc="http://purl.org/dc/elements/1.1/"
                 xmlns:fixr="http://fixprotocol.io/2020/orchestra/repository"
                 xmlns:functx="http://www.functx.com"
                 xmlns:xs="http://www.w3.org/2001/XMLSchema"
                 name="FIX.Latest"
                 version="FIX.Latest_EP269">
   <fixr:metadata>
      <dc:title>Orchestra</dc:title>
      <dc:creator>unified2orchestra.xslt script</dc:creator>
      <dc:publisher>FIX Trading Community</dc:publisher>
      <dc:date>2021-08-14T22:38:48.950856Z</dc:date>
      <dc:format>Orchestra schema</dc:format>
      <dc:source>FIX Unified Repository</dc:source>
      <dc:rights>Copyright (c) FIX Protocol Ltd. All Rights Reserved.</dc:rights>

$ git checkout cd61eb4
HEAD is now at cd61eb4 EP272

$ head -15 FIX\ Standard/OrchestraFIXLatest.xml 
<fixr:repository xmlns:dcterms="http://purl.org/dc/terms/" xmlns:fixml="http://fixprotocol.io/2022/orchestra/appinfo/fixml" xmlns:fixr="http://fixprotocol.io/2020/orchestra/repository" version="FIX.Latest_EP272" name="FIX.Latest">
    <fixr:metadata>
        <dcterms:title>
            FIX.Latest_EP272</dcterms:title>
        <dcterms:created>
            2022-08-15T16:15:49.035692</dcterms:created>
        <dcterms:date>
            2022-08-04T15:17:49Z</dcterms:date>
        <dcterms:rights>
            Copyright (c) FIX Protocol Ltd. All Rights Reserved.</dcterms:rights>
        <dcterms:conformsTo>
            Orchestra v1.0</dcterms:conformsTo>
        <dcterms:source>
            FIX Unified Repository 2010 Edition</dcterms:source>
    </fixr:metadata>
kleihan commented 2 years ago

@outofphase, some changes were intentional (see GTC Blog) but the reformatting of the DC terms section was not. I had also detected the issue with Tablature and added issue #67. Don is looking into that.

Orchestra XML is now generated from Unified with Python instead of XSLT.

donmendelson commented 2 years ago

I fix has already been committed for the extra newline in the metadata section. (Trims leading and trailing whitespace from Unified.)

outofphase commented 2 years ago

Thank you Hanno and Don. I will try to dig a bit deeper into the issues that I found with Tablature then.

These are all with elaboration sections that span multiple lines. I think it has to do with duplication of data between documentation sections with purpose SYNOPSIS and with purpose ELABORATION.

The end result is that I can't now, with the new XML format, get a stable roundtrip: md -> xml -> md. A new set of documentation lines gets generated in the process. The new XML format looks better, but it is creating some strange results for me which I need to dig into.

In EP269 we had:

         <fixr:code name="ExecutionReport" id="35009" value="8" sort="9">
            <fixr:annotation>
               <fixr:documentation purpose="SYNOPSIS">
         ExecutionReport
      </fixr:documentation>
               <fixr:documentation purpose="ELABORATION">
         The execution report message is used to:
         1. confirm the receipt of an order
         2. confirm changes to an existing order (i.e. accept cancel and replace requests)
         3. relay order status information
         4. relay fill information on working orders
         5. relay fill information on tradeable or restricted tradeable quotes
         6. reject orders
         7. report post-trade fees calculations associated with a trade
      </fixr:documentation>
            </fixr:annotation>
         </fixr:code>

followed by:

            <fixr:documentation purpose="SYNOPSIS">
         The execution report message is used to:
         1. confirm the receipt of an order
         2. confirm changes to an existing order (i.e. accept cancel and replace requests)
         3. relay order status information
         4. relay fill information on working orders
         5. relay fill information on tradeable or restricted tradeable quotes
         6. reject orders
         7. report post-trade fees calculations associated with a trade
      </fixr:documentation>

Now in EP272 we have:

            <fixr:code name="ExecutionReport" id="35009" value="8" sort="9">
                <fixr:annotation>
                    <fixr:documentation purpose="SYNOPSIS">
                        ExecutionReport</fixr:documentation>
                    <fixr:documentation purpose="ELABORATION">
                        The execution report message is used to:</fixr:documentation>
                    <fixr:documentation purpose="ELABORATION">
                        1. confirm the receipt of an order</fixr:documentation>
                    <fixr:documentation purpose="ELABORATION">
                        2. confirm changes to an existing order (i.e. accept cancel and replace requests)</fixr:documentation>
                    <fixr:documentation purpose="ELABORATION">
                        3. relay order status information</fixr:documentation>
                    <fixr:documentation purpose="ELABORATION">
                        4. relay fill information on working orders</fixr:documentation>
                    <fixr:documentation purpose="ELABORATION">
                        5. relay fill information on tradeable or restricted tradeable quotes</fixr:documentation>
                    <fixr:documentation purpose="ELABORATION">
                        6. reject orders</fixr:documentation>
                    <fixr:documentation purpose="ELABORATION">
                        7. report post-trade fees calculations associated with a trade</fixr:documentation>
                </fixr:annotation>
            </fixr:code>

followed by:

            <fixr:annotation>
                <fixr:documentation purpose="SYNOPSIS">
                    The execution report message is used to:</fixr:documentation>
                <fixr:documentation purpose="SYNOPSIS">
                    1. confirm the receipt of an order</fixr:documentation>
                <fixr:documentation purpose="SYNOPSIS">
                    2. confirm changes to an existing order (i.e. accept cancel and replace requests)</fixr:documentation>
                <fixr:documentation purpose="SYNOPSIS">
                    3. relay order status information</fixr:documentation>
                <fixr:documentation purpose="SYNOPSIS">
                    4. relay fill information on working orders</fixr:documentation>
                <fixr:documentation purpose="SYNOPSIS">
                    5. relay fill information on tradeable or restricted tradeable quotes</fixr:documentation>
                <fixr:documentation purpose="SYNOPSIS">
                    6. reject orders</fixr:documentation>
                <fixr:documentation purpose="SYNOPSIS">
                    7. report post-trade fees calculations associated with a trade</fixr:documentation>
            </fixr:annotation>
kleihan commented 2 years ago

@outofphase you are quoting two different sections of the Orchestra XML file, there is no duplication:

  1. Synopsis and elaboration for MsgType(35)= 8 (ExecutionReport) of code set MsgTypeCodeSet
  2. Synopsis (numbered list not in a separate elaboration) for message ExecutionReport(35=8)

The second snippet you quote above is missing the beginning for which it is an annotation (always comes at the end of a message definition):

        <fixr:message name="ExecutionReport" id="9" category="SingleGeneralOrderHandling" added="FIX.2.7" abbrName="ExecRpt" msgType="8">

The extra newline issue remains and is being looked into by @donmendelson.

donmendelson commented 2 years ago

In the Unified Repository phrases file, a <phrase> element can contain any number of <para> elements, as in the case of the numbered list mentioned by @outofphase. In the Orchestra schema, an <annotation> can contain any number of <documentation> elements (as well as <appinfo>). The logic of the Python orchestra-transposer script is to turn each <para> into a <documentation> while the XSLT just joined the paragraphs with newlines. A rationale for the Python method is that we are not restricted to storing documentation as plain text in Orchestra, and different markups have different paragraph notations. In fact the preferred documentation format in the future will be markdown, but other formats are possible, e.g. <p> for paragraph in HTML. A tool can render the format according to the contentType attribute that stores the media type (formerly MIME). The default value is "text/plain".

To control format, it would useful to turn those numbered lists into markdown in Orchestra when we are no longer dependent on conversion from Unified.

kleihan commented 2 years ago

Tablature behavior needs to be tested:

  1. Create Test.md as a small markdown file containing ER message with a few fields
  2. Use md2orchestra with FIX Latest as reference file to create Orchestra file Test.xml
  3. Check Test.xml for the ER annotation
  4. Use orchestra2md to convert Test.xml to a second markdown file Test2.md
  5. Check Test2.md for the ER annotation
  6. Post results of steps 3 and 5 in this thread
JThoennes commented 2 years ago

I also found that the initial <?xml version="1.0" encoding="UTF-8"?> is missing. And the xmlns:fixml reference xmlns:fixml="http://fixprotocol.io/2022/orchestra/appinfo/fixml" points to a not-existing page.

donmendelson commented 2 years ago

@JThoennes you are correct that the appinfo/fixml schema has not yet been made public. It is used internally to mark elements for FIXML schema generation. The appinfo element is extensible with

<xs:any processContents="lax"/>

The lax processing is supposed to inform an XML processor that if the schema cannot be found, no errors should be reported. Your experience may be different.

The lack of xml declaration is an error; we will look into it.

kleihan commented 2 years ago

And the xmlns:fixml reference xmlns:fixml="http://fixprotocol.io/2022/orchestra/appinfo/fixml" points to a not-existing page.

@JThoennes the XML namespace reference is not a URL, but a URI to establish a unique identifier for the prefix. We simply chose the resource name to look like a URL.

JThoennes commented 2 years ago

@kleihan My IntelliJ IDE marks this URI as an error because it cannot be downloaded. I will try to register it locally. I currently assume that the IDE has a good understanding of XML and report any findings here.