MIDI data XML for an event symbol (W3C Proposal)

Edit (13.04.2018): On 05.04.2018, I opened a related issue on the W3C CG Github site: https://github.com/w3c/mnx/issues/95. https://github.com/w3c/mnx/issues/95#issuecomment-380403933 provides the frame for including the <score:midi> definition below, which I will presumably soon be copying to the W3C issue. For <score:midi> below (which is defined inside a unique event) read <midi-msgs event-id="event1.2.3">. In the W3C GC version:

The <temporal-information> element is defined separately from the graphics, but with event-ids pointing into the graphics. This is better since it allows for repeat signs in the notation. An event symbol has a different MIDI definition each time it is repeated.
The <temporal-information> element does not require a namespace, since it is part of a proposed new file format (not SVG). However, the scheme can also be used in SVG, in which case the <temporal-information> element would be namespaced and at the top level of the file (after <defs>): <score:temporal-information>.

Original text: In symbolic music notations, an event symbol is just a graphical object that has a temporal meaning. All such notations have event symbols. That includes CWMN, tablatures, all kinds of neumes, Asian notations that read vertically, Moritz' notation and probably more.

I would like to propose that the W3C Music Notation Contact Group agree on a standard definition of an XML element that describes the temporal meaning of any event symbol in terms of milliseconds and MIDI messages. This definition must be strictly independent of the symbol's graphics so that no conflicts arise. Such an element could be included in any XML that contains event symbols -- including the proposed MNX standard, MEI and the SVG written by Moritz. (I don't know if MusicXML has, or can have, event symbols.) In the MNX proposal, CWMN event symbols (<event>s) are chords, notes and rests.

Moritz currently writes SVG containing <score:midi> elements that meet this description, and I'd like to use their definition as the starting point for this discussion. In SVG, the proposed element has to be prefixed by a namespace (score in this case). Other XMLs could just use <midi>. For the purposes of this proposal, I'll include the namespace and call the proposed element <score:midi>.

More than one layer of <score:midi> elements could be included in a score's event symbols. Each layer would contain the definition of a particular performance of the score. A default, metronomic performance could be saved in just one of these layers.

The <score:midi> definition below is used by the latest versions of my authoring and client software (Moritz and the Assistant Performer). Maybe someone could help by translating this element definition into a DTD or schema. I don't yet know how to do that in a way that would enable the definition to be included in other DTDs or schemas, or as an extension to SVG.

The proposed `<score:midi>` element definition

Overview A <score:midi> element contains one <moments> element and zero or one <envs> elements. The <moments> element contains one or more temporally sequential <moment> elements. An <envs> element contains one or more <env> elements. A <moment> element has an msDuration attribute, denoting the duration (in milliseconds) to the following <moment>, and a list of <msg> elements describing MIDI messages that are to be sent "synchronously" but in the order they occur in the file. The <msg> elements are enclosed in the following sub-elements: <noteOffs>, <switches>, <noteOns> (see below). The sum of the <moment> durations is the duration of the <score:midi> element. A <msg> element has an m attribute that consists of a string of numbers separated by spaces. The numbers are the 8-bit values that comprise the message. The number of 8-bit values depends on the value of the first number (the status byte). An <env> element describes an envelope for a continuous controller. All <env> elements have an s (status byte) attribute, and contain one or more <vt> (vertex) elements. There are a few different types of <env>, distinguishable by the value of the status byte (see the detailed example below). Each <vt> element has an msDur attribute denoting the duration (in milliseconds) to the following <vt> element. The sum of the <vt> durations must always be equal to the sum of the <moment> durations. The client software decides the rate at which to send the continuous controller messages, calculating the values to send according to the position between the vertices.

The nesting looks like this:

<score:midi>
    <moments>
        <moment msDuration=...>
            <noteOffs>
                <msg m=... />
                <!-- more <msg>s -->
            </noteOffs>
            <switches>
                <msg m=... />
                <!-- more <msg>s -->
            </switches>
            <noteOns>
                <msg m=... />
                <!-- more <msg>s -->
            </noteOns>
        </moment>
        <!-- more <moment>s -->
    <moments>
    <envs>
        <env s=...  ...>
            <vt ... />
            <!-- more <vt>s -->
        </env>
        <!-- more <env>s -->
    </envs>
<score:midi>

The proposed definition in detail, with comments and some example values:

<!--
All event symbols, that contain one or more <score:midi> elements, contain the same number of
<score:midi> elements, regardless of their class (rest, note, chord etc).
-->    
<score:midi>
    <!--
    There must be exactly one <moments> element in a <score:midi> element.
    At run-time, messages having the same timestamp are always sent in the following order:
                moment/noteOffs, moment/switches, envs messages, moment/noteOns.
    Messages inside each of these categories will be sent in the order defined in the SVG.
    For easier debugging, Moritz writes status bytes in hexadecimal notation and both data1
    and data2 values as ordinary decimals.
    -->  
    <moments>
        <!--
        There must be one or more (sequential) <moment> elements here.
        Each <moment> has an msDuration attribute that is an integer greater than zero.
        This is the number of milliseconds between the beginning of this <moment> and the
        beginning of the next (possibly in the <score:midi> element in the following
        event symbol). The msDuration of this <score:midi> element (event symbol)
        is thus the sum of the msDurations of its <moment> elements.
        Each <msg> element defines a message as a string of space-separated (8-bit) bytes,
        that the client converts to a Uint8 array before sending it to the output device.
        Most <msg>s have three bytes, but some (e.g. patch change) only have two, and sysEx
        has an unlimited number.
        -->        
        <moment msDuration="500">
            <!--
            Each <moment> may have zero or one <noteOffs> element.
            -->
            <noteOffs>
                <!--
                One or more noteOff (or NoteOn+Velocity0) <msg> elements go here.
                -->
                <msg m="0x80 60 64" /> <!-- NoteOff+channel, note, velocity -->
                <msg m="0x80 64 64" /> <!-- NoteOff+channel, note, velocity -->
                <msg m="0x80 67 64" /> <!-- NoteOff+channel, note, velocity -->
            </noteOffs>

            <!--
            Each <moment> may have zero or one <switches> element.
            A <switches> element contains a sequence of one or more <msg> elements.
            The messages are sent in the order they appear here in the file.
            No plausibility checks are done by the client inside the <switches> element.
            Some commands and controllers may be sent from both here and the <envs>
            element (e.g. pan), but it makes no sense for this to happen inside the
            same <score:midi> element because messages sent with the same timestamp
            from the <envs> element are sent later than (i.e. override) any messages
            sent from the <switches> element.
            -->
            <switches>
                <!--
                Two simple message examples
                -->
                <msg m="0xB0 0 3" /> <!-- ControlChange+channel, bankControl, value -->
                <msg m="0xC0 14" /> <!-- PatchChange+channel, value -->

                <!-- pitchWheel deviation example
                Changing the pitchWheel deviation means sending two (or four) messages that
                use the following MIDI constants:
                        const PDS = 0; (PITCHWHEEL_DEVIATION_SELECT);
                        const RPC = 101; (REGISTERED_PARAMETER_COARSE)
                        const RPF = 100; (REGISTERED_PARAMETER_FINE);
                        const DEC = 6; (DATA_ENTRY_COARSE);
                        const DEF = 38 (DATA_ENTRY_FINE);
                Moritz currently just sets the coarse setting (semitones).
                -->
                <!-- set coarse pitchWheel deviation value to (e.g.) 12 semitones -->
                <msg m="0xB0 101 0" /> <!-- ControlChange+channel, RPC, PDS -->
                <msg m="0xB0 6 12" /> <!-- ControlChange+channel, DEC, semitones -->
                <!-- set fine pitchWheel deviation value to (e.g.) 5 cents -->
                <msg m="0xB0 100 0" /> <!-- ControlChange+channel, RPF, PDS -->
                <msg m="0xB0 38 5" /> <!-- ControlChange+channel, DEF, cents -->

                <!-- System Exclusive example
                The first byte in a sysEx message must be 0xF0, the last must be 0xF7.
                SysEx messages sent to a device having the wrong Manufacturer's ID are ignored.
                Moritz never writes system exclusive messages into a score.
                The Assistant Performer will send sysEx messages, but cannot respond to their
                return value (if any).
                All numbers in a SysEx msg must be in hexadecimal notation.
                -->
                <msg m="0xF0 0x41 0x10 0x42 0x12 0x40 0x00 0x7F 0x00 0x41 0xF7" />
            </switches>

            <!--
            Each <moment> may have zero or one <noteOns> element (rests have no <noteOns>).
            --> 
            <noteOns>
                <!--
                One or more noteOn <msg> elements must go here.
                -->                         
                <msg m="0x90 64 90" /> <!-- NoteOn+channel, note, velocity -->
                <msg m="0x90 68 100" /> <!-- NoteOn+channel, note, velocity -->
                <msg m="0x90 71 127" /> <!-- NoteOn+channel, note, velocity -->
            </noteOns>
        </moment>
    </moments>
    <!--
    There must be zero or one <envs> element in a <score:midi> element.
    (Moritz never writes an <envs> element into a rest, but other apps might. In Moritz,
    chords are never laissez vibrer...)
    Note that all controllers that can be set using an envelope can also be set using a simple
    <msg> in the above <switches> element.
    -->   
    <envs>
        <!--
        There must be one or more <env> (=envelope) elements here.
        In each <env> and nested <vt> (vertex) element, s is status, d1 is data1, d2 is data2.
        msDur is the duration in milliseconds to the following <vt> or <score:midi> element.
        The total duration of each <env> element must equal the total duration of the <moments>.
        Messages sent between those in the <vt> elements will be calculated and sent by the client
        application.
        All msDur values must be greater than zero, with one exception: If the <env> has
        more than one <vt> element, the final one may have a duration of zero milliseconds,
        so that the client can calculate the preceding intermediate messages. However, the
        client never actually sends the data values from a <vt> having msDur equal to zero.
        Note that <env> elements can only have status bytes with the following high nibbles:
            0xA (aftertouch),
            0xB (control change),
            0xD (channel pressure),
            0xE (pitch wheel).
        An <env> element will be ignored if its status byte has any of the following high nibbles:
            0x0, 0x1, 0x2, 0x3, 0x4, 0x5 0x6, 0x7, (MIDI status bytes never have these values) 
            0x8 (noteOff),
            0x9 (noteOn),
            0xC (patch change),
            0xF (system exclusive, real time or undefined)
        -->
        <!--         
        There can be zero or more Aftertouch <env> elements (for different note numbers)
        -->
        <env s="0xA0" d1="100"> <!-- Aftertouch+channel, d1=note number -->
            <!--
            There must be one or more vt elements
            d2 (the pressure amount) is in range 0..127. 
            the msDur values add up to the total duration of the moments (here 500ms)
            -->                    
            <vt d2="99" msDur ="250" />
            <vt d2="127" msDur ="249" />
            <vt d2="0" msDur ="1" />
        </env>
        <!--
        There can be zero or more CC controller env elements (for different continuous controllers).
        -->
        <env s="0xB0" d1="11"> <!-- ControlChange+channel, d1=control number -->  
            <!--
            There must be one or more <vt> elements
            d2 is in range 0..127, and is the (d2) value of the controller. 
            the msDur values add up to the total duration of the <moments> (here 500ms)
            -->  
            <vt d2="0" msDur ="100" />
            <vt d2="90" msDur ="150" />
            <vt d2="10" msDur ="249" />
            <vt d2="0" msDur ="1" />
        </env>
        <!--
        There can be zero or one channel pressure env element
        -->
        <env s="0xD0"> <!-- ChannelPressure+channel --> 
            <!-- 
            There must be one or more <vt> elements
            d1 is in range 0..127, and is the pressure amount.
            (d2 is undefined since not needed by this command) 
            The msDur values add up to the total duration of the <moments> (here 500ms)
            -->                    
            <vt d1="99" msDur ="255" />
            <vt d1="127" msDur ="45" />
            <vt d1="0" msDur ="200" />
        </env>
        <!-- 
        There can be zero or one pitch wheel env element
        -->
        <env s="0xE0"> <!-- PitchWheel+channel -->
            <!--
            There must be one or more <vt> elements
            d1 and d2 are in range 0..127, and combine to be the setting of the pitch wheel. 
            the msDur values add up to the total duration of the <moments> (here 500ms)
            -->         
            <vt d1="99" d2="99" msDur ="120" />
            <vt d1="127" d2="127" msDur ="379" />
            <vt d1="64" d2="64" msDur ="1" /> <!-- 64 is default, centre value -->
        </env>
    </envs>
</score:midi>

Messages sent by the client application The client application must send AllSoundOff and AllControllersOff messages to each channel at the end of each performance (partial or otherwise). If the sound is supposed to die away at the very end of the score, then a sufficiently long moment must be defined there. When beginning to play after the beginning of the score, the client must:

Send AllSoundOff and AllControllersOff messages to each channel
Set the non-default values of any controllers, patches etc. after finding the most recent value to which they were set before the start position.
Start performing.

Example of a <score:midi> element containing four moments and an env: The total duration of the moments is 250 + 366 + 250 + 368 = 1234 (ms). The total duration of the envelope is 616 + 617 + 1 = 1234 (ms). Note that the authoring software could write comments into the file to make it more readable by humans:

<score:midi>
    <moments>
        <moment msDuration="250">
            <switches>
                <msg m="0xB0 0 0" />
                <msg m="0xC0 24" />
                <msg m="0xB0 101 0" />
                <msg m="0xB0 6 3" />
            </switches>
            <noteOns>
                <msg m="0x90 61 127" />
                <msg m="0x90 73 127" />
            </noteOns>
        </moment>
        <moment
            msDuration="366">
            <noteOffs>
                <msg m="0x80 61 64" />
                <msg m="0x80 73 64" />
            </noteOffs>
            <noteOns>
                <msg m="0x90 61 127" />
                <msg m="0x90 73 127" />
            </noteOns>
        </moment>
        <moment
            msDuration="250">
            <noteOffs>
                <msg m="0x80 61 64" />
                <msg m="0x80 73 64" />
            </noteOffs>
            <noteOns>
                <msg m="0x90 61 127" />
                <msg m="0x90 73 127" />
            </noteOns>
        </moment>
        <moment
            msDuration="368">
            <noteOffs>
                <msg m="0x80 61 64" />
                <msg m="0x80 73 64" />
            </noteOffs>
            <noteOns>
                <msg m="0x90 61 127" />
                <msg m="0x90 73 127" />
            </noteOns>
        </moment>
    </moments>
    <envs>
        <env s="0xB0" d1="11">
            <vt d2="38" msDur="616" />
            <vt d2="127" msDur="617" />
            <vt d2="38" msDur="1" />
        </env>
    </envs>
</score:midi>

Moments and rests: Every <moment> in the score has an msDuration greater than zero. Rests always have a single <moment> element. If the rest is at the start of the score, then its <moment> will be empty. The <moment> will, of course, still have an msDuration attribute that determines the rest's duration. Rests should never contain noteOn messages. Moritz never writes an <envs> element (that describes control envelopes) into a rest, but other applications might. This is how Moritz writes a <score:midi> element containing noteOff messages inside a rest:

<score:midi>
    <moments>
        <moment msDuration="500">
            <noteOffs>
                <msg m="0x80 60 64" />
                <msg m="0x80 64 64" />
                <msg m="0x80 67 64" />
            </noteOffs>
        </moment>
    </moments>
</score:midi>

Examples in real scores: The Assistant Performer's scores can be found in its scores folder on GitHub To look at the <score:midi> elements and their containers, go to the link, download a page, and search for <score:midi>. Note that Song Six is currently in abeyance, and that Tombeau 1 is work-in-progress.

notator / Moritz