search

w3c / mnx

Music Notation CG next-generation music markup proposal.
179 stars 18 forks source link

altMNX: An alternative to the CWMNX+GMNX proposal #25

Closed notator closed 6 years ago

notator commented 7 years ago

I think the GMNX stucture described in §8 of the current proposals document (26th April 2017) needs to be turned inside out. The W3C already has a natural match for a page of a score (an SVG document) and a standard container for such SVG pages that can represent a score (HTML). I think its a mistake to try to re-invent the wheel. We need to take full advantage of all the existing technology for parsing such files.

Here, for discussion purposes, is some skeleton code showing a) how both MIDI information and media files might be synchronized with the SVG graphics, and b) how synchronized cursors could be realized in scores containing any type of music notation.

begin Edit 16.01.2018 Cursor handling is currently being revised. See the Cursors Issue below. end Edit 16.01.2018

Note that the svg/data-scoreType attribute tells the client application the type of the notation written in the file, and that corresponding semantic information for the SVG elements is provided in their class attributes.

This code sample contains some extended comments. The best way to read it is to copy it into a proper code editor.

<?xml version="1.0" standalone="no" ?>
<svg width="1000" height="1000" viewBox="0 0 1000 1000"
     xmlns="http://www.w3.org/2000/svg" version="1.1"
     xmlns:xlink="http://www.w3.org/1999/xlink"
     xmlns:score="http://www.demo_website.de/scoreNamespace.html"
     data-scoreType="http://www.demo_website.de/cmn_1950.html">
    <!--
    The xmlns:score declaration is a standard SVG namespace declaration.
    The data-scoreType declaration says that this file contains particular classes of graphic
    element, and that it reads in particular directions. As with namespaces, these conditions
    could be described in the referenced file, but the client application does not have to open
    it to discover that this score's type is "cmn_1950".
    For example: "cmn_1950" systems read from left to right, in top to bottom sequence, and can,
    for example, contain <g class="tuplet"> elements.
    Other score types could include "asian" or "cmn_core".
        "asian" would be the base type for "okinawan".
        "cmn_core" could be a base type for "cmn_1950".
    "cmn_core" and "cmn_1950" could be identical, except that "cmn_core" would not include the
    "tuplet", "augmentationDot" and "graceNote" element classes.

    Performing applications have to check
        1. That this file defines the "score" namespace.
        2. That they can perform the particular score type declared in the data-scoreType value.
    Applications that can perform "cmn_core" scores can also perform "cmn_1950" scores, albeit
    with a restricted functionality. The same goes for "asian" and "okinawan" scores.
    -->

    <title>Skeleton GMNX score page</title>
    <desc>
        Skeleton SVG file, showing how an SVG page could be synchronized
        with MIDI and media file information, and how synchronized cursors
        could be realized in any type of music notation.
        The (external) media file could, for example, be an mp4.
        (Each SVG file represents one page of a score. Multi-page scores can
        be combined in an HTML container.)
    </desc>

    <!--
    The following element can be omitted if the score does not reference any media files.
    Begin Edit 29.09.2017
    Media files can be synchronized with scores in this format even if they are not
    referenced explicitly. The <score:mediaFiles> element may therefore be redundant.
    All media files that can be played by the HTML5 <audio> or <video> tags are going
    to be played in that way. It should be possible to synchronize a score by querying
    the <audio> or <video> tag's currentTime property (at intervals just less than the
    screen refresh rate) and using the returned value (via a synchronization table) to
    update the position of the running cursor in the score.
    The <mediaFile> element allows the performing application to create such a
    synchronisation table, but that is not the only way to do that. The application
    could have its own list of media files together with their corresponding 
    synchronisation tables. All the application would need is access to the
    defaultMSPositions of the event symbols in the score, and a function for updating
    the cursor position. This would simplify the file format, for example removing the need
    for event ids.
    Issue: Could the <score:mediaFiles> element be completely deleted from this proposal?
    Issue: Synchronizing MIDI files. An approach similar to the above could be used for
    MIDI files if a standard SMF player were available. This would mean that event symbols
    would not need to contain MIDI info at all. The Standard MIDI File player would have
    properties similar to those of the <audio> tag, and have to be attached to an on-line
    MIDI Output device such as my ResidentSf2Synth (https://github.com/notator/WebMIDISynthHost).
    End Edit 29.09.2017
    -->
    <score:mediaFiles>
        <mediaFile src="demo_media_file.mp4">
            <track>
            <!--
            The first mediaEvent in each track starts at the beginning of the recording.
            Subsequent mediaEvents are in the sequence they have in the recording.
            msDuration is the duration of each mediaEvent in integral milliseconds.
            This could be done more efficiently using simple json.
            Note that these values are approximately, but not exactly, equal to
            the defaultMsDuration attributes of the corresponding <event> symbols in
            the score (see below).
            Issue: I'm not sure that this will _really_ work accurately enough in practice.
            Synchronization with external files (processes) may always be something
            of a compromise on the web - but that's true, whichever structure is chosen
            for GMNX fles.
            -->
            <mediaEvent id="me1" msDuration="1800" />
            <mediaEvent id="me2" msDuration="1795" />
            <mediaEvent id="me3" msDuration="1780" />
            <mediaEvent id="me4" msDuration="1785" />
            <mediaEvent id="me5" msDuration="1814" />
            <mediaEvent id="me6" msDuration="1798" />
            <mediaEvent id="me7" msDuration="1806" />
            <mediaEvent id="me8" msDuration="1798" />
            <mediaEvent id="me9" msDuration="1811" />
            <mediaEvent id="me10" msDuration="5555" />
            <!-- etc.-->
            </track>
            <!-- more tracks could go here -->
        </mediaFile>
        <!-- More mediaFiles could go here. -->
    </score:mediaFiles>

    <g class="systems">
        <g class="system">
            <score:cursorInfo>
                <!--
                The performing application first constructs a list of the unique "eventMoments" in
                each system using the values of the score:defaultMsDuration attributes in each
                <g class="event"> element.
                Each "eventMoment" in the system's "eventMoments" list consists of a list of
                synchronous <g class="event"> ids.
                A cmn_1950 score then only needs to supply the top and bottom coordinates of
                the cursor line for this system, and a list of x-coordinates matching the above
                "eventMoments".
                A score type that reads from top to bottom inside each system would need to
                supply the left and right coordinates of the cursor line for this system, and a
                list of y-coordinates matching the synchronous "eventMoments".
                Score types in which the cursor rotates in some way (or uses some other simple
                transformation) would supply the cursor definition and the list of transformations.
                Score types that have cursors that change shape (if there are any) would supply
                a _list_ of cursor definitions and transformations.

                Note that the cursor algorithm initially means that the cursor will jump from
                moment to moment (after the manner of digital clocks). Performing applications
                can, however, implement smoothly moving cursors by calculating the cursor's
                (varying) speed. (The speed between two eventMoments is the distance between
                them divided by the duration between them.)
                -->
            </score:cursorInfo>
            <!--
            Issue: The container hierarchy used in this file should be
            defined as part of the score type.
            -->
            <g class="measure" id="m1">
                <g class="staff">
                    <g class="voice">
                        <!--
                        A score:defaultMsDuration value must be supplied
                        for all score types, not just CWMN types.
                        -->
                        <g class="event" id="note1" score:defaultMsDuration="1800">
                            <g class="graphic">
                                <!-- ordinary SVG graphics go here -->
                            </g>
                            <score:timeInfo class="mediaFile" mediaEventID="me1" />
                            <score:timeInfo class="midi">
                                <!-- midi info (see https://github.com/notator/Moritz/issues/3 -->
                            </score:timeInfo>
                        </g>
                        <g class="event" id="note2" score:defaultMsDuration="1800">
                            <g class="graphic">...</g>
                            <score:timeInfo class="mediaFile" mediaEventID="me2" />
                            <score:timeInfo class="midi">
                                <!-- midi info -->
                            </score:timeInfo>
                        </g>
                        <g class="event" id="note3" score:defaultMsDuration="1800">
                            <g class="graphic">...</g>
                            <score:timeInfo class="mediaFile" mediaEventID="me3" />
                            <score:timeInfo class="midi">
                                <!-- midi info -->
                            </score:timeInfo>
                        </g>
                    </g>
                    <!-- end of voice -->
                </g>
                <!-- end of staff -->
                <!-- more staves go here -->

            </g>
            <g class="measure" id="m2">
                <g class="staff">
                    <g class="voice">
                        <g class="event" id="note4" score:defaultMsDuration="1800">
                            <g class="graphic">...</g>
                            <score:timeInfo class="mediaFile" mediaEventID="me4" />
                            <score:timeInfo class="midi">
                                <!-- midi info -->
                            </score:timeInfo>
                        </g>
                        <g class="event" id="note5" score:defaultMsDuration="1800">
                            <g class="graphic">...</g>
                            <score:timeInfo class="mediaFile" mediaEventID="me5" />
                            <score:timeInfo class="midi">
                                <!-- midi info -->
                            </score:timeInfo>
                        </g>
                        <g class="event" id="note6" score:defaultMsDuration="1800">
                            <g class="graphic">...</g>
                            <score:timeInfo class="mediaFile" mediaEventID="me6" />
                            <score:timeInfo class="midi">
                                <!-- midi info -->
                            </score:timeInfo>
                        </g>
                    </g>
                    <!-- end of voice -->
                </g>
                <!-- end of staff -->
                <!-- more staves go here -->

            </g>
            <g class="measure" id="m3">
                <g class="staff">
                    <g class="voice">
                        <g class="event" id="note7" score:defaultMsDuration="1800">
                            <g class="graphic">...</g>
                            <score:timeInfo class="mediaFile" mediaEventID="me7" />
                            <score:timeInfo class="midi">
                                <!-- midi info -->
                            </score:timeInfo>
                        </g>
                        <g class="event" id="rest1" score:defaultMsDuration="1800">
                            <g class="graphic">...</g>
                            <score:timeInfo class="mediaFile" mediaEventID="me8" />
                            <score:timeInfo class="midi">
                                <!-- midi info -->
                            </score:timeInfo>
                        </g>
                        <g class="event" id="note8" score:defaultMsDuration="1800">
                            <g class="graphic">...</g>
                            <score:timeInfo class="mediaFile" mediaEventID="me9" />
                            <score:timeInfo class="midi">
                                <!-- midi info -->
                            </score:timeInfo>
                        </g>
                    </g>
                    <!-- end of voice -->
                </g>
                <!-- end of staff -->
                <!-- more staves go here -->

            </g>
            <g class="measure" id="m4">
                <g class="staff">
                    <g class="voice">
                        <g class="event" id="rest2" score:defaultMsDuration="5400">
                            <g class="graphic">...</g>
                            <score:timeInfo class="mediaFile" mediaEventID="me10" />
                            <score:timeInfo class="midi">
                                <!-- midi info -->
                            </score:timeInfo>
                        </g>
                        <!-- end of voice -->
                    </g>
                    <!-- end of staff -->
                    <!-- more staves go here -->

                </g>

                <!-- more measures go here -->

            </g>

            <!-- more systems go here -->
        </g>
        <!-- more systems could go here -->
    </g>
</svg>
joeberkovitz commented 7 years ago

I've retitled this to make it clearer what the issue really is here.

The example attached incorporates a number of other ideas and those should be moved to other separate issues and more clearly explained.

I would like to point out that I've spoken with the chair of the SVG Working Group @svgeesus (Chris Lilley), who's said that either approach (SVG-inside-GMNX or GMNX-inside-SVG) is valid and neither one amounts to reinventing the wheel. So it's really just a question of which one works better for us. I'll present some arguments later on about why I think it makes sense to have SVG inside GMNX, but for now I'll let this conversation unfold further.

One other point to defuse here: even if we put SVG inside GMNX, that doesn't mean we can't break the SVG out into a separate file of its own, and have GMNX incorporate it by linking to it. That is also an attractive option.

notator commented 7 years ago

@joeberkovitz Thanks for the constructive tone above. :-)

We can open and clarify subsidiary issues when we have agreement on the main issue here: What role(s) does SVG play in a GMNX file.

Its an interesting suggestion, that the SVG could be broken out into a separate file of its own. Perhaps you could say some more about that? What uses do you envision for the standalone SVG file? What uses do you envision for the GMNX file? SVG files can be embedded in HTML, so what advantages would GMNX have over HTML? I'm trying to ask serious, not polemical questions here.

joeberkovitz commented 7 years ago

If the SVG piece of GMNX as proposed is broken out into a separate file, then we make it easier for SVG-only applications to just use that document alone as pure graphics, which is one of the points raised by @notator.

I don't see any change at all in the content of the SVG being needed to make this happen, relative to what's been proposed so far. The SVG would still contain IDs and optional references back to a semantic document (e.g. a CWMNX file). GMNX would still contain performance information like timed events, or time ranges in a separate audio file, and would tie these back to regions and event symbols in the SVG.

So at the end of the day, the role that SVG plays in a GMNX file is the same after the file is broken out. Namely...

joeberkovitz commented 7 years ago

I've broken out issue #31 so that the issue of GMNX graphics being directly consumable by SVG-aware applications does not get confused with this issue, which is about whether GMNX performance data is directly embedded within SVG elements.

joeberkovitz commented 7 years ago

At this point I'll try to describe this issue more clearly, by contrasting the two proposed ways (embedding and cross-referencing) of encoding performance and graphics information, and then discussing pros and cons. Because both approaches are capable of representing exactly the same information, both of my examples are going to use exactly the same element names and the same terms to describe them (the ones that are used in the present GMNX proposal). The aim is to minimize the confusion from bundling many other minor changes into this issue.

Scenario

The example will be a very short CWMN passage: a two-note measure that lies within a repeated section of the score, and is thus played twice. Performance directions in the score prescribe that the first repeat is f, and the second repeat pp.

Furthermore, there are two different performances to be encoded in the GMNX representation: an audio performance in an MP4 file, and a timed-event description in terms of pitch/onset/duration/dynamics.

Omitted for clarity are...

Finally, to eliminate the issue of SVG being usable as a standalone file or not, we'll assume that in both cases the SVG is encoded as a separate document (although it obviously looks different in each case).

Cross-referencing approach

In this scheme, a GMNX document represents each performance as a single element with descendants occuring in time order. The events and regions within each performance are cross-referenced to the corresponding elements in an SVG file, using their IDs.

score.gmnx:

<graphics src="score.svg"/>
<performance>
    <performance-name>Event sequence</performance-name>
    <performance-events>
        <track>
            <sequence>
                ...events prior to first repeat
                <event graphic="#note1" start="5" duration="0.25">
                    <note pitch="E4" dynamics="100"/>
                </event>
                <event graphic="#note2" start="5.25" duration="0.25">
                    <note pitch="F4" dynamics="100"/>
                </event>
                ...events prior to second repeat
                <event graphic="#note1" start="10" duration="0.25">
                    <note pitch="E4" dynamics="20"/>
                </event>
                <event graphic="#note2" start="10.25" duration="0.25">
                    <note pitch="F4" dynamics="20"/>
                </event>
                ...remainder of score
            </sequence>
        </track>
    </performance-events>
</performance>
<performance>
    <performance-name>Audio recording</performance-name>
    <performance-media>
        <media-file src="score.mp4"/>
    </performance-media>
    <performance-regions>
        ...audio regions prior to first repeat
        <region graphic="#note1" start="5.01" duration="0.27"/>
        <region graphic="#note2" start="5.28" duration="0.24"/>
        ...audio regions prior to second repeat
        <region graphic="#note1" start="9.98" duration="0.26"/>
        <region graphic="#note2" start="10.24" duration="0.24"/>
        ...remainder of score
    </performance-regions>
</performance>

score.svg:

<svg>
    ...other parts of graphical score

    <g id="note1">
        ...graphics for the first note
    </g>
    <g id="note2">
        ...graphics for the second note
    </g>
</svg>

Embedding approach

In this scheme, a GMNX document is an SVG document. All events and regions in each performance are embedded within the graphic that they refer to.

Structurally, this is along the same lines as @notator's example at the beginning of the issue, except that it doesn't reinvent the way that performances are described or introduce special uses of class etc. which are now covered in other issues.

score.svg:

<svg>
    ...preamble providing overall performance data
    <mnx:performance id="eventPerf">
        <mnx:performance-name>Event sequence</mnx:performance-name>
    </mnx:performance>
    <mnx:performance id="audioPerf">
        <mnx:performance-name>Audio recording</mnx:performance-name>
        <mnx:performance-media>
            <mnx:media-file src="score.mp4"/>
        </mnx:performance-media>
    </mnx:performance>

    ...other parts of graphical score...

    <g id="note1">
        <mnx:event performance-ref="eventPerf" start="5" duration="0.25">
            <mnx:note pitch="E4" dynamics="100"/>
        </mnx:event>
        <mnx:event performance-ref="eventPerf" start="10" duration="0.25">
            <mnx:note pitch="E4" dynamics="100"/>
        </mnx:event>
        <mnx:region performance-ref="audioPerf" start="5.01" duration="0.27"/>
        <mnx:region performance-ref="audioPerf" start="9.98" duration="0.26"/>
        ...graphics for the first note
    </g>

    <g id="note2">
        <mnx:event performance-ref="eventPerf" start="5.25" duration="0.25">
            <mnx:note pitch="F4" dynamics="100"/>
        </mnx:event>
        <mnx:event performance-ref="eventPerf" start="10.25" duration="0.25">
            <mnx:note pitch="F4" dynamics="100"/>
        </mnx:event>
        <mnx:region performance-ref="audioPerf" start="5.28" duration="0.24"/>
        <mnx:region performance-ref="audioPerf" start="10.24" duration="0.24"/>
        ...graphics for the second note
    </g>
</svg>

Discussion

  1. The embedding approach results in a scattering of events and regions throughout the SVG document. In this example, one can see this effect both for repeats and for multiple performances: each graphic contains four separate events and regions (2 repeats * 2 performances). The clear advantage of embedding is that each event/region is closely associated with the graphic that it describes. The disadvantage is that there is no single time-ordered description of a performance: the entire SVG document must be searched to collect lists of events and regions and then sort them by time. (A separate, time-ordered index for each performance could be added to the embedding approach to address this, but this begins to once again split up event/region data between two places, defeating the "close-association" goal.)

  2. In both cases, substantial information is required to simply describe the performances. In the embedding case, this results in a non-trivial lump of data sitting in the SVG document, one that has no real association with any graphical element.

  3. It's worth noting that there is a sort of hybrid approach possible in which the entire cross-reference-style encoding is stuffed into an SVG file as a unit, without modification. (This is not shown above, but it's easy to imagine.) This results in a single SVG file that packages everything, but fails to closely associate performance data with graphics data.

notator commented 7 years ago

@joeberkovitz To reduce friction, I've changed the name of the namespace in my skeleton code at the top of this issue, from score to mnx. I'm going to keep an eye on the related issues, but only contribute to this issue for the time being. I'll leave it to you to raise any new issues as you think fit.

Audio files In the case of Audio files, I think we agree that your cross-referencing and embedding examples are logically equivalent to the way I linked to an existing recording in my skeleton code: In all these cases <g> id attributes reference a portion of an existing recording. The only important difference between cross-referencing and embedding is that in embedding, the ids reference information present in the SVG itself. The Audio file is always external. As I said in my example, there is an unresolved issue here: I'm not sure that Audio (or MIDI) files playing in a separate process can really be synchronized accurately enough in practice. Maybe there is a way round that, but I know that Verovio was discussing this issue some weeks ago.

Repeats and Performances Neither your embedding example nor my skeleton code shows how repeats are controlled, but I don't think there's a fundamental problem there. Ad hoc, I could imagine defining mnx:beginRepeat="repeat1id" and mnx:nRepeats="3" attributes on the first event in the repeat, and a corresponding mnx:endRepeat="repeat1id" on the final event in the repeat. Repeats would then be nestable. Similarly, my example does not show how to deal with separate performances in the presence of repeats. That does not seem insoluble either. A special attribute on my mnx:timeInfo elements would do the trick.

In the introduction to Embedding approach, you say:

Structurally, this is along the same lines as @notator's example at the beginning of the issue, except that it doesn't reinvent the way that performances are described or introduce special uses of class etc. which are now covered in other issues

Could you say why you object to the way I have described performance information? If you mean my use of MIDI to describe events, I must ask how you propose to describe ornamented chords and continuous controllers. (This may have to be done in different ways in CWMNX and GMNX.) There may be a high-level approach to that, but I've come to the conclusion that a low level approach is better: A low level approach, in which MIDI messages are stored as numeric information in the file, means that MIDI support is complete without having to define large numbers of standard constants. We are talking about code that is written and read by machines and MIDI experts, so I don't think the legibility problems are very great -- especially as I am proposing named containers for the main message categories.

Argument 1 in favour of the embedded approach: In issue #28 I said:

Forcing the class information to be stored in a separate file actually prevents the use of CSS in the SVG. Or, put another way, if classes are used for CSS purposes in the SVG, then there is no need to reference an external file to find the same information.

I think the standard use of SVG class attributes is a must, owing to their relation to CSS, and that their use makes a separate file containing the class information redundant.

I've pointed out before, that CWMNX element names generally correspond to class names in the instantiated SVG. (e.g. a CWMNX <event> element corresponds to a GMNX <g class="event" ...>. Such a correspondence would be produced by the app that converts the CWMNX (or MusicXML) to the GMNX score instantiation. Possibly, there could be some reason for packaging the CWMNX and GMNX in some container later, but the information in the abstract CWMNX file has already been transferred to the instantiation.

This is true, not only for event symbols but also for containers. The container hierarchy can also be usefully preserved from CWMNX to GMNX: a CWMNX <staff> (if there is one) should correspond to a <g class="staff">. Since we are talking about containers that don't overlap graphically, there will be no problem with the graphic representation. (We have already discovered that tuplet brackets are an exception to this: in CWMNX they are currently XML containers, but in GMNX they are not.)

Container hierarchies are part of my proposed score typing scheme (see the comment in the skeleton code above).

Notice that you have not included the container hierarchy in your Embedded example, and then consider what happens if the score is not CWMNX but simple text having the container hierarchy page/line/word. By definition, if the words are associated with temporal information then they are event symbols, and the file contains music notation. If there is a neume above each word, then the combination of word+neume is an event symbol. There could easily be a "simpleNeumes" score type that could be performed by software designed for the purpose. All that is needed for a definition of a score type is a definition of the container hierarchy, the directions in which the containers are to be read, and agreement on the names of the SVG element classes that can/will exist in the file.

Argument 2 in favour of the embedded approach: An application that starts without an imported CWMNX file, or creates non-CWMN scores, would first have to create a file containing the abstract element structure (a CWMNX file or some equivalent non-CWMN file) and then write an SVG using ids to reference the abstract elements. It would be much simpler just to declare the score type and use element classes directly in the SVG.

Your discussion points:

  1. The disadvantage [of the embedding approach] is that there is no single time-ordered description of a performance: the entire SVG document must be searched to collect lists of events and regions and then sort them by time.

I don't think that's really a problem. On the web, Javascript is extremely fast, and desktop applications would if anything be faster. The score can be read before any performance begins. Some very large scores might require a multiprocessing approach (reading and performing at the same time), but I don't think that's fundamentally unsolvable by today's software.

  1. In both cases, substantial information is required to simply describe the performances. In the embedding case, this results in a non-trivial lump of data sitting in the SVG document, one that has no real association with any graphical element.

Applications that export performable SVG files (GMNX) could very easily also export simple SVG files that don't include the performance data. These are just two closely related export options.

  1. It's worth noting that there is a sort of hybrid approach possible in which the entire cross-reference-style encoding is stuffed into an SVG file as a unit, without modification. (This is not shown above, but it's easy to imagine.) This results in a single SVG file that packages everything, but fails to closely associate performance data with graphics data.

Not sure that I understand that. It sounds rather like the approach I took to linking to Audio files. But If the file fails to closely associate performance data with graphics data, then it can't really be a candidate for what we are looking for.

Other discussion points: Performing at speeds other than those given in the file(s). This can't be done using Audio files, but is possible using MIDI data.

siennamw commented 7 years ago

I think the standard use of SVG class attributes is a must, owing to their relation to CSS, and that their use makes a separate file containing the class information redundant.

Actually, in addition to class selectors, CSS uses IDs, types, and attributes as well as hierarchy-dependent selectors (descendants, direct children, siblings, adjacent siblings). For your reference: https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors.

As for the general subject of this thread, I support cross-referencing rather than embedding.

@notator is making many specific arguments about the advantages that he sees in embedding, but from a developer's perspective keeping the different data types separate makes them much easier to work with. There is an argument to be made that embedding might be the more musically-intuitive model, but let's put an end to the idea that there would be a technical advantage in that approach.

notator commented 7 years ago

I'd just like to say that I have now improved my Assistant Performer's documentation by providing mp4 videos of the app in action. These can be viewed without having to install any browser plug-ins. Hopefully, being able to view Study 2 will make it easier to understand a couple of things:

Two other things I should say while I'm here:

notator commented 7 years ago

I've been asked to continue all my closed issues here. These include:

26 Tuplets and augmentation dots become annotations in the presence of explicit timing data

29 The definitions of "event symbol" and "music notation"

36 The relation between CWMNX and GMNX

37 The core of CWMN

36 first:

This issue deals with the specific dialect of GMNX that represents a CWMN score (cwmnGMNX). As described above, a cwmnGMNX score would have a data-scoreType="http://www.demo_website.de/cwmn.html" attribute in its top level <svg> elements, telling the parser that this <svg> conforms to a standard way of describing a CWMN score page instantiation. It would obviously be convenient/best if the container structure and element class names were, as far as possible, the same as those used by (score-timewise) MusicXML. Since all GMNX scores must be easily printable, its best if there is a separate <svg> element for each page. Multipage scores are embedded in an html container.

If there is a standard format for cwmnGMNX scores, then the following scenario becomes possible: A CWMN music editor application (capella, Dorico etc.) could begin editing with a blank score, or by importing a score that is either in its own native format, in MusicXML format, in CWMNX format or in cwmnGMNX format. Since cwmnGMNX is a standard format, it wouldn't matter which application actually created it, so it is also an interchange format between different music editor applications. Its a successor to MusicXML in that respect. After importing the cwmnGMNX file (which may or may not contain performance information), applications could simply overwrite the specific instantiation information, replacing it with new or edited information. They could then re-export the edited file (with or without performance information) in cwmnGMNX format. They could also export the score in MusicXML or CWMNX format as far as that is possible. That's good for backward compatibility, and protecting the investments of those who use MusicXML or CWMNX.

Further to #26 and #37: It would be quite possible to define cwmnGMNX with tuplet brackets at the same container level as they are in MusicXML, but to define other "partly CWMN" GMNX dialects in which they dont exist at all. Begin Edit 28.09.2017: If cwmnGMNX is defined with tuplet brackets at the same container level as they are in MusicXML, their meaning would still be overridden by any temporal data in the <g class="event" ... > elements. As always, its extremely important to preserve the conceptual difference that exists between the appearance (graphics) and meaning (temporal information) of symbols. End Edit 28.09.2017: Augmentation dots will, of course, exist in the usual way in cwmnCMNX, but they don't belong to the core symbols, so they dont need to exist in all GMNX dialects.

New Issue: All GMNX dialects should have a standardized, non-functional "annotations" or "user" level for comments and other non-standard reading aids (user notes, spectacles etc.).

Issue: CWMNX vs. cwmnGMNX? The above thought experiment shows, I think, that the cwmnGMNX standard does not depend on the existence of a CWMNX standard (and vice versa). The two formats are completely different, and will have different use cases. cwmnGMNX is a score instantiation, CWMNX is not. The formats are not competitors. Proposed improvements to MusicXML in the current CWMNX debate, should of course flow into the cwmnGMNX definition, regardless of whether CWMNX becomes a format in its own right. Issue: There should be a schema (closely related to MusicXML and/or CWMNX) describing cwmnGMNX, but I don't know how to write a schema that is an extension of SVG. Can anyone help?

notator commented 6 years ago

I've changed the title because the new MNX Draft Specification says that GMNX is

a kind of universal encoding for scores having arbitary graphical and audio content. In consequence, it is relatively free of semantics.

I was using the term GMNX above to express the fact that this proposal is extensible to non-CWMN notations.

notator commented 6 years ago

Issue: cursors

The treatment of cursors described above (on 2nd May 2017), could be much improved by describing them with explicit JSON strings having a structure similar to the one below. This string contains the definitions of the three, simultaneously running cursors in my oggPngPlayer. [1]

let cursorsJSON = `{
    "cursors": {
        "cursor1": {
            "name": "cursor1_red",
            "type": "rect",
            "positions": {
                "pos1": { "times": {"t1": 0.3}, "x": 0, "y": 0, "width": 0, "height": 0 },
                "pos2": { "times": {"t1": 4.4}, "x": 143.26, "y": 289.47, "width": 176, "height": 23 },
                "pos3": { "times": {"t1": 7.2}, "x": 326.61, "y": 289.47, "width": 136.3, "height": 23 },
                "pos4": { "times": {"t1": 10}, "x": 472.8, "y": 289.47, "width": 211.83, "height": 23 },
                "pos5": { "times": {"t1": 12}, "x": 79.92, "y": 425.31, "width": 79.04, "height": 23 },
                "pos6": { "times": {"t1": 15}, "x": 167.63, "y": 425.31, "width": 171.02, "height": 23 },
                "pos7": { "times": {"t1": 18}, "x": 340.62, "y": 425.31, "width": 161.88, "height": 23 },
                "pos8": { "times": {"t1": 21.5}, "x": 167.63, "y": 425.31, "width": 171.02, "height": 23 },
                "pos9": { "times": {"t1": 25}, "x": 340.62, "y": 425.31, "width": 161.88, "height": 23 }
            }
        },
        "cursor2": {
            "name": "cursor2_green",
            "type": "rect",
            "positions": {
                "pos1": { "times": {"t1": 0.3}, "x": 0, "y": 0, "width": 0, "height": 0 },
                "pos2": { "times": {"t1": 6.8}, "x": 143.87, "y": 314.45, "width": 285.54, "height": 23 },
                "pos3": { "times": {"t1": 10.2}, "x": 434.43, "y": 314.45, "width": 178.33, "height": 23 },
                "pos4": { "times": {"t1": 11.6}, "x": 617.16, "y": 314.45, "width": 66.86, "height": 23 },
                "pos5": { "times": {"t1": 16.5}, "x": 81.13, "y": 458.81, "width": 276.4, "height": 23 },
                "pos6": { "times": {"t1": 22.4}, "x": 357.07, "y": 458.81, "width": 221.58, "height": 23 },
                "pos7": { "times": {"t1": 24.7}, "x": 617.16, "y": 314.45, "width": 66.86, "height": 23 },
                "pos8": { "times": {"t1": 28.5}, "x": 81.13, "y": 458.81, "width": 276.4, "height": 23 }
            }
        },
        "cursor3": {
            "name": "cursor3_blue",
            "type": "rect",
            "positions": {
                "pos1": { "times": {"t1": 0.3}, "x": 0, "y": 0, "width": 0, "height": 0 },
                "pos2": { "times": {"t1": 6.5}, "x": 143.26, "y": 338.81, "width": 171.63, "height": 23 },
                "pos3": { "times": {"t1": 15}, "x": 325.39, "y": 338.81, "width": 358.63, "height": 23 },
                "pos4": { "times": {"t1": 18}, "x": 84.18, "y": 490.49, "width": 138.13, "height": 23 },
                "pos5": { "times": {"t1": 28}, "x": 226.11, "y": 490.49, "width": 352.54, "height": 23 },
                "pos6": { "times": {"t1": 30}, "x": 0, "y": 0, "width": 0, "height": 0 }
            }
        }
    }
}`;

The object hierarchy

"cursors" is simply a container for a list of cursor definitions. Each cursor has

Each position definition contains

Sub-issues

This is an issue. Constructive suggestions for improvements, alternatives etc. would be very welcome.

[1] The oggPngPlayer was originally written to duplicate the behaviour of @joeberkovitz' GMNX viewer playing the Lutoslawski example. Many thanks, Joe, for the provocation! See Issue #51.

notator commented 6 years ago

Sorry about changing this issue's title so much, but ideas have been developing very fast recently.

I have decided to give this proposal a name (altMNX), so that I/we don't have to keep using circumlocutions.

The altMNX specification proposes that this CG's goals (and more) can be achieved using standard SVG, but with a class and class-nesting structure specialized for different music notation types. (See #23)

For example, CWMNX would be an SVG file containing an <svg> attribute of the form: data-scoreType="http://www.demo_website.de/cwmn.html". This is rather like a standard SVG namespace declaration: Parsing applications don't have to open the data-scoreType file in order to know that the current file contains the CWMN dialect of altMNX.

notator commented 6 years ago

Since I need more space to think about, discuss and develop this altMNX proposal, I have now created a child repository on my own GitHub site. The cursor sub-issues, mentioned above, have moved there.

As a first step, I've rewritten the oggPngPlayer application (also mentioned above) using a proper cursor type (RectCursor) that is used multiple times, while creating a new CursorLib library containing functions that can be used by any cursor type. The rewritten application can be found here.

The second step was to define the usual cursor type used in CWMN (VLineCursor), and use it together with the CursorLib in an application that synchronizes four different audio recordings with the score of Fauré's Après un rêve. Note, in particular, that the theoretical, absolute durations notated in the score are ignored. See #26.

These applications relate to Use Cases RLP8 and RLP9.

I think the cursor JSON formats need discussing and standardizing, so that:

  1. Any music editing application that can export a score in SVG format can export standardized cursor data for that score.
  2. Applications can be written that will be able to read that standardized cursor data and synchronize further recordings with such scores.
  3. Users of performing applications will be able to "conduct" such scores (i.e. define how time passes).
dspreadbury commented 6 years ago

The following use cases (described here) cannot be practically accommodated by the proposal to use SVG with additional markup.

MC1: Composer wants to share work with a collaborator using another editing program MC2: Composer wants to migrate work to another editing application… MC4: Editor wants to compare two versions of digital sheet music… MC11: Sound Engineer wants to use an interactive score to access low level information in a DAW

It’s unreasonable to expect authors of editing programs (especially non-notation and indeed non-graphical focused applications, like DAWs) to handle SVG rather than a more direct encoding of the musical semantic data (as mentioned by Adrian Holovaty in response to #67).

MP4: Publisher wants to prepare digital editions that can be viewed on any device and also printed MP5: Engraver wants detailed control… while allowing for more flexible rendering on arbitrary devices MP6: Publisher wants to decouple semantic formatting of notes and text from physical formatting MP11: Publisher wants to be able to present user-visible notice of credits, copyrights in a way that may vary depending on device form factor RLP1: Performer wants to reformat sheet music to her mobile device’s display RLP2: Performer wants to reformat sheet music as per personal preference RLP4: Performer wants to view only his own musical part from a larger ensemble score RLP5: Performer wants to transpose music to suit a specific performance situation RLP6: Performer wants to view her own musical part formatted more prominently… RLP14: Performer wants to apply cuts, then share the modifications with fellow performers RLP17: Performer wants to select from a set of alternative readings of a work… RLP18: Performer wants to insert, change or delete a cue in their part ED1: Student wants to complete a music theory assignment AC1: Users with disabilities with to interact with a score via accessible input/output modalities AC2: Low-vision users with to view an arbitrary score in Braille notation DEV3: Developers want to render entire scores dynamically in applications for specific purposes DEV4: Developers want to render specific portions of scores dynamically in response to user actions

All of these use cases rely on a separation between semantic and presentational data, in order to allow the music to be rendered dynamically by a rendering library or application, to support the reflowing, re-laying-out, reformatting etc. of the music, in a way that a fixed graphical representation in SVG cannot accommodate.

There are other use cases that I believe may also be made more complex by this approach of embedding other data in SVG whilst not being necessarily completely impractical (e.g. MP12, MP13, MP15, ARL2, ARL3, ED5, others).

This proposal to use SVG plus markup as a kind of “altCWMNX” proposal compromises too many of the group’s agreed use cases, and therefore I cannot support it.

notator commented 6 years ago

@dspreadbury

This proposal to use SVG plus markup as a kind of “altCWMNX” proposal compromises too many of the group’s agreed use cases, and therefore I cannot support it.

Firstly: I cannot agree that including semantic data in SVG files would compromise any of the group's use cases. If anything, it would enable a lot more. Its quite possible that both the chair's and my proposals could be useful in different scenarios and that both could be implemented. But we don't yet have a final version of either, so its too early to rule anything out. Are you definitely going to support the chair's proposal whatever it turns out to be? Secondly: I think cwmnGMNX can be made powerful enough to be used as an interchange format, so many of the use-cases you cite above could indeed be supported.

I would be very interested to know why it should be so difficult for CWMN music editors to include semantic data in their SVG output. Why is there so much resistance to doing that? [begin speculation] Possibly, the major vendors are all using plugins that don't allow class information to be written. If that's the case, it might look as if they are going to have to do a lot of work rewriting their SVG export code. In such a case, the work would be much reduced if they all agreed to use the same, re-written plugin. [end speculation]

dspreadbury commented 6 years ago

Perhaps you could clarify, @notator, how you anticipate supporting the use cases that rely on flexible and dynamic reformatting of musical data by embedding semantic data in SVG?

The chair's goal is to start ruling things in or out as soon as possible so that we can make concrete progress towards building the specification. The chair believes that it is necessary to determine in the immediate future whether we should consider using SVG as the carrier for CWMNX data, because if so, this would require us to change the approach that we have already presented for the outline of the CWMNX specification.

I would require a great deal of persuasion – with concrete examples – that this "altCWMNX" proposal could meet the requirements of the use cases I listed in my previous comment on this issue. One of the goals of CWMNX is to provide a format that could be used as the native format for music editors: I'm pretty sure that there are very few music software developers who are interested in using SVG as a native file format. This alone is a significant body blow to the proposal, but even that is a niggle in comparison to being unable to meet any of the use cases that the group has identified that require dynamic reflowing/reformatting of the music notation, which SVG (at least based on my own understanding of its capabilities) cannot support.

notator commented 6 years ago

If all the containers and (most of the) graphic elements in a (mulitpage) SVG score are given an agreed class attribute, then the logical structure of the score can be retrieved using the standard getElementsByClassName() method while ignoring all the specific graphic info. The importing application can then construct its own abstract class structure for the whole score, and use its own defaults to edit/format that structure in any way it likes.

If, for example, the expected container hierarchy (from the page level down) were defined as: svg->system->staff->voice->eventSymbol then the systems on the page would be retrieved using systems = document.getElementsByClassName("system") and the number of systems on the page would be systems.length.

Similarly, the (number of) staves per system would be found by calling staves = system.getElementsByClassName("staff") for each system. The (number of) voices per staff would be found by calling voices = staff.getElementsByClassName("voice") and the (number of) eventSymbols per voice would be found by calling eventSymbols = voice.getElementsByClassName("eventSymbol")

Using a custom namespace, all SVG elements (including such eventSymbol container elements) can be provided with arbitrary attributes or contained elements. That information could easily include the logical duration of each eventSymbol (the info previously coded in MusicXML). So I think its possible to reconstruct the whole logical structure of such a multipage SVG score, without looking at the instantiation data.

I think that both the container structure and all the class names should be inherited as far as possible from MusicXML (learning, of course, from the MusicXML experience) so that MusicXML programmers moving to SVG know what they are dealing with. This time we need to get agreement from the whole community, and "nail things down" properly... Which element classes exist, and in which containers are they expected to appear?

Should there be a need for styling hints or other, more subtle, forms of information, I think they could also be incorporated using the custom namespace.

If there's anything that's not clear, or you see any snags, I'm all ears.

joeberkovitz commented 6 years ago

@notator, I think you continue to leave unanswered the basic objections that we have raised before, including these new ones articulated by @dspreadbury and @adrianholovaty. Let's recap:

Responding once again that you believe that your approach can work, is not going to change the chairs' minds. If you want a serious consideration of your point of view, we really would need to hear either new arguments not yet advanced by you, or other developers' voices supporting this alternative.

notator commented 6 years ago

If I'm right, it should be possible to translate a MusicXML file into an SVG file that contains no instantiation. @dspreadbury is right to ask for a concrete example, and I would like to provide one.

However: I am not a MusicXML expert. It would be best for me to demonstrate the principles (get this project off the ground), and then ask for help from the community (especially @joeberkovitz and @mdgood) in fleshing things out. MusicXML is a large specification, and its obviously going to take a little time to cover it all while deciding on the best ways to treat various features.

I am going to start by translating a significant part of the FaurReveSample-MusicXML.xml file (to be found in the examples folder in the MNX repository). Once the chair understands how this approach works, and is convinced that it can be extended for the whole of MusicXML, we can take things from there.

I think it will eventually be possible for such translations to be done programmatically, and that different translators will be able to cope with source files that make different assumptions (i.e. MusicXML that has been exported from different applications).

To answer the points @joeberkovitz summarizes above:

Applications that dynamically render CWMN scores (the majority of use cases that @dspreadbury cited) have no need of a ready-made graphical instantiation that looks different from the one that they will create. So you are asking that developers do a great deal of extra work to handle incoming SVG that is not relevant to their needs. What's more, SVG created by such applications can only reflect one dynamic rendering, of the many renderings that the app is capable of.

No. Developers who are interested only in the logical structure of the file, only need to look at the class information and the information in the mnx (or musicXML ? ) namespace. They can ignore any instantiation data in the file, and dont have to do any more work than they would when parsing ordinary MusicXML. They will probably have less work to do, because the standard will be properly fixed, and they don't have to handle different input assumptions. The SVG file does not have to contain any instantiation information at all.

Focusing on "music editors" alone is a mistake: it is easy to think of many applications that do not render the score at all. Consider a utility program that reads in a CWMNX score and transposes it by some interval, writing out a transformed CWMNX score. In such cases, the application has no machinery on hand to generate a graphical rendering of the score. To require the app to create SVG here goes beyond mere inefficiency and burden, and becomes nonsensical. This would oblige the application to incorporate a rendering library or tool for a purpose unrelated to its needs or to other use cases.

I'm not sure that I understand this. If the app does not render the score at all, then it can't write out a transformed CWMNX score. I'm not focussing on "music editors" alone. As I outline above, its quite possible to treat a cwmnGMNX file that has no instantiation as being logically equivalent to MusicXML.

You seem to assume that a graphical structure will mimic the logical structure of a score -- e.g. that an element marked as "staff" would contain elements marked as "voice", which in turn would contain elements marked as "events". Such an assumption flies in the face of the goal of the SVG hierarchy, which is to organize elements in a way that simplifies their generation and display, not their ability to be parsed by music notation applications.

The goal of the SVG hierarchy is not to simplify the generation and display of the elements, but to determine their z-order. SVG elements that are written later than others (i.e. lower down in the file) are written on top of elements written earlier (in the z-order). In music notation, containers having the same class are never written on top of each other (in the z-order), so the order in which they are written does not matter for the rendering, and we are free to organize the files in any way we like.

I'm now going to start making an SVG file that is equivalent to a significant part of FaurReveSample-MusicXML.xml. I'll report back here when I think I've got something you need to look at.

joeberkovitz commented 6 years ago

I am sorry, but it appears you continue to avoid answering these objections.

They can ignore any instantiation data in the file, and dont have to do any more work than they would when parsing ordinary MusicXML. They will probably have less work to do, because the standard will be properly fixed [...]

I sincerely doubt this claim is true given the need to weave semantic information into an SVG file. You need to prove that it actually is simpler, with a detailed example that encodes everything that CWMNX does.

Note that the standard is being "properly fixed" in either case, so saying this adds nothing to the discussion. For my part, I'd claim that SVG is not going to lend more rigor to an encoding of semantic data, and will likely subtract from it since it was not designed for this purpose.

The SVG file does not have to contain any instantiation information at all.

In which case, SVG is not serving any purpose at all, and using SVG as an encoding is a hindrance -- a bunch of empty groups with class tags and assorted injected non-SVG elements. I don't see any possible argument that SVG-without-graphics is a simpler package to deal with than straight semantic markup as currently proposed.

If the app does not render the score at all, then it can't write out a transformed CWMNX score.

"Render" means the creation of graphics. So certainly this app can transform a score, by simply changing notes, clefs, etc. from the original, and writing them out without creating any graphics of the transformed score. (The original graphics would be lost in any such transformation.) But now it's clearer that you think SVG makes sense as an encoding even when there are no graphics.

The goal of the SVG hierarchy is not to simplify the generation and display of the elements, but to determine their z-order. SVG elements that are written later than others (i.e. lower down in the file) are written on top of elements written earlier (in the z-order).

This last point about incompatible hierarchies is a really significant objection, and I don't think you have ever understood it properly, or responded in a way that makes me feel that you understand it.

You seem to have a naive understanding of the SVG hierarchy and of the overall design of SVG. Z-ordering is just one of a number of essential concerns. Please study the coordinate transformation capabilities of the <g> element; this feature forms a key aspect of complex SVG renderings. The ability to group elements together that share a common local coordinate system is essential. Any hierarchy-driven distortion of SVG that interferes with this, makes SVG far less expressive than it should be.

The hierarchy also serves to group together elements with common interactive event handlers, and applications displaying SVG will not always have the same event-handling concerns to shape their hierarchy. So here too we have great potential for the force-fit of semantic parsing onto SVG to disrupt all the good things about SVG -- which are largely absent, anyway, if SVG doesn't contain any actual graphics!

notator commented 6 years ago

I said

The SVG file does not have to contain any instantiation information at all.

That is, of course, overkill. Part of what one might call "creative thinking". What I meant was that it should not be necessary for a parser that is only interested in the logical information in the score to look at the precise locations in the instantiated elements. For example, its not necessary to look at system instantiation information to discover how many systems there are on the page. In CWMN, there are many situations in which the necessary logical information cannot be extracted from the graphics (e.g. the duration of an event as a fraction of a quarter note, or the event symbols that are connected by a beam). I think it will always be possible to supply such information in namespaced elements or attributes.

You seem to have a naive understanding of the SVG hierarchy and of the overall design of SVG.

You may be right, but I've been working with SVG so long that I think its unlikely. The only way to find out is for me to provide the concrete example of a cwmnGMNX score that @dspreadbury is asking for.

Contrary to my previous posting above, I've decided to tackle this from the other end: I'm going to try to restructure the elements in the Aprés un rêve SVG file as cwmnGMNX, calling the namespace cwmn. Ideally, I'd like to convert the whole file, but that may not be necessary. I already know, after working on the cursor synchronization example, that it is organized systemwise, so I might report back when I have converted the first system (or have failed to do so). The lower systems could be left, unstructured, where they are...

Let's postpone further discussion on this until we have something concrete to discuss. Of course, if you are right, and I find out that it really is impossible to do the conversion I have in mind, that will be the end of the discussion. :-)

joeberkovitz commented 6 years ago

I said, "the SVG file does not have to contain any instantiation information at all". That is, of course, overkill. Part of what one might call "creative thinking". What I meant was that it should not be necessary for a parser that is only interested in the logical information in the score to look at the precise locations in the instantiated elements.

Now that you're agreeing that this is overkill, you're still not answering the fundamental objection: many applications have absolutely no need to generate graphics, such as the example of a command-line score transposition tool. Forcing such an application to package its output as SVG makes no sense. You are left with two alternatives, both of which seem unacceptable on their face: 1) emit SVG with no graphics and only semantic information, or 2) force all applications to include a graphical rendering in their output, even if they neither need to consume nor produce graphics.

When you work up your concrete examples, please be sure to prepare something that addresses this set of use cases.

joeberkovitz commented 6 years ago

A note on process, not on content: You asked us to postpone further discussion until we have something concrete.

The chairs cannot postpone discussion of this issue for very long, because work on the spec must continue, and this issue proposes to abandon the current direction and go somewhere very different.

We expect to move in short order on resolving this issue one way or the other in the next few weeks. Therefore all materials in support of your argument should be provided ASAP. You've been raising this issue in one form or another for nearly a year, and the work of the group cannot accommodate an open-ended timeframe in which the issue is continually brought back to life.

joeberkovitz commented 6 years ago

I've prepared a more complete example that shows exactly how the current MNX spec supports a combination of SVG and semantic CWMN data. This can serve as a basis for comparison with alternatives that somehow pack CWMN into SVG, as this issue proposes.

No spec changes were required; this is simply a further buildout of the existing GMNX materials using the <score-mapping> element which already existed.

The example lives here, with comments included: https://joeberkovitz.github.io/gmnx-viewer/examples/ApresUnReve/ApresUnReve-gmnx.xml

Here is a brief summary of the approach.

The key advantages of this approach are:

  1. The addition of element IDs does not disrupt either SVG or CWMNX in any way.
  2. There is no CWMN-related information added to SVG documents, either as class attributes or as non-SVG elements like data-xxx or custom namespaces. Thus, the "plain old SVG" may be created by any compatible graphics library (e.g. Raphael, paper.js, or various native libraries), and may be imported/exported without loss by any SVG-compatible drawing program.
  3. There are absolutely no constraints on the hierarchical structure of the SVG.
  4. CWMNX remains a pure semantic dialect and may be read/written by applications that don't care about graphics.
  5. Any number of GMNX instantiations may be mapped back to the same CWMNX score. This seems important given that multiple graphical variations (e.g. for different display surfaces) will often be the norm.
  6. XML Schema documents may be created to describe the CWMNX schema and automatically validate CWMNX documents, or used to power schema-driven parsers and generators. It is not clear how this could be done with a set of conventions on the use of SVG class attributes.

When a concrete alternative is proposed, it should be compared against the speced proposal in terms of these objective advantages.

notator commented 6 years ago

@dspreadbury asked

Perhaps you could clarify, @notator, how you anticipate supporting the use cases that rely on flexible and dynamic reformatting of musical data by embedding semantic data in SVG?

I've now uploaded two files: FaurReveSample.cwmnNoGraphics.svg and FaurReveSample.cwmn.svg. Both of these contain the CWMN semantics for the first eight bars of the FaureReveSample, as defined in the <cwmnx> element in @joeberkovitz' FaurReveSample-cwmnx.xml file. The only difference between my two files is that the first contains no SVG drawing elements, and therefore renders as a blank page. The second contains SVG data that originated in FaurReveSample.svg.

It is just as easy to retrieve the CWMN semantics from my SVG files as it is from FaurReveSample-cwmnx.xml. These files therefore support flexible and dynamic formatting of the score. (There's a more detailed explanation of how this works below.)

FaurReveSample.cwmn.svg (the version with graphics) is structured in the usual SVG way, with both <style> and <defs> elements in the preamble to the score proper. <style> contains standard CSS, that applies to the named line, path and rect types. <defs> contains definitions of objects that are (re)used later in the score (in <use> statements). Both these sections can be skipped, if you are only interested in how the semantic info is written.

Namespaces are a standard mechanism for including foreign data in SVG files. Once a namespace has been delared (I called mine cwmn), it can be used as a prefix to any element or attribute name. SVG renderers (browsers) will ignore anything that is namespaced in this way, and anything lower down the XML tree. Parsers that are expecting to parse the cwmn namespace do so in the same way as they parse non-namespaced elements and attributes.

Elements that are going to contain SVG drawing instructions cannot be prefixed because that would prevent SVG renderers from seeing those instructions. Such elements are therefore redeclared as <g> (group) elements having a class whose name is the original name of the element. For example, the following snippet from FaurReveSample-cwmnx.xml

    <event value="/8">
        <note pitch="Ab3" accidental="flat" staff="2"/>
        <note pitch="C4" staff="2""/>
        <note pitch="D4" staff="2"/>
        <note pitch="F4"/>
    </event>

gets translated to the following SVG

    <g class="event" cwmn:value="/8">
        <!-- SVG drawing code (stem, flags etc.) -->
        <g class="note" cwmn:pitch="Ab3" cwmn:accidental="flat" cwmn:staff="2">
                <!-- SVG drawing code -->
        </g>
    <g class="note" cwmn:pitch="C4" cwmn:staff="2">
                <!-- SVG drawing code -->
        </g>
        <g class="note" cwmn:pitch="D4" cwmn:staff="2">
                <!-- SVG drawing code -->
        </g>
    <g class="note" cwmn:pitch="F4">
                <!-- SVG drawing code -->
    </g>
    </g>

There is a note at the top of the FaurReveSample-cwmnx.xml file to say that it is not complete. It is therefore very possible that I've made mistakes when filling in the blanks, or have made debatable choices at some points. But I don't think that's a serious problem. The point is, that however CWMNX turns out in the end, there will be a way to translate its semantic information into a form that can be included in SVG files. That means, in turn, that I am in no way trying to upturn the CWMNX debate. With the proviso that I think the mnx container is redundant, its simply not true, that this approach

would require us to change the approach that we have already presented for the outline of the CWMNX specification.

My concern is with the delivery of the semantics, not their content, and with the extra power that semantically enhanced SVG would give to programmers. SVG instantiations enable an important set of use-cases (web applications, performance, synchronisation etc.). Analysing the Finale plug-in's completely unstructured SVG (FaurReveSample.svg) was no picnic! :-)

I'll reply to @joeberkovitz in a separate post. This one is long enough already.

notator commented 6 years ago

@joeberkovitz (in reply to your most recent posting):

Basically, my objection to the current version of the <gmnx> tag (as used in your ApresUnReve-gmnx.xml) is that it is too complicated, unwieldy and error-prone.

As I have demonstrated in FaurReveSample.cwmn.svg, there is a straightforward way to write CWMN scores in SVG, together with the cwmn semantics you are developing for <cwmnx>. This removes the need for multiple files that have to be linked with object ids, and makes the development of web applications (and other applications that use semantically enhanced SVG) much easier. The programmers only have to read one file, and there is no need to keep track of ids because the graphics are embedded in the semantics.

Notations other than CWMN (that use a different container hierarchy and semantics) can be treated in a similar way, by defining a corresponding namespace (e.g. asianmn or textmn instead of cwmn). (My use of data-scoreType in the example at the top of this issue, turns out to be superfluous.) There is no limit to the number of notations (=namespaces) that could be defined, but I suspect both that the necessary number is actually quite small, and that the container hierarchies and semantics are much simpler in non-CWMN notations. Note that each such notation (=namespace) needs both authoring and parsing software.

To reply to the "key advantages" you see in the current approach to <gmnx>:

The addition of element IDs does not disrupt either SVG or CWMNX in any way.

I think the SVG needs to be disrupted (enhanced with semantic information). The applications that author the SVG have all the necessary information to be able to embed the graphics in the semantics. Apropos fixed coordinates and z-order: It turned out, that Finale's output SVG (in FaurReveSample.svg) was ordered bar-wise, while your FaurReveSample-cwmnx.xml is ordered part-wise. That means that I had to move the SVG drawing instructions to a completely different place in the file. (That was a bit fiddly, but in principle not a problem.) I moved each object inside the <g> element containing its semantics, but there is a choice as to how to organize the z-order within such elements. There are a couple of places where I think the z-order needs to be considered.

Applications can, of course, change the z-order if they know what they are doing, but it would be a good idea to agree on some sensible defaults.

There is no CWMN-related information added to SVG documents, either as class attributes or as non-SVG elements like data-xxx or custom namespaces. Thus, the "plain old SVG" may be created by any compatible graphics library (e.g. Raphael, paper.js, or various native libraries), and may be imported/exported without loss by any SVG-compatible drawing program.

The usual way to create semantically enhanced SVG will be to use an editor that understands both the graphics and the semantics. Any SVG-editor worth its salt ought to preserve any namespaced data that it does not understand. Even Inkscape does that! It is, of course, possible to write/edit such files by hand: I'm using Visual Studio (Community Edition 2015), which (like a browser) renders the graphics very well while ignoring namespaced data. (Thank you Visual Studio!)

There are absolutely no constraints on the hierarchical structure of the SVG.

That's a problem, not an advantage. How is the client software supposed to know what is in the file? In other words: if you want to define areas in the SVG without looking inside the SVG, you will have to do more (one-off) work to do so. It would be much easier to have all the synchronisaton information already present and parsable in the file.

CWMNX remains a pure semantic dialect and may be read/written by applications that don't care about graphics.

I've provided an example of an empty SVG file that contains the same semantic information as in CWMNX. Such files would use the same semantics-parser as for non-empty SVG files. But, if there is sufficient demand, I see no objection to defining a simple .cwmnx format that would (like MusicXML) contain the semantics but no instantiation. Such a format would need a different parser -- or maybe a different version of the same parser. Applications could probably support both parsers without any great effort. This needs to be debated.

Any number of GMNX instantiations may be mapped back to the same CWMNX score. This seems important given that multiple graphical variations (e.g. for different display surfaces) will often be the norm.

Applications that create SVG files can format them in any way they like. I see no problem in creating different instantiations of a particular score for different purposes.

XML Schema documents may be created to describe the CWMNX schema and automatically validate CWMNX documents, or used to power schema-driven parsers and generators. It is not clear how this could be done with a set of conventions on the use of SVG class attributes.

There are ways to automatically validate XML files having multiple namespaces. Its not easy, but it can (apparently) be done. This is a problem that would only have to be solved once, thank goodness! I've seen more official docs about how to do this, but have lost the links. You might take a look at the following: Niklas Tech Blog Microsoft: Schema location

dspreadbury commented 6 years ago

The co-chairs have discussed this issue at length and offer the following summary of their objections to the proposal here to use SVG as the carrier for both CWMNX and GMNX data.

The burden of using SVG as the carrier for CWMNX data is too great. When opening “altCWMNX” files, applications that do not want to render graphics at all, or can render graphics themselves from the semantic information in the file, will have to work harder to ignore the irrelevant SVG elements; when writing “altCWMNX” files, these kinds of applications will likewise have to write irrelevant SVG elements. Forcing all applications that want to work with CWMNX data to also work within the SVG graphical schema is unacceptable to the co-chairs, even if no actual graphics are included in the document.

The “altCWMNX” proposal can not address many of the use cases that the group has identified as important, as detailed in this previous comment. SVG can provide one or more static instantiations of semantic musical data, but it cannot provide the kinds of dynamic, flexible layout and formatting demanded by many of the group’s target use cases. Not supporting this many of the group’s target use cases is likewise unacceptable to the co-chairs.

Finally, the proposal runs counter to the group’s agreed charter: the charter states that the group will define two formats, CWMNX and GMNX, as well as the MNX container format, and the direction of the group’s activities was agreed by consensus and discussion over a long period of time. CWMNX was specifically agreed as a semantic format, using a definition of "semantic" that we agreed on as per issue #50. Combining CWMNX and GMNX into a single format, and using SVG as the carrier for that musical data, is outside the scope of the group’s charter, and is hence unacceptable to the co-chairs.

The co-chairs propose that this issue should be closed based on the above objections. Since the co-chairs already understand @notator’s position, the time has come for other members in the community to provide input and have a chance to speak on this subject.

bhamblok commented 6 years ago

I agree to definitely NOT use SVG as the carrier for both CWMNX and GMNX data. It makes no sense to use a format with a graphical static nature as the carrier for music.

Kalvos commented 6 years ago

I have been following this issue as an observer for quite some time. I hope I'm not speaking out of turn with my first comment here, but some 30 years ago, there was a proposal that, because of the hardware expense, we all learn to read so-called computer-compatible digits. That taught me the lesson about circumscribing approaches to present perspectives, limitations, or expense. So I don't agree that systems and developers in 2018 should limit their approaches. @notator has been in touch regarding one of my scores that may help make his point perhaps more effectively than some of the examples so far, though I'm not sure he will have the time to do that if the issue is closed soon. Please fee free to delete this comment if it's inappropriate.

mscuthbert commented 6 years ago

It would make my life much easier, as a creator of music analysis software that for the most part does NOT directly deal with the graphical representation or audio performance of a score, if I could simply ignore all <svg> tags and not need to worry that they had semantic musical meaning embedded inside them.

clnoel commented 6 years ago

I agree. @notator, the purpose of the CWMNX format is not supposed to be primarily a graphical interchange, since a large portion of the use cases have to do with changing the graphics (reflow), altering the music without display (transpositions), producing audio only, or displaying on alternate platforms and screen sizes.

I, for example, work directly with GDI for display in our music editing program. All the SVG display information would be ignored by my system when importing a score, even though I AM displaying the music graphically. In addition when exporting a score, I'd need to completely rewrite my display commands to produce SVG. Or, I could produce no SVG (as you suggest) in my exported file, in which case I might as well be producing an XML file, which is easier to validate than an SVG with class and namespace tags.

I think it's great that you have found a way to produce SVG from the CWMNX spec, but I think that the SVG system you have created needs to be a consumer of CWMNX and not it's main carrier.

--Christina Noel Musicnotes

notator commented 6 years ago

@dspreadbury said:

The co-chairs propose that this issue should be closed based on the above objections. Since the co-chairs already understand @notator’s position, the time has come for other members in the community to provide input and have a chance to speak on this subject.

I will reply to the co-chair's objections in detail below, but I'm afraid I don't think you can/should close this issue completely. It is the only place where any serious discussion of GMNX has taken place, and there are a number of outstanding issues there. If you close this issue, you (or I) will immediately have to open a new one to cope with those issues. An even better alternative, if you close this issue, would be for you to open a separate repository for GMNX. That way you could keep me out of the CWMNX discussion, and we could all keep better track of the GMNX issues. I'm afraid the chair's proposal to close this issue is a further instance of its well-known bias towards CWMN and the MusicXML. In forgetting about GMNX, you are ignoring both the CG's Charter and the needs of non-CWMN notations (including all the world's other notation traditions, experimental notations developed during the 20th century, and other notations yet to be developed). If you are really only interested in the notation that is edited by Dorico, I would suggest that this should be a proprietary Steinberg forum, and that it should be hosted by them, not the W3C.

I take @clnoel 's points (both here and in #2): Using SVG as the carrier of semantic information for CWMN would make the huge body of existing CWMN software unnecessarily complicated. So we can forget about that idea. On the other hand, the fact that semantic information can be included in SVG files is important for GMNX, for which there is very little, or no legacy software. It also means that CWMN and all other notations have common, programmable ground, and that their relation is one of siblings, not parent-children as CWMN programmers tend to think.

I currently don't really understand the issues raised in #2, and am eagerly awaiting @joeberkovitz' continuation of #57. His FaurReveSample-cwmnx.xml crucially omits the page-layout information, so I need to see that in order to update my FaurReveSample.cwmn.svg. There may well be lessons to be learned for GMNX notations as well. (By the way, thank you @clnoel for not jumping down my throat when I ask a simple question.)

To the chair's objections in detail. @dspreadbury said:

The burden of using SVG as the carrier for CWMNX data is too great.

Point taken, and accepted (see above). As I said above:

I see no objection to defining a simple .cwmnx format that would (like MusicXML) contain the semantics but no instantiation.

Apropos: I think the chair is dodging issue #23. This ought now to be under Active Review. The NAMM 2018 minutes say that @mdgood was going to open an issue about replacing the MNX container with .zip. This has not yet happened.

@dspreadbury again

The “altCWMNX” proposal can not address many of the use cases that the group has identified as important, as detailed in this previous comment.

This is covered by my acceptance of the objection in the chair's previous comment.

@dspreadbury again

Finally, the proposal runs counter to the group’s agreed charter: the charter states that the group will define two formats, CWMNX and GMNX, as well as the MNX container format...

As detailed in #23, the Charter does not include implementation details for any of these formats, so we are not bound to accept @joeberkovitz' current proposals. I think the MNX container is likely to be redundant. Would it, for example, impose the same kind of unacceptable burden on legacy software as the use of semantically enhanced SVG?

Re @Kalvos' remarks: I am currently trying to generalize the concept of a "cursor" as the fulcrum between time and space in music notation. A first application was my use of JSON code in this mp3-SVG synchronization example, but its important that I test the concept in more extreme situations. I therefore contacted @Kalvos to ask his permission to use one of his graphic scores in the public domain. Fortunately, he agreed and this project is currently under way. To paraphrase Sir Harrison Birtwistle (in a speech to members of the music industry, on the occasion of recieving an important award for one of his recordings): "Watch out for the composers. Without them none of you would have a job!" :-)

clnoel commented 6 years ago

So, perhaps, this proposal with svg grouping and container classes will be the way we properly do gmnx, which is still very much in the early stages of formation. Since cwmnx will also need to be represented in gmnx, the systems that @notator has created to make an svg will be the necessary "converter." Because gmnx is necessarily representing graphical scores that should never be reflowed, as well as music from other cultures that would have different reflow rules than cwmn has, we would not worry about those kinds of use cases when in gmnx, but would rather be dealing with a "static" instance, which svg is very well equipped to represent.

In which case, this particular issue should be closed, since we won't be taking cwmnx into altmnx. But @notator is also right about either needing more issues or another repository. GMNX is not getting enough attention, and part of the point was to finally pay attention to it. I really don't want to fall into the trap of hitting CWMNX hard and trying to perfect it and leaving GMNX as the place that we just sort of shove things into to get them out of the way.

Christina Noel

p.s. I don't personally know much about the modern music notation systems, or the needs of other culture's music, so if I overstep or assume something in my ignorance, please educate me! I love to learn!

mscuthbert commented 6 years ago

btw -- am I the only one who has trouble following what all the MNX beyond simple MNX mean? Would it be better to call them "MNX-Classical" "MNX-Graphic" etc.?

joeberkovitz commented 6 years ago

It's great to see more discussion of the question at hand about CWMNX and SVG. I'm only jumping in here to make assurances about the future of GMNX, not to repeat my earlier observations about the altMNX proposal.

It would make me very happy to see the group as a whole pay more attention to GMNX. My personal background didn't seem important to relate before, but perhaps it's worth mentioning briefly. I'm a composer whose work includes many non-CWMN scores, and this was my first, decade-long profession before working as a software engineer and architect for the next 35 years. I have a conservatory education, not a computer science one. And while I'm best known for founding a company (Noteflight) whose success is all about CWMN, that doesn't fully reflect my motivations in working with this group, which are very much aligned with composers working at the edges of what is possible, and with musicians in other cultures who do not employ a European framework for thinking about music.

So GMNX is profoundly important to me, because it seems like part (although not all) of a notation solution that can be culturally and historically neutral. My combined background doesn't make me right about everything, needless to say -- but it does speak to the reason why I proposed GMNX in the first place. I desire to see it succeed, while remaining founded on the best possible principles of software architecture. The other chairs value it also, and I hope and expect we will have more GMNX issues to discuss than we do today. (At present there are about ten).

To wind this up, I've just added a couple more GMNX issues (#74 and #75) to reflect some of the side points made above, because I want these discussions to be guaranteed a life outside #25. Note that the SVG-grouping-class-name issue already exists as a separate issue, #28, which deserves its own discussion.

notator commented 6 years ago

@joeberkovitz Thanks, Joe, for letting us know some more about your background. Birtwistle's remarks apply to you too then! :-) I'd naturally be very happy for this issue (#25) to be closed if GMNX is given its own repository. That seems the best solution, given the current state of the debate.

clnoel commented 6 years ago

@mscuthbert Possibly. As a programmer, acronyms come easily to me, and I tend to store them up.

FYI to those who were also confused by this discussion, CWMN is "Common Western Music Notation", so CWMNX is our new encoding scheme for that. It's my understanding that GMNX is "General" MNX (not "Graphical"), meant to be a scheme by which ANY music score could be visually and audibly reproduced, regardless of culture of origin.

Perhaps, to avoid confusion on that last bit, we rename it GenMNX? It's just as fast to pronounce and moderately clearer in meaning.

Christina Noel Musicnotes

P.S. On a side note, and this is probably the wrong place to ask this, is tablature notation considered part of CWMN for the purposes of our discussions, or will it need it's own representation?

mdgood commented 6 years ago

Contemporary tablature is part of CWMN / CWMNX. Hopefully historic tablature styles can be supported as well - at least as well as in MusicXML if not better.

joeberkovitz commented 6 years ago

@mscuthbert @clnoel Issue #74 exists to carry any forthcoming of better names and explains the rationale (however faulty) for the current names. Let’s carry on the naming discussion there, rather than here.

@notator Issue #75 has been created as a venue for community discussion of the question of a separate GMNX repo.

joeberkovitz commented 6 years ago

The co-chairs agreed today that the community feedback regarding this issue #25 is clear and that the objections to packaging CWMNX in SVG stand. We are therefore closing this issue. We appreciate the original proposal, as well as the healthy discussion that has taken place.

The chairs would also like to append the following additional observations, looking towards the future:

notator commented 6 years ago

Thanks, @joeberkovitz for giving me permission to write to this closed issue. Re #65: As far as I can see, there is nothing CWMN-specific about #65, so I was rather surprised when @mdgood suddenly accused me of being off-topic. I was thinking mainly about GMNX/MNX-Generic files that are (as you say in [1]) based on SVG. GMNX/MNX-Generic files also need an annotation scheme, and annotations happen to be rather easily implemented in SVG, so I thought you ought to be aware of that when considering #65 in future.

We also know that semantically rich instantiations of <mnx-common-western> code can be created in SVG, so my remarks about annotations also apply to such instantiations. Obviously, I was oversimplifying when I said "We seem to have agreed that MNX is going to be based on SVG", and stand corrected by [1]. Everything else is over-reaction.

I continue to think that SVG instantiations must play a central role in any effort to integrate music notation into existing W3C standards, and that SVG namespaces are the mechanism by which to differentiate between the necessarily different semantic structures of the various notations.

[1] @joeberkovitz' comment in #65