search

w3c / mnx

Music Notation CG next-generation music markup proposal.
174 stars 19 forks source link

Create XSD schema document #73

Closed Xenoage closed 1 day ago

Xenoage commented 6 years ago

I'd like to start implementing experimental CWMNX import into my music notation app, http://zong-music.com . As an intermediate step, I want to represent the CWMNX data structures in Java classes.

Can you tell me if an XML Schema (XSD) is planned for CWMNX? If yes, I would try to generate the required code for (de)serializing the XML data from the XSD, otherwise I'll have to write the code by hand. Which way would you suggest?

joeberkovitz commented 6 years ago

Yes, an XSD is most definitely planned. We've been waiting for the main spec structures to settle down first.

I'll retitle this issue to reflect his fact.

webern commented 3 years ago

I have an idea about how XSD production might flow naturally from the current workflow. Currently the workflow is to specify MNX constructs by way of example XML fragments (or documents) and to implement these in a python library here: https://github.com/w3c/mnxconverter. As such, the mnxconverter serves as the reference implementation of MNX. I suggest that the XSD might also be produced as an output of the mnxconverter program.

This might prove especially expedient if the mnxconverter represented MNX in classes. For example an <event> being represented by a python class named Event. The class might hold additional static metadata for the production of an <xs:element> specification in XSD.

The current approach appears more procedural and not OO, so maybe this idea will not prove useful, but I wanted to share it somewhere.

adrianholovaty commented 3 years ago

@webern Thanks for the idea. I'm hesitant to overload the mnxconverter in that way, but it might be doable. Will file the idea away for when it's time to think XSDs. :)

notator commented 3 years ago

77 has come to the conclusion that namespaces (except the default namespace) should not be used in the MNX Schema.

So, unless anyone else wants to do it, I'm now intending to write a Draft Schema after next week's co-chair meeting, when I`ve caught up with the housekeeping, and the other issues I've raised have settled down.

dspreadbury commented 3 years ago

I believe the intention is that we should be able to generate the XML schema for MNX from the planned tool for producing the specification document, so I don't think you should do this now as it will be an unnecessary duplication of effort.

adrianholovaty commented 3 years ago

Yes, James, no need to do that. We're planning to generate a schema automatically.

notator commented 3 years ago

Great! I'm currently concentrating on cleaning up MNX by Example etc. so that your tool will have something good to get its teeth into. :-)

notator commented 3 years ago

Actually, I'm not really convinced that generating the schema automatically is a good idea. Editing a Draft Schema directly would be easier than going through an intermediary application that needs to be programmed around an unstable specification. But there's no danger of duplicating effort. I'm not currently working on this.

evanlenz commented 3 years ago

Have you considered maintaining the schema using RELAX NG Compact syntax and using Trang to generate an XSD? This is best practice in my world, just because RELAX NG Compact is so nice and easy to use (and can still work with XSD simple types if needed). Trang is stable software, tried and true. Here's a tutorial on the syntax: https://relaxng.org/compact-tutorial-20030326.html

Re: "default namespace" have you decided on a namespace URI? I keep hearing "default namespace" and I wonder if we all are on the same page about what that means.

On Tue, Jan 12, 2021 at 3:13 AM James Ingram notifications@github.com wrote:

Actually, I'm not really convinced that generating the schema automatically is a good idea. Editing a Draft Schema directly would be easier than going through an intermediary application that needs to be programmed around an unstable specification. But there's no danger of duplicating effort. I'm not currently working on this.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/w3c/mnx/issues/73#issuecomment-758586290, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAETWLTPMIQLLM574POLSATSZQVE7ANCNFSM4ESJECBA .

notator commented 3 years ago

I understand "default namespace" to be the "target namespace". That's the one defined without a ":" in its name. For example, xmlns="http://ww.jazz.org" in the first example at scienceDirect. Beyond being unique, the value we actually use isn't very important. We could probably use "https://github.com/w3c/mnx".

@evanlenz Thanks for the link to RELAX NG. It looks easy enough to learn, and has some interesting capabilities, especially that it can create schemas from example files. But I'm a bit sceptical:

If the document that is the subject of a PR is in RELAX NG format, then there's an intermediary step between the PR and getting a Schema that can actually be used to verify MNX files. The PR administrator would have to generate and publish the associated Schema, so that it can be used routinely by those using it to work on new examples. The Schema itself has to be publicly visible. This means that it would also be possible to create PRs on it directly. If that is done, admin will have to update the RELAX NG file. That looks error prone and unnecessarily complicated. I think it would be simpler just to have a vanilla Draft Schema, and not force everyone to install any particular software in order to edit it (and create Pull Requests on it). There are different solutions out there:

It should be possible for both admin and non-admin people to create PRs on the Schema, but I suspect that most discussions here will not be centred around the Schema itself, but around the details of which elements to define, and what their contents/attributes are. And those discussions will be centred around concrete examples (that may show errors when validating with the current version of the Schema, indicating that it has to be changed). One shouldn't have to be schema-savvy for that.

notator commented 3 years ago

As I understand it, the main purpose of @adrianholovaty's documentation generation tool is to create a document that replaces our current Draft Spec (see yesterday's co-chair Minutes). The tool is designed to consume (MusicXML) Schemas, not to create them. So, while we are both working from the same data (The Draft Spec and MNX by Example) we are not fundamentally in conflict. On the contrary, I would like to co-operate on the content of our common database (which currently needs some work).

My road map for the next few weeks ends with the creation of a Draft MNX Schema:

  1. Housekeeping
    • continue to correct errors found in the Draft Spec (in the notator:specCorrections1 branch -- PR #213)
    • update MNX by Example (in a new notator:mnxByExample branch, with a corresponding PR)
  2. Create a Draft MNX Schema that can be cross-validated with the examples in MNX by Example. (I've done some tests in this direction already.)
    • Create a PR to add the Schema to the MNX repository (from a new notator:mnxSchema branch)
    • Update all the examples in MNX by Example to reference the new schema.
adrianholovaty commented 3 years ago

@notator I do indeed plan for the new docs tool to generate an XSD schema as well. This would keep everything in one place, with the "single source of truth" being the docs tool's database. As such, I wouldn't recommend spending time making an MNX schema — that work will not actually be used!

The MNX By Example document will also be replaced by this new docs tool. I've already imported the markup/images of the examples, and you can see them (in context of the new tool) at this link. It's greatly improved already, as you can click on each element or attribute value to go directly to the appropriate documentation. As such, I wouldn't recommend spending time housekeeping the old By Example page; it is going away, and I consider it "read-only" at this point.

Apologies that everything is in flux at the moment!

The single best thing you can do to help at the moment is to create some more MNX examples, for purposes of expanding the documentation. For each example, we'd need:

This doesn't have to be in any particular format; even just pasted into the text of a new GitHub issue (one issue per example) would be good enough. The list of specific examples you gave (jumps 3, cross-staff beams, mid-measure repeats, grace notes) looks good to me.

notator commented 3 years ago

@adrianholovaty The important thing now is to get MNX by Example sorted out.

In the long term, it would be easier to find/add examples if they were sorted into folders according to category. The flat list in your current version is going to get very unwieldy when we've got a lot more examples. I'd suggest at least having separate folders for

It would also help to ensure that you don't later have to make unnecessary corrections to your tool, if the new MNX by Example version were to be publicly visible and open to suggestions (like the above). Also I could then help with some of the housekeeping. For example: Once you've decided on the basic structure (for links etc) I could speed things up by doing the mechanical job of transferring the demo code to the correct place. Or unify the scale of the screenshots... That would leave you more time to concentrate on the tool itself.

I'll post the jumps 3, cross-staff beams and mid-measure repeats examples into new issues later today (they are already prepared). I haven't done the grace notes example yet, but its very simple. Will get to that today or tomorrow.

After that, I want to start writing a Draft Schema, even if you are not going to use it. I'm interested in understanding exactly how they work, and the best way to do that is to write one. And, who knows, maybe I can save you some work! :-)

adrianholovaty commented 3 years ago

@notator The new tool's list of examples is 100% dynamically generated via a template. It would be about 10 minutes' worth of work to separate it into separate pages/sections. When it comes to the organization of examples and the underlying system, I'm afraid I need to say "Trust me, I know what I'm doing" on this one.

notator commented 3 years ago

Of course I trust that you know what you are doing! :-) But I would urge you to fix the final addresses for the example code as soon as possible, otherwise we are all going to end up with broken links. Of course, there could be a linear index like the one in your current page. That's up to you. But it would be helpful if the example addresses were fixed early, and then don't change.

On the schema: maybe I'm wrong, but I simply don't think its possible to infer passable regular expressions programmatically from the existing examples and Draft Spec. Inferring the arguments to the elements inside the schema's complexTypes isn't going to be easy either. Hand coding such things is much simpler. If my efforts are wasted, so be it, but the problem interests me, and I've got nothing better to do.

notator commented 3 years ago

I have now created a Draft Schema in the "schemas" branch of my fork of this repository.

The schema itself can be viewed at https://github.com/notator/mnx/blob/schemas/schemas/common1900.xsd It verifies .mnx files stored in folders in the folder https://github.com/notator/mnx/tree/schemas/mnx-examples/common1900/graphicsOnly The parallel musicXML comparisons are stored in folders in the folder https://github.com/notator/mnx/tree/schemas/mnx-examples/common1900/MusicXMLcomparison

The above folders contain code copied from all the (readonly) MNX by Example examples, except the two Jump examples (which I think are incorrect), and graphics-only versions of the code recently presented in #222, #224, #219, and #221, #220, #218. Appropriate attributes have been added to the .mnx files, so that they link to, and can be verified by, the schema.

As far as I know, the code in all these examples is non-controversial, except for

I have not used Actions (#214), MIDI (#216) or Tick-based timing (#217) yet.

A number of issues arose while writing the schema. These have been documented in the comments and can be dealt with later. The schema works correctly with all the .mnx files (Visual Studio says there no errors), but it may, of course, contain typos or other errors. As always, suggestions for improvements would be very welcome.

@adrianholovaty Please note that the Draft Schema, your project and the example code should all be kept in synch as MNX evolves. The schema is complementary to, and not an attack on, your project. It just means that you can forget the last step in your Remaining to-do list.

The Schema is not a replacement for proper documentation. For example: I would like to change the meaning of the measure component in the Draft Spec's measure-location definition. I think it would be better if the measure component (the part before the ':' character) was a relative not an absolute value (1:1/4 should mean 1/4 after the beginning of the bar following the one where the definition occurs, not 1/4 after the first [or second?] bar in the piece.) Such things can't be checked by the schema, they need documentation.

I don't want to get too far ahead of group consensus, so my next step is going to be to create, and integrate, an example containing (graphics only) part groups according to https://github.com/w3c/mnx/issues/185#issuecomment-643174127 and the subsequent discussion. This was proposed by the co-chair last June.

adrianholovaty commented 3 years ago

@notator As I mentioned in the comment above, our plan is to auto-generate this schema from the new documentation system. You are of course free to do this, especially since you say you've "got nothing better to do," but please have some realistic expectations: it's highly unlikely that your work will actually be used. I appreciate your excitement about MNX but fear you're muddying the waters.

As for whether ID references begin with a "#" character: If you think that's a good idea, I suggest creating a new issue proposing it, so we can have a focused discussion. At the moment you're acting as if this idea, proposed a bit off-handedly deep within a GitHub comment thread, somehow became a fait accompli.

For the record, I believe that approach has zero practical benefits and has the downside of being a bit harder to parse (instead of a simple string.split() on the attribute value, now developers would have to deal with stripping out the hash character). And beyond the annoying string parsing, it introduces the possibility of people forgetting to use "#" characters in their MNX documents — which naturally will lead to MNX parsers becoming tolerant of missing "#" characters (just as MusicXML parsers are tolerant of missing/invalid documents), which naturally will defeat the purpose of having the "#" character in the first place. Best to keep it simple.

notator commented 3 years ago

@adrianholovaty you said:

our plan is to auto-generate this schema from the new documentation system.

As I said, I simply don't believe that that is possible. But lets wait and see. We can always compare results later.

In thinking that '#'s are a bad idea, you are in conflict with the expressed opinion of more than one experienced member of this CG. So why don't you join the discussion? For me, absolute priority has to be given to the way the rest of the world handles this. You have to convince us that there is a very good reason for not doing so.

bhamblok commented 3 years ago

@notator, in css the # is used as an id-'targetter'. It is not part of the id itself. For example if you have a css-declaration: #my-id { color: red; } ... than you are targetting the dom-element with <element id="my-id">.

If you make the # part of the id itself, everybody in the css-working-group would protest against you :-) ... You would have to declare ##my-id { color: red; } ... to target <element id="#my-id"> (I wonder if this works though :-) )

Ever wondered why a webpage scrolls down to a specific part of the page, when you append the url with a '#' and some id? That's because you target the anchor-element, with the value you enter after your '#' in the url. For example: https://example.com#my-second-paragraph ... would scroll the webpage down to the dom-element having the id "my-second-paragraph" <h2 id="my-second-paragraph">...</h2>

So a '#' is a 'targetter', not part of the target.

When I think about it, it's similar on Twitter (or any social media platform) ;-) You target a subject with '#' as a targetter. #ILoveMNX, so you're labeling your post with the subject: "I Love MNX"...

notator commented 3 years ago

Hi @bhamblok :-) That's exactly what I'm saying. A string "#evt2" is a reference to the object whose id is "evt2". "evt2" is not a reference to "evt2". Its very important, in all programming languages, to distinguish between objects and references to those objects. HTML, SVG etc use the '#' character. C++ uses a "*". So my schema defines different regular expressions for validating idTypes and idReferenceTypes. Your example <element id="#my-id"> would fail to validate as my idType, because it starts with a '#'.

adrianholovaty commented 3 years ago

@notator Again, if you'd like to propose this "#" idea, please use a new issue rather than appending discussion to this one.

notator commented 3 years ago

We were discussing '#' characters in PR #190, which is still open. Its your turn to say something there.

webern commented 3 years ago

👀🍿

webern commented 2 years ago

Can this be moved to the v1 milestone instead of uncommitted?

adrianholovaty commented 2 years ago

@webern Sure thing — thanks for bringing this up!

adrianholovaty commented 1 day ago

Marking this as closed, for the sake of cleaning up our issue database. This is very outdated — it deals with the older XML version of MNX.

Note that the JSON version of MNX has a JSON Schema that gets automatically updated each time we make changes to the spec. The direct link: https://w3c.github.io/mnx/docs/mnx-schema.json