Reorganize/Rewrite NDR for NIEM 6.0

[TomCarlson-NTAC]

Steps to a NIEM 6.0 NDR:

1. Separate normative from guidance that should go into other support documents
   1. Organize into Base/CFM, XML, etc
      1. Leverage existing work from Dr. Scott
2. Schematron-based rules for XML NDR (concurrent with above, these are a priority for the tooling)
   1. Test against NIEM 6.0
      1. Update or remove as needed
      2. Add any needed additions
   2. Remove Schematron rules to an attachment or appendix
   3. Replace with plain English descriptions
      1. Extract rules text from assert structures, e.g. "A simple type definition MUST be top-level."
      2. Potentially leverage ChatGPT for initial explanations, e.g. "So, in simpler terms, this Schematron pattern is ensuring that in an XML Schema document, any definition of a simple type (like a basic data type) should be placed directly within the schema and not inside another construct. If it finds a simple type definition nested inside something else, it will raise an assertion error with the specified message."
      3. Modify and improve manually.
   4. Pass/fail test cases for new or modified rules
3. Non-Schematron-based rules
   1. Re-word other rules in plain language as needed, many will be fine as-is
4. Resolve and close outstanding issues
5. Reorganize options:
   1. Potential Organization from [[Christina Medlin]]
   2. Reorganize by conformance targets?
   3. Along training lines?
      1. Mirror IEPD process?
   4. Top-down rather than bottom-up?
      1. Group concepts, then rules for that concept?
   5. Don't reorganize
6. Other updates:
   1. Update definitions
   2. Update general text
   3. Request and apply OASIS template
7. Schematron rules for CMF, where useful

[iamdrscott]

on # 1. Separate normative from guidance – I think this is an open question on # 2. Schematron rules: I sorted the existing rules by new conformance target. There are four SCH files in the dev branch; these can be tested now. We need new rules for the open issues and the new appinfo. I will work on those. What is the drop dead date? on # 7. I hope to avoid Schematron rules for CMF, because they can't be translated to CMF.

[iamdrscott]

NDR Rewrite [2024-02-07 draft]

What do we have yet to write?

I think that sooner or later we have to write the following normative specifications:

• RDF entailments for XSD. We have about half of that in NDR 5 section 5. We don't have any of the SHACL/DASH stuff, or any of the new appinfo.
• XML and JSON-LD equivalence. After that, we don't need any of the XML to RDF entailments in section 5, because JSON-LD is an RDF syntax.
• RDF entailments for CMF. If we write CMF to RDF, then we don't have to write a CMF to XSD mapping. (We just provide conversion software that does the right thing.)
• I have all of these in my head, and I bet Eric Jahn can check my work.

Do we want to write mappings for metamodel to XSD and metamodel to CMF? Or do we want the metamodel to just be informative documentation? There are about a dozen NDRs that can be expressed once at the metamodel level, if we write the mappings. But we might want to reiterate them in the XNDR anyway.

• XSD NDR – Mostly done, but take another look at the conformance targets.
• CMF NDR – Mostly captured already in the model (and testable by schema validation).

We aren't writing JSON Schema NDR, because JSON Schema is not a modeling formalism. I think the CMF to JSON Schema mapping is informative. I don't think it needs to go in the new NDR.

I suspect we will eventually need a Message Specification Specifcation, but for now it's just going to be guidance.

We want (early in the document) a How to read this document section and a Explaining the crazy s$!@ you're about to read section. (Titled, perhaps, Overview of the NIEM Technical Architecture. in the final draft.) I think perhaps we can get most of the latter section out of the data modeling paper (section 1), the Understanding the NIEM Technical Architecture paper, and the principles from NDR 5. I can glue that together and add anything else we want in it.

What do we not have to write?

• We don't need the JSON 5.0 specification, though we may get some words from it.
• We can leave the code list specification alone for now, though we may borrow from it or refer to it.
• CTAS just needs a URI and version number change.
• Conformance spec? Perhaps the contents can be distributed into other specs we have to write anyway. (The tricky part may be where to put the no-unnecessary-duplication rule.)
• I have no opinion on the web services API spec.
• I think everything else in reference.niem.gov/niem/specification is OBE and we can ignore it.

When do we have to write it?

Tools are more important than specifications at this point. What specifications are critical path for tools?

• CMFTool doesn't care, I think – either no dependencies, or I'm writing software and specification at the same time.
• Contessa 6.0 (or whatever) needs the final Schematron rules, yes?

The message specification guidance is worth writing as soon as the tools are usable. Or maybe a little before that.

What must be part of NDR 6.0 PS01? What can wait until later?

How do we want to organize it?

• Are we really still writing a Naming and Design Rules document? Would another title be better?
• One document, or several?
  • I don't see the point in putting CMF and XSD into separate documents. They are both NIEM modeling formalisms.
  • We could separate out the things we aren't going to write until later.
  • We could have an Explanation of things you need to know document and a Definition of things you don't need to know document. Sort of like Webb's separate user manual and technical specification, eh?
• Ordering – it surely makes send to put the things that most people do not need, like RDF entailments, toward the back.

How should you respond now?

• If you disagree or question a declarative statement above, please say so -- in email to me, a comment here, or at the next meeting.
• There are several open questions. What is your answer
  • Informative metamodel, or mapped?
  • What can be left out of NDR 6.0 PS01?
  • Is the title still Naming and Design Rules, or something else?
  • One document, or several?

[TomCarlson-NTAC]

Contessa 6.0 (validation functionality in the API) needs the final Schematron rules, yes!