Removal of duplicate resources

[WJSchakel]

Resources were sometimes copied to other projects or test packages as resources were not shared between them. With the new coupling, duplicate resources can be removed.

[averbraeck]

There is no issue removing the xsd files and include xml files for running a model in a derived project that is dependent on ots-parser-xml. It can nicely find the XSD definitions and the XML include files, using ../xsd/ots.xsd and, e.g., ../xsd/defaults/default_gtutypes.xml.

[WJSchakel]

Can the resource also be found by the dependent project? In ots-editor the following line gives a null url.

URL url = URLResource.getResource("../xsd/ots.xsd");

[averbraeck]

It is, actually, quite a struggle. There are different situations I am testing right now:

1. Referencing an XSD for validation in the Eclipse editor when editing an XML file
2. Including another XML in the Eclipse editor when editing an XML file

3. Referencing an XSD for validation during the execution of a program in Eclipse that parses an XML file
4. Including another XML during the execution of a program in Eclipse that parses an XML file
5. Finding / parsing an XSD during the execution of a program in Eclipse that executes as a regular program

6. Referencing an XSD for validation during the execution of a unit test in Eclipse that parses an XML file
7. Including another XML during the execution of a unit test in Eclipse that parses an XML file
8. Finding / parsing an XSD during the execution of a unit test in Eclipse that executes as a regular program

9. Referencing an XSD for validation during the execution of a unit test in Github Actions that parses an XML file
10. Including another XML during the execution of a unit test in Github Actions that parses an XML file
11. Finding / parsing an XSD during the execution of a unit test in Github Actions that executes as a regular program

12. Referencing an XSD for validation during the execution of a program in an executable JAR that parses an XML file
13. Including another XML during the execution of a program in an executable JAR that parses an XML file
14. Finding / parsing an XSD during the execution of a program in an executable JAR that executes as a regular program

15. Referencing an XSD for validation during the execution of a program in another project that uses ots-parser-xml as a jar in the pom file that parses an XML file
16. Including another XML during the execution of a program in another project that uses ots-parser-xml as a jar in the pom file that parses an XML file
17. Finding / parsing an XSD during the execution of a program in another project that uses ots-parser-xml as a jar in the pom file that executes as a regular program

@WJSchakel : Your situation would right now be number 5, and later 8, 11, 14 and 17. Correct?

[averbraeck]

@WJSchakel : From the above, right now Item 1 works, Item 2 is a nightmare. I will come back to it in a separate comment. Items 3 and 4 work. Item 5 is your case that probably needs URLResource.getResource("/resources/xsd/ots.xsd") or URLResource.getResourceAsStream("/resources/xsd/ots.xsd") to retrieve the XSD file that is present in the referenced ots-parser-xml project. Items 6 to 11 still need to be explored and tested. Items 12 to 14 are very difficult. I cannot generate an executable jar anymore of the Eclipse project that includes the resources. I tried regular export from Eclipse, maven-jar-plugin, maven-assembly-plugin, maven-shade-plugin. Each have their own issues, and I have not found a single solution that is satisfactory at the moment. I will make a separate issue for this. Items 15 to 17 still need to be explored and tested.

[averbraeck]

Interestingly enough, situations 1 to 17 all worked reasonably well when referencing an on-line version of the XSD files and XML include files, using a sentence like:

<xi:include href="https://opentrafficsim.org/docs/xsd/1.03.00/defaults/default_gtutypes.xml">

Of course this is still possible using either the old solution pushing the XMLs or XSDs to opentrafficsim.org. Situation 2, however, does not work anymore, no matter how I try. Even a LOCAL include in the same folder does not work in the Eclipse editor. I am going to try with a fresh installation of Eclipse and a new, independent project. Something might be broken or set wrongly in the settings of the OTS project...

[averbraeck]

Items 12 to 14 are very difficult. I cannot generate an executable jar anymore of the Eclipse project that includes the resources. I tried regular export from Eclipse, maven-jar-plugin, maven-assembly-plugin, maven-shade-plugin. Each have their own issues, and I have not found a single solution that is satisfactory at the moment. I will make a separate issue for this.

See Issue #60

[WJSchakel]

@WJSchakel : Your situation would right now be number 5, and later 8, 11, 14 and 17. Correct?

I think my current situation is a combination of 5 and 17. Aren't 3 through 14 also all applicable referring to the same, or another project? (Almost doubling the number of cases...)

Given the level of complexity, perhaps we can be more pragmatic. Having a base-usage with online resources, and a known manual procedure to make things work locally during development; even if only at 1 computer. By "manual procedure" I mean referring to absolute local paths, in all likelihood.

[averbraeck]

Aren't 3 through 14 also all applicable referring to the same, or another project? (Almost doubling the number of cases...)

They are actually quite different. As examples, finding a file inside a JAR file, for instance, is very different from finding that same file in the file system. Unit tests have to access the files from /src/test/java which is a different path, and a different place (/target/test-classes) where the classes are stored) than /src/main/java (/target/classes); and it is not guaranteed that files from a different project end up in the right place for the unit tests. This is completely different in Eclipse and on a Linux system (Github Actions) where the unit tests are called from a terminal unsing the maven program. Etc. So, the fact it works in one case is no guarantee it works in any of the others (as I already found out the hard way).

[averbraeck]

Given the level of complexity, perhaps we can be more pragmatic.

For sure! But, e.g., Item 2 does not even work with an absolute path. Even worse, for me Item 2 does not even work anymore when I place the file in the SAME folder... So, we probably messed up a setting in Eclipse (that has propagated through Git) that I need to set differently. I know that I worked for days solving it before, where I ended up in pushing the files to opentrafficsim.org, and especially item 2 caused me a lot of trouble back then. I think I solved it by adding the files to some catalog in Eclipse. But that cannot be done project specific.

[averbraeck]

Making progress on item 2 above. I made a minimal reference project where the error occurred. It turns out that the include fails (and reports that it cannot find the file, where actually, it cannot include the file for other reasons) when either the included file has an error, or the inclusion itself leads to an error in the resulting file after the inclusion. A clear example is an included file that starts with:

<?xml version="1.0" encoding="UTF-8" ?>

This cannot appear in the middle of an XML file, which will happen if you include that file. Yet, if you have an included file that does NOT start with the xml version tag, and the error definitions are strict, THAT file gives an error, and as a result cannot be included. So, a combination of releasing some checking rules with leaving out 'offending' lines, can work in the end (I tested this in a minimal example in a separate project).

A second issue has to do with namespaces in XML. I saw that the included file has to be 100% strict on using the namespace lead tag. Don't know exactly why, but for OTS this would mean, as an example, that GTUTYPE definitions should be mentioned as ots:GTUTYPE in the defaults files, which is not a problem of course.

[averbraeck]

I read hundreds of posts and pages about XML, XML Schema files, XPath, XInclude, key/keyref validation, and XML validation in general. Two main conclusions:

1. What we try to do is quite unique; I could not find a single implementation that combined the techniques we use... Even the use of key / keyref is almost non-existent.
2. The techniques are not compatible. I wrote a minimal project that replicates the problem for item 2 above, and clearly shows the flaw. See below.

In a nutshell, the problem s as follows:

• We use XInclude to include XML files with default definitions
• We use key/keyref with XPath to test usage of keywords against the definitions in the included files as well as in the non-included files
• The order in which XInclude and building of the set of keywords is done, is not specified; each program can choose to implement when to include at any moment
• In Eclipse, the order is wrong. When you include a file, the definitions contained inside are not recognized. When you copy the code inside the XML, they are (XInclude copies the code into the XML, so there is no difference between doing this manually and using XInclude).
• If there is only the slightest mistake in the included file, the errors presented are the same (e.g., key not recognized), but then as a result of not being able to include the file. It is almost impossible to see the difference between mistakes in the include file, and the error of the wrong order. It took me 3 days to find out.
• Our earlier versions with external XSD and include XML files seemed to work, but I tested it and found out that they did not check at all -- the inclusion contained errors, and validation was turned off on our old project to prevent red crosses in our XML code.

Way forward:

• Keep the checking version of the schema files, because (1) an external check can force the XInclude action to be performed before the building of the validation key sets. Also, (2) when XInclude is not used, the key checks work fine. So, a first workaround is to just COPY the default definitions into the XML instead of including them.
• Create a second version (it's only the file ots.xsd that is different, and it is just a matter of keeping the first 36 lines) of ots.xsd that does not have built-in checks, E.g., ots-nocheck.xsd that can be used in the editor. For the parsing, it makes no difference. So, workaround 2 is to include a no-checking schema file.
• For ots-parser-xml, this all makes zero difference. It happily parses the files with the use of a special EntityResolver class for the includes, where I circumvent any issues. So, Parsing works fine.
• To create a better grip on the namespaces (necessary for XPath and in a sense also for XInclude), I included the namespace with all tags. This means that instead of GTUTYPE ID="CAR" /> we have to specify ots:GTUTYPE ID="CAR" />, etc. This is also the recommended way, not to use a blank namespace, especially in complex projects.

[averbraeck]

There is now a good workaround for Eclipse.

XML files start with:

<ots:OTS xmlns:ots="http://www.opentrafficsim.org/ots" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://www.opentrafficsim.org/ots 
        ../../../../../ots-parser-xml/src/main/resources/xsd/ots.xsd"
    xmlns:xi="http://www.w3.org/2001/XInclude">

The includes are done as follows:

<xi:include href="../xsd/defaults/default_gtutypes.xml">
  <xi:fallback>
    <xi:include href="../../../../../ots-parser-xml/src/main/resources/xsd/defaults/default_gtutypes.xml" />
  </xi:fallback>
</xi:include>
<xi:include href="../xsd/defaults/default_linktypes.xml">
  <xi:fallback>
    <xi:include href="../../../../../ots-parser-xml/src/main/resources/xsd/defaults/default_linktypes.xml" />
  </xi:fallback>
</xi:include>
    <xi:include href="../xsd/defaults/default_lanetypes.xml">
   <xi:fallback>
    <xi:include href="../../../../../ots-parser-xml/src/main/resources/xsd/defaults/default_lanetypes.xml" />
  </xi:fallback>
</xi:include>

This ensures that the Eclipse editor can find the definitions and includes (through the <xi:fallback> tag). Eclipse runs and executable jar files can find the includes through the <xi:include> tag, and the ots-file through the provided <xsi:schemaLocation>.

The nice thing is that the Eclipse XML editor and validation work properly, so hints and explanations are given in the Eclipse editor. The key/keyrefs for included elements give an error during editing, but that disappears as soon as the editor is closed.

Items 1, 2, 3, 4, 5, 12, 13, and 14 from the comment above have now been tested and work properly.

[averbraeck]

Tested items 6, 7, 8 above successfully with a unit test in the ots-parser-xml project.

[averbraeck]

Tested situations 16 and 17 above successfully. 15 does not and can not work, unless we provide the xsd files on a network location.

Situations 9, 10 and 11 can only be tested when we deploy a next version of OTS. All other situations have been tested.

In total, we now have a workable situation w.r.t. the XSD files. They are available only in the ots-parser-xml project, and not in any other location. Setting up an XML file works, and parsing and executing works in Eclipse, Jar files, and other projects that are dependent on ots.