Support loading from an index of `.exp` files or a directory that contains `.exp` files

[ronaldtse]

Expressir should support loading multiple schemas two ways: (as per 2021-01-18 discussion with @TRThurman @brandonsapp)

1. Given a directory, read from all the .exp files inside, the REFERENCE FROM links are to be resolved to schema paths considered they are under the same directory (e.g. file ./foo.exp has REFERENCE FROM bar, means that bar.exp is under ./ as well).
2. Given an index that links to a set of .exp files, read those files. For example, the data/library/arm_index.xml file in iso-10303-stepmod. (arm_index.xml.zip)

Which looks like this:

<express_index >
    <!-- Schemas -->
    <schema name="Activity_arm" directory="activity" type="module" />
    <schema name="Activity_as_realized_arm" directory="activity_as_realized" type="module" />
...
    <!-- Schema: Activity_arm -->

    <!-- Defined types -->
    <type name="activity_item" directory="activity" type="module" />

    <!-- Entities -->
    <entity name="Activity" directory="activity" type="module" />
    <entity name="Activity_relationship" directory="activity" type="module" />
    <entity name="Activity_status" directory="activity" type="module" />
    <entity name="Applied_activity_assignment" directory="activity" type="module" />
...

The arm_index.xml format contains data that Expressir does not need. It would be cumbersome to reverse engineer and re-implement express_index.

Question for @TRThurman :

• Can we just go a simple (YAML) index that contains a list of file paths of .exp files to read from?

[TRThurman]

Given a directory, read from all the .exp files inside, the REFERENCE FROM links are to be resolved to schema paths considered they are under the same directory (e.g. file ./foo.exp has REFERENCE FROM bar, means that bar.exp is under ./ as well).

Also need to resolve the USE FROM links.

Can we just go a simple (YAML) index that contains a list of file paths of .exp files to read from?

Not sure I understand the question. The use of that specific index is only for performance reasons in the publication process. Note that because it is the result of an external process, it may not be current. Sometimes we move ENTITY declarations from one file to another, so the build process will need a link check to ensure link validity.

[ronaldtse]

@TRThurman right, the keyword here is "validation". We want to ensure that a schema is valid with all the REFERENCE/USE FROM statements leading to proper locations/schemas and entities.

[zakjan]

Loading multiple files into a single Repository instance is implemented here in Expressir with Parser.from_files(array) method.

The knowledge of parsing iso-10303-stepmod directory structure to an array of files is implemented in stepmod-utils by reading repository_index.xml. https://github.com/metanorma/stepmod-utils/blob/master/exe/stepmod-find-express-files It can be enhanced to support other index files.