XML validation and related matters

[DavidHaslam]

@ThomHehl

• The top level file moffatt.xml fails to validate to the OSIS 2.1.1 schema!
• This seems to be a template rather than a complete file. There's no usable content.
• The individual files in the xml folder are incomplete as XML files.
• Some of these files also contain numerous errors.

General points

• The fact that a file passes XML syntax check does not mean that it's correct.
• Syntax checking is merely a first step that must pass before a file can be validated.
• Even a file that validates to the OSIS schema may contain other errors.

[DavidHaslam]

I have developed a bespoke TextPipe filter to convert the 12 existing XML files to USFM.

In the course of this activity:

• A few minor typos were discovered and fixed
• Chapter & verse elements were converted with the appropriate numerical values
• Incorrectly cased XML attributes were processed as if they were OK
• Incorrect XML element <lq>...</lq> was treated as <lg>...</lg>
• Milestone elements were replaced by USFM footnotes \f - ... \f* with no caller symbol
• Verse text was joined where possible and sensible
• Soft hyphens were removed
• Multiple hyphens were replaced by a horizontal bar
• Some sensible adjustments were made to the resulting USFM (q.v.)

USFM adjustments

• Move quoted text start tag into the poetry line
• Move quoted text end tag from the start of line
• Move quoted text end tag from the start of verse
• Insert a space before a quoted text tag
• Ensure that every poetry tag \q is at the start of a line
• Move a verse tag in poetry to after the poetry tag
• Replace each paragraph tag within a poetry passage by the blank line tag \b
• Remove the paragraph tag before a poetic passage
• Add USFM file header lines – see #8

[DavidHaslam]

The activity revealed that in some books, there are passages with anomalous markup.

• Several passages wrapped with italics tags (some of these are lengthy passages)
• Several passages wrapped with square brackets (sometimes double; not always matched)

IMHO, it would be sensible to remove all 192 pairs of italics tags completely. Then review which text should be marked with italics, if any. NB. My TextPipe filter now optionally removes all the italics markup.

Likewise, the use of square brackets should be throughly reviewed.

[DavidHaslam]

After the conversion, there are 26 locations where a paragraph tag \p has text following on the same line.

Here are the search results:

\p Then the Pharaoh woke up, to find he had been dreaming.
\p Then Joseph went through the land of Egypt.]
\p Moses covered his face; he was afraid to look at God.]
\p Aaron said nothing.
\p As the Eternal commanded Moses, so he did.
\p The Israelites who left Egypt were as follows:
\p You must place it before the Eternal your God, bowing down before the Eternal your God
\p For forty years the land lay safe.
\p he went to his father’s house at Ophrah and there, on a single stone, he murdered seventy men, his brothers, the sons of Jerubbaal. Jotham, the youngest son of Jerubbaal, alone survived, for he hid himself.
\p and when she arrived, her mother-in-law said, \qt1 How have you fared, my daughter? \qt1* She
\p They had just entered the gate-house, when there was Samuel, coming out in their direction, on his way up to the height!
\p deserting him.
\p Saul numbered the people who were left with him, about six hundred men.
\p On that day Saul made a vow and laid an oath on the troops, saying, \qt1 A curse on any man who eats any food before evening, till I take vengeance on my enemies! \qt1*
\p David and his men were in the steppes of Maon, in the wady of the Arabah to the south of the Jeshimon.
\p Tamar lived on in the house of her brother Absalom, all forlorn,
\p So David chose the pestilence.
\p The Eternal's angel was beside the threshing-floor of Araunah the Jebusite.
\p Now Ahijah could not see, his eyes were so dim with age.
\p He came upon Elisha the son of Shaphat as he was ploughing behind twelve yoke of oxen, Elisha being next the twelfth pair. Elijah went across to him and flung his mantle over him;
\p When the man of God spied her at a distance, he said to his servant Gehazi, \qt1 Look, there is the Shunammite!
\p So Naaman started off, taking over four thousand pounds in silver and over twelve thousand pounds in gold, with ten festal robes:
\p He had gone only a short distance,
\p Ahaziah the son of Jehoram king of Judah went down to visit Jehoram the son of Ahab at Jezreel, where he lay ill.
\p The king's descendants, numbering seventy, were living with the chief men of the town, who were in charge of them;
\p And then Zedekiah rebelled against the king of Babylon.

A few of these look as though they should not be the start of a paragraph.

[DavidHaslam]

Verses out of order

This exercise also reveals a much more serious problem.

The translation contains passages in which the verses are not in the proper order of the common KJV versification scheme.

Although this might be OK in a printed edition, it's certainly a major problem for an electronic edition.

OSIS and USFM do not support verses that are out of sequence.

The OSIS might look OK and may even validate with out of sequence verses, but the conversion to a SWORD module using osis2mod would simply reorder the verses according to their osisID value, and any duplicated verses would be overwritten by the later occurrence.

This is a very critical problem – one that requires a major work around.

Here's an example of such a passage:

\c 8
\p
\v 1 But God remembered Noah and all the living creatures and the animals that were with him in the barge; God made a wind blow over the earth, till the waters abated;
\v 2 also, the fountains of the abyss and the sluices of heaven were closed:
\v 3 and at the end of the hundred\f - page 8\f* and fifty days, the waters began to subside.
\v 4 In the seventh month, on the seventeenth day of the month, the barge grounded on the mountains of Armenia.
\v 5 Till the tenth month the waters steadily subsided, and on the first day of the tenth month the tops of the mountains were seen;
\v 13 on the first day of the first month of the six hundred and first year, the waters had dried off the earth;
\v 14 on the twenty-seventh day of the second month, the earth itself was dry.
\p
\v 6 At the end of forty days the
\v 2 from heaven ceased,
\v 3 and the waters went hack steadily from the earth.
\v 6 Then Noah opened the window he had made in the barge,
\v 7 and sent out a raven, which went flying here and there till the waters had dried off the earth.
\v 8 He waited seven days and then sent out a dove, to see if the waters had drained off the earth;
\v 9 but, as the dove could find no rest for the sole of her foot, she flew back to him in the barge; he put his hand out, caught her and took her in beside him in the barge.
\v 10 After waiting seven days more, he sent the dove out of the barge again;
\v 11 in the evening the dove came back to him, and there, in her beak, was the fresh leaf of an olive! So Noah knew that the waters were drained off the earth.
\v 12 After waiting seven days more, he sent out the dove, but she never came back to him.
\v 13 Then Noah removed the covering of the barge and looked out. There lay the surface of the ground, all dry!
\p
\v 15 Then said God to Noah,
\v 16 \qt1 Leave the barge, you and your wife and your sons and your sons’ wives,
\v 17 and take out every\f - column 2\f* living thing that is with you, every creature, bird and beast and reptile, that they may swarm on earth and be fruitful and multiply on earth. \qt1*
\v 18 So Noah came out, along with his sons and his wife and his sons’ wives;
\v 19 also every beast, every reptile, every fowl, everything that moves on earth, all came out of the barge, arranged in families.
\p
\v 20 Then Noah built an altar to the Eternal; he took some clean beasts and birds of every kind and burned them in sacrifice on the altar,
\v 21 and when the Eternal smelt the soothing fragrance, the Eternal said to himself,
\qt1 Never again will I curse the ground, though the bent of man's mind is indeed towards evil from his youth; never again will I destroy every living creature as I have done.
\v 22 So long as earth remains, seedtime and harvest, cold and heat, summer and winter, day and night, shall never cease. \qt1*

Observe that several verses are out of sequence! Also observe that there are two separate lines tagged as verse 2 and two as verse 13.

[DavidHaslam]

Observe too that text in the above chapter omits some words and mispells others.

[DavidHaslam]

@ThomHehl

Had I not performed this back conversion to USFM, it would have been much harder to discover all the latent problems in the OSIS XML.

USFM files are generally much easier for us humans to read and see where the problems are.

It also helps when the USFM tags are syntax highlighted, as is the case with my own USFM user defined language file for Notepad++.

NB. I plan to commit the 12 resulting USFM files to my branch before issuing a pull request.