See Ghost characters when fixing Hebrew alignments

[PhotoNomad0]

Reported by users - See Ghost characters when fixing Hebrew alignments. Could to be due to issue when word is changed in original language after alignment. See comments below. This will need the steps to reproduce This will need testing notes

Example:

• Note that in the example יְנֻגָּֽעוּ is duplicated and some letters are replace with ? symbol

[PhotoNomad0]

Notes from user: Has anyone seen this happen before? Notice in the aligning window the Hebrew word with 2 question marks in it instead of Hebrew letters and also notice that it is effecting the highlighting in gE since the phrase "they do" should be highlighted in the UST but is not. What is strange is that a few minutes ago it was a different Hebrew word in this verse that had two question marks in it but the question marks switched words after refreshing. I just refreshed again and now the question marks are staying in the same Hebrew word and effecting the highlighting. I’m stumped.

I believe the way to recreate this is to change the form of a word in the Hebrew Bible but then have the old form in the alignment data for the ULT and/or UST. At least that's how I've had this problem before.

When I had this problem, I unaligned the whole verse, saved that, and then realigned from scratch.

[PhotoNomad0]

Developer Notes:

• it looks like this will have to be fixed in word-aligner-rcl
• in the branch auto-Grant_Ailie-PSA, it looks like the UST alignment to the original language was changed from אָ֝דָ֗ם to אָ֝��ָ֗ם. The Unicode 05D3 was replaced with two FFFD characters. So the original language was not changed, but GWE changed the original language word. When it saved the alignment.
• Still trying to reproduce the steps involved to recreate the issue. It looks like it is more complicated than just a change of the original language words. Determined this by reviewing the changes in grants branch of the en_ust for that verse and comparing to hbo_uhb master branch (which had the verse changed last in 2020). There is no obvious reason how an original language letter is being replaced in the alignments

[PhotoNomad0]

@birchamp @elsylambert - In tCore we do alignment validation each time the repo is opened. So at a minimum It looks like we need to improve the alignment validation in GWE. Words that are not in the original language should not appear in the alignment tool.

[PhotoNomad0]

@birchamp @elsylambert - In tCore we do alignment validation each time the repo is opened. So at a minimum It looks like we need to improve the alignment validation/migration in GWE. Words that are not in the original language should not appear in the alignment tool.

[PhotoNomad0]

Moved the validation and migration work into a separate issue https://github.com/unfoldingWord/gateway-edit/issues/565 . Will leave this open until we can reproduce the corruption of the alignments seen.

[PhotoNomad0]

@elsylambert @danielklapp - Maybe move this to QA watch. This should not happen again. But since we haven't been able to come up with a way to reproduce, it could reappear.