SPIKE: Plan out architecture for making the app use aligned data

[benjore]

Moved to Wiki for future access/edits/maintainability: https://github.com/unfoldingWord-dev/translationCore/wiki/Post-Alignment-Change-Proposal

Alignment Data Changes

• In theory, we can separate concerns by continuing to store target language verses as strings.
• The extra data in the aligned USFM 3 project should be persisted in the alignment reducer.
• Review data to make sure nothing would be lost in the import/export.
• https://docs.google.com/drawings/d/1H5SqAp1RKRKq7bowC1jpRGZrRrHg5RKXEalZAJgqMfU/edit

Previous Projects

• Projects import as regular USFM, whether through tS or tC.
• Projects store targetLanguage verses as strings.
• Can be aligned and store alignment data separately.
• Alignment data is to be exportable as USFM 3.

Aligned Projects

• Projects import as USFM 3 with embedded alignment data in the words object.
• Parsed alignment data is persisted via alignment reducer.
• Parsed verse data could be persisted as a string to maintain backward compatibility.
• Using tools to edit the verse can modify the string but would invalidate some alignments.
• Alignment data is to be exportable as USFM 3.

Invalidated Alignments

• Alignment data matches the BHP version used in the app.
• Uses of the alignment data such as scripture pane and alignment tool need to gracefully handle changed BHP data.
• Edited verses in tools like Autographa and translationWords invalidates alignment data.
• App and Tools need to gracefully handle invalidated alignments due to verse edits.
• Export needs to be able to validate alignments.

Inline Markers such as Footnotes

• Inline Markers need to be maintained in the target language verse text to ensure they are not lost.
• Inline Markers need to be filtered dynamically when handled in Alignment tool.
  • Footnote text should be filtered out, other marker data should be left in but not the marker itself.
  • This allows the alignment to function on the verse text but bypasses the non-verse text.
  • There may be other markers to be treated as footnotes, but not sure which ones are used.
  • Handle footnotes in a way that can be extended in case others arise.

USFM.js

• Current parsing doesn't maintain punctuation, only word objects.
• Needs to maintain data like footnote markers and footnote text on import/export.
• Needs to maintain markers like quotes that are inline and have verse text in them.

Resource Bibles

• Gateway Langauge Resource Bibles should now have alignment data but not all of them.
• Importing Resource Bibles needs to maintain alignment data.
• How do we store and load GL Resource Bibles so that the alignment data can be used?
  • Store as an array of word objects like Primary Languages like Greek/Hebrew.
  • Store as USFM 3 string and dynamically parse when used like in Scripture Pane?
  • This deviates too much from how we handle Primary Languages like Greek/Hebrew.
  • Maintaining alignment data outside of word objects does not seem practical here.

Scripture Pane

• Primary Bible to be the BHP, the Primary Language.
• Highlighting phrases from the tool needs to be the Primary Language.
• Gateway language Bibles with alignment data look to see if a word has been aligned to the highlighted Primary Language word object.
• Languages without alignment data or no aligned matches found will not highlight any words.

Verse Check

• Maintaining target language as strings ensures backward compatibility.
• Verse edits may invalidate alignments but may not be a concern here.
• Footnotes and inline markers may need filtered on view but not on edit to ensure they are not lost.
• Selections made can populate the word alignment reducer.

Word Alignment Tool

• Validate alignments as changes may have invalidated some.
  • The verse may have been edited since last aligned.
  • Primary Language version may have been updated.

TranslationWords Tool

• The tool needs an overhaul to base check off of the Strongs number associated with the check.
• Quote needs updated to be the Primary Language word via Strongs number instead of ULB.
• Check Card needs to display the Primary Language word and Strongs number along with GL.
• ContextIds in GroupData needs to be based on Strongs and Greek word in Quote.

Tools GroupData Menu

• The previous tool checking menu is showing English words from the article.
• Is this still the expected behavior? It seems as though it should be.
• Support for showing translated article titles in the menu, when source tool articles are translated.

Core Helpers

• May need core helpers for common functionality.
• Pivot a verse that is an array of word objects to a string and alignment data.
• Pivot a verse that is a string and alignment data to an array of word objects.
• Finding Gateway Language words to be highlighted with the provided Primary Language word.

Core Actions

• May need new actions for alignment prediction.
• Any actions affected by having associated alignment data with Resource Bibles or project?

Core Reducers

• May need a new reducer for alignment prediction.
• Any reducers affected by having associated alignment data with Resource Bibles or project?

Alignment Prediction

• A new tool may need to be created to keep the code maintainable.
• May need a new reducer to handle alignment prediction look-ups efficiently.
• Alignment prediction can be included in Verse Check tool to predict selections.

[klappy]

Tasks:

[x] PostAlignment: Team to discuss changing aspects of data and areas to explore [x] PostAlignment: Explore changes for tC_Resources Importing (Manny) [x] PostAlignment: Explore changes for preserving punctuation (Klappy) [x] PostAlignment: Explore changes for Project Importing (Jay) [x] PostAlignment: Explore changes for Project Exporting (Jay) [x] PostAlignment: Explore changes on Core Helpers (Klappy) [x] PostAlignment: Explore changes on Core Actions (Klappy) [x] PostAlignment: Explore changes on Core Reducers [x] PostAlignment: Explore changes for App Data flow [x] PostAlignment: Explore changes for Tool Data flow [x] PostAlignment: Explore changes on Scripture Pane (Manny) [x] PostAlignment: Explore changes on Verse Check (Manny) [x] PostAlignment: Explore changes on translationWords tool (Klappy) [2] PostAlignment: Explore changes on wordAlignment tool [x] PostAlignment: Explore changes on USFM.js library (Klappy) [x] PostAlignment: Team to review proposed changes [6] PostAlignment: Team to explore holes and potential remaining issues [4] PostAlignment: Explore potential holes [4] PostAlignment: Explore remaining issues

[mannycolon]

Changes for tC_Resources Importing

• Create a script or function that parses data from USFM 3 to the desired bible resource format used in tC (Chapter, Verse, word Objects format)

Changes on Scripture Pane

• Determine how to highlight words for the current check in the Greek bible resource using alignment data
• Determine how to highlight words for the current check in the ULB bible resource using alignment data
• Gracefully handle showing or not showing word details (lexicon information) for non-Greek bibles resources.

[mannycolon]

Changes on Verse Check

• No additional findings besides what was already stated above.

[klappy]

Preserving Punctuation

resourcesReducer.bibles.targetLanguage[1][1] #=> "Jesus, wept."

[
  { word: 'Jesus', attributes: '...' },
  { text: ', ' },
  { word: 'wept', attributes: '...' },
  { text: '.' }
]

There are quite a few places in the code base that join the BHP words into a string. If we were to create a helper for doing this it would be in a single place, and extend it for things other than words/punctuation.

[
  { word: 'Jesus' attributes: '...' },
  { text: 'Christ. Here is the rest of the verse.' },
  { footnote: 'Here is a foot note that needs to be preserved but not words in the verse.' }
]

[klappy]

Core Helpers

• Pivot a verse that is an array of word objects to a string and alignment data.
• Pivot a verse that is a string and alignment data to an array of word objects.
• Finding Gateway Language words to be highlighted with the provided Primary Language word.
• Convert verses that are array of word objects with punctuation
• CSV Export - to prepare alignment data for CSV export
• Project Details Helper - Tool Details - Calculate Progress to handle alignment data for word alignment tool (maybe relocate?).
• Project Validation - validate alignments?
• Selection Helpers - occurrence/occurrences in a string... if used on ULB that is going to change.
  • May not be needed since word objects include occurrence/occurrences.

[klappy]

Core Actions

• CSV Export - actions to export alignment data if needed
• GroupDataActions - explore changes to completed verse alignment not from empty word bank
• Import Local - handling aligned USFM 3 files and pivoting into verse strings and alignment data
  • word markers with alignment data populates alignments
  • text/word markers without alignment data populate the word bank.
• Merge Conflict Actions - support aligned usfm 3 files
• Resource Loading - always load/generate alignment reducer for uses with selections
• Project Details - Do we want to show if alignment data is available?
• Project Validation Actions - check/validate alignment data
• Project Selection Actions - load/generate alignment data
• Resources Actions - always load/generate alignment data for other uses.
• Selections Actions - leverage selections as alignments and alignments as selections
• Target Language Actions - update USFM import/export to handle alignment and USFM 3.
• USFM Export Actions - support alignment data and USFM 3, also support USFM 2 w/o alignment.
• Word Alignment Actions - Generate selections when alignments are created?

Core Reducers

• No changes needed from what we can tell.

Other Changes

• Rename BodyUIActions to something more specific.
• LoaderActions - see if sendProgressForKey is really being used? Do we wan to finish this?
• ModalActions - see if it is needed and remove, tools has its own system.
• OnlineModeAction - see if it is needed and remove
• Recent Projects Actions vs My Projects, do we need "recent" projects or just My Projects?
• Remove all code and components related to old modal.
• SideBar Actions - Update naming conventions for consistency to GroupMenu and fix it.

[klappy]

translationWords Tool

• The tool needs an overhaul to base check off of the Strongs number associated with the check.
• Quote needs updated to be the Primary Language word via Strongs number instead of ULB.
• Check Card needs to display the Primary Language word and Strongs number along with GL.
• ContextIds in GroupData needs to be based on Strongs and Greek word in Quote.
• Multiple occurrences of a strongs number in the Greek to be verified how to handle.
• False positives where strongs number may occurr.
• Scripture Pane Greek highlighting to be done via Strongs number in check and should be present in the contextId, potentially the Greek word as quote as well.
• Check Info Card to additionally pass Greek info into it and needs design on how to display.
• Show greek lexical information in the Check Info card or anywhere else?

Other Changes

• Remove tons of defunct code no longer in use.
  • Folders: filters/js/scripts/translation_words/utils
  • Files: loader.js/USFMParse.js
• Fix linting errors

[klappy]

USFM.js

• Bidirectional support for mixing word markers and non word markers
  • Preserve punctuation through support of mixed word markers and text
  • Support for all usfm markers
• Add as few test use cases as possible to have good coverage of issues.

Scripture Pane

• Add support for non-word marker objects that come from non-word markers.

Other Changes

• Exports in index.js to include more than just select functions?