RFC: tC + tS Interop Proposal

[da1nerd]

UPDATE: See new specification in this comment below.

Use case: translationCore and translationStudio need to be able to open and edit each other's projects.

Requirements

• Textual changes made to a project from either app must be recognized by the other. e.g. changes to wording in a verse must be recognized, whereas alignment data need not be recognized.
• The existing project structure for tC projects and their manifest files must not change. tS projects will be migrated to a compatible form.
• Require the minimum number of changes to projects and applications to support this.

Proposal/Progress

TLTR; content changes will be shared through the top level usfm file.

• [x] Migrate tS project data into .apps/translationStudio
  • [x] Remove tS specific metadata from the top level manifest file.
  • [x] Generate top level usfm file as seen in tC projects. Except it will be in usfm 2 instead of 3.
• [x] Support importing tC projects into tS by parsing it's top level usfm 3 file.
• Support importing tS projects into tC by parsing it's top level usfm 2 file
• The top-level usfm file will be located at ./[lang]_[resource]_[book]_book.usfm. For example en_ult_rut_book.usfm
• Applications will update the generator block in the manifest file for imported projects. e.g. projects exported/uploaded from tS will have a generator name of ts-desktop, and projects exported/uploaded from tC will have a generator name of tc-desktop.

Here is a sample top level structure. This is what tC is currently using except for the extra rut.usfm file that it doesn't appear to use anyway. Note: in the case of tS projects the rut dir may not exist.

Implementation

When importing a project into tC or tS the application should parse the top level usfm file and then just update the target translation with the data.

Note: alignment information in the usfm file is ignored for the purpose of this import since a) tC generates this automatically, and b) tS does not provide this data, and c) the goal of this compatibility update is to share textual changes between applications.

Final Remarks

The "bible burrito" is a new specification being developed that will replace both the ResourceContainer spec and the project formats in use with tC and tS. The proposal above is a temporary solution until the bible burrito can be completed and properly implemented.

[PhotoNomad0]

Let's change from using terminology of USFM 2 & USFM 3 to using USFM with & without alignments.

Since this will require changes to tS and tC, what should we do for older project support? We could:

• For tC to tS - if the generator is tC and the USFM file is present in base folder, then tS removes alignments, otherwise it's an old project and not supported. Or should we change tC to also export a usfm file without alignments?
• for tS to tC -
  • if the generator is tS and the USFM file is present in base folder, and if there are alignments tC merges the USFM with previous alignments.
  • if the generator is tS and the USFM file is present in base folder, and if there are NOT alignments we do USFM import (I think this is current behavior)
  • if the generator is tS and the USFM file is NOT present in base folder we do the normal tS import.

[da1nerd]

@PhotoNomad0 tS is actually only generating usfm 2.

• tC to tS - that's exactly how I planned it. :+1: If the usfm file exists in the tc project it will be imported. tS will remove anything it does not need.
• tS to tC - in this situation there will never be alignments in the usfm because tS will have removed those. So it can be assumed only the last two sub-points apply.

[da1nerd]

I just thought of a complication.

the tc and ts data is fundamentally differently in how it's organized. tS is prone to merge conflicts so we had a whole ui in the app to handle that. tC resolved that by timestamping everything so this was never an issue. but because of this difference we cannot actually support importing a whole project from tS to tC and vice versa. The best we can do is import some of the data out of the project. The biggest problem is with importing tS projects into tC. it will not be possible to maintain any of the tS metadata during that process without encountering eventual merge conflicts and corrupting the project. so we cannot do anything like moving a project from tS->tC->tS to do that would either require a) a complete overhaul of the tS project structure or b) add support to tC to handle merge conflicts :sweat:.

With all that in mind I'm back at my previous idea which was to simply add support for importing/exporting usfm and support merging that usfm with existing projects.

[mannycolon]

@neutrinog I thought tC supported merged conflicts. am I missing something ? @benjore @PhotoNomad0 @cckozie

[da1nerd]

@mannycolon I was not aware of that. Do you know how that works?

[mannycolon]

@RoyalSix Hey could talk a little bit about how we handle merged conflicts. I remember you worked on that feature.

[mannycolon]

@neutrinog tC does handle merged conflicts

[da1nerd]

After extensive discussion we have decided on the below solution. The tC issue for this at https://github.com/unfoldingWord-dev/translationCore/issues/5958

• Neither the tC or tS project structure will change.
• tC already has support for importing a tS project locally and will be updated to include support for
  • merging imported tS projects into an existing tC project.
  • import tS projects from DCS (if not already supported)
• tS will add support for importing tC projects and
  • merging into an existing tS project
  • create a new tS project as needed

For both tools the following rules should be followed:

• during the import (online or local) no git history in the project will be preserved. Data will simply be read and the rest discarded.
• Pushing a tS project to a tC project on DCS is not supported and vice versa. We should already be safe from this because tC and tS both use different naming conventions (should be confirmed). The addition of discarding the git history as described previously will save us from accidents.

[jag3773]

@neutrinog Under this scenario, what would the (flowchart) workflow look like for someone who wants to go back and forth between tS and tC on a daily basis?

[da1nerd]

@jag3773

1. do work in tC and upload to DCS (tc repos are named differently than tS so there should not be a collision)
2. import work from tc repo and merge with project in tS
3. do work in tS and upload to DCS (will go into a different repo because of the naming difference)
4. go back to tC and import the ts repo and merge into the tc project.
5. repeat

So two different repositories will be regularly used.

[jag3773]

OK, not ideal, but that should work. Thanks!