Some translations are not being published

[jbphet]

@solaolateju and @RVieyra have let me know that some translators have reported that their translations are not showing up on the website. Here is a list of the ones that are believed to have been submitted through Rosetta but never appeared:

• Locale = Kikuyu (ki):
  • My Solar System
  • Natural Selection
  • Neuron
  • Normal Modes (bizarrely, it's not showing up on the published or "not yet translated" list...
  • Number Compare
  • Number Line: Distance
  • Number Line: Integers
  • Number Line: Operations
  • Number Play
  • Pendulum Lab
  • Plinko Probability
  • Projectile Motion
  • Proportion Playground
• Locale = Shona (sn):
  • Number Play
  • Number Line
  • Fractions Intro
  • Friction

[jbphet]

I did some initial investigation into this and found that the translated sims are actually being built, they just aren't appearing on the website. I talked with @mattpen and @chrisklus about this, and they looked into it while we were on a Zoom call together. From @mattpen's interpretation of the website logs, it seems the the publication step where the screen names are updated in the DB is failing, and this is causing the update of the translation list to fail. They've seen some problems that sound closely related to this before so they have some ideas on how to potentially address this one.

I'll assign this to @mattpen to follow up, and he feels like this can be addressed by the end of the month (July).

I'm also going to leave it assigned to me and do a bit more investigation into whether all translation publications are failing or just some of them, and will describe how the users can work around this in the short term.

[jbphet]

@RVieyra and @solaolateju - Since it will likely be around a week before this is fixed, I thought I'd describe way to work around the problem in the short term in case you need it.

Since the sims are built and are available from the website, users can get to the translations by choosing a translation in a different language and then making a small modification to the URL. As an example, here is how one could access the Kikuyu translation of My Solar System:

1. Go to the main sim page (https://phet.colorado.edu/en/simulations/my-solar-system)
2. Click on "Translations" (https://phet.colorado.edu/en/simulations/my-solar-system/translations)
3. Select any translation. I usually just use the first one on the list, which in this case is Amharic (https://phet.colorado.edu/sims/html/my-solar-system/latest/my-solar-system_am.html)
4. Modify the locale portion of URL to be the desired locale. In this case, you'd change am to ki to get https://phet.colorado.edu/sims/html/my-solar-system/latest/my-solar-system_ki.html.
5. The sim should load. When I do this, I see the following:

[mattpen]

I looked into this today. I found some error messages that indicated that the website sync feature is having some problems with database operations (included below) during project syncronization. This is causing new translations to not be saved to the database, which is used to populate the translations table on sim pages on the website. I tried to reproduce the problem by manually triggering a syncronization of wave-interference (the latest sims to fail due to this problem) both on ox-dev and on production, but both instances were succcessful.

I added a line of code that should syncronize tomcat's memory with the file system before the operation that failed, which hopefully will prevent this problem from recurring but without reproducability it will be difficult to measure success. So I also added a step that will email @chrisklus and myself if it fails for this reason again. I'll deploy these changes to production tonight.

Meanwhile, it looks like retriggering a project syncronization is fixing the problem temporarily so I just kicked this off for all HTML5 sims. This should finish within a couple hours. I'll check the logs afterward to see if the error popped up again. If it succeeds, then the website pages should display the translated sims within 24 hours after the process completes (due to caching), although this should accelerate as I'll clear the cache during the deploy tonight.

[solaolateju]

Thank you @mattpen for looking into the issue. I can confirm the sims are now accessible. I will let you know what we hear back from the translators.

[mattpen]

Thanks Sola. I missed the window to deploy the changes to (hopefully) fix this permanently last night. I will try again tonight. Afterwards I'll rerun the syncronization again in case the problem has popped up again since yesterday.

[mattpen]

The screen title translation problem is still occurring today. And there is a new error in the email notifications. I'll see if I can reproduce this on ox-dev.

[solaolateju]

Hi @mattpen, does this have any impact on user(translators) experience? If yes, what should they avoid or expect at this time?

[mattpen]

@solaolateju - It looks like this will delay some translations in terms of their visibility on the Translations tab on sim pages. I'm currently planning to check on this and resync the website a couple times a week until we figure out the root cause, so the delay should be less than 1 week. I'm hoping to figure this out within the next few days.

The direct links to sims will still be available within an hour of submission and will NOT be affected by my continuing work.

[mattpen]

The problem is still happening, but now I am getting email notifications when they do at least.

Reading up on https://stackoverflow.com/questions/2302802/how-to-fix-the-hibernate-object-references-an-unsaved-transient-instance-save, I'm going to try to add cascade properties to the simulations related hibernate mappings.

[mattpen]

I think this is in a stable enough spot now that we should attempt to retrigger builds since June in order to make sure translation credits are up to date.

Assigning to @jbphet to retrigger builds. Moving forward, I'll be receiving emails for failed syncs that include the sim, translator id, and locale and I can easily retrigger these manually.

Unfortunately it doesn't look like the cascade properties worked as I hoped, so I'm still working on a fix.

[mattpen]

I ran a script to identify and remediate missing translation credits today, which was successful. There were 7 translations identified afterwards that are missing their file entirely, so I resubmitted these builds on the rosetta admin page.

So, we should be caught up in terms of missing translations, but I'm still stumped on what is causing the sync failures.

[RVieyra]

Thank you for this, @mattpen !

[mattpen]

I deployed another hopeful fix for this bug tonight. The email notifications have been working well, so I've been manually syncing failed translations in the interim. There have been quite a few (maybe 5-10 per day)!

[mattpen]

I haven't received any alerts that builds have failed or syncronizations errored out, so I believe that this issue is now fixed! Closing.