wk4 2024_jan_epic: ANDON ASAP SIM PROD - Missing 2 teams and saved experiments created on 1/23/24 and 1/24/24

lzim / teampsd

Team PSD is using GitHub, R and RMarkdown as part of our free and open science workflow.

GNU General Public License v3.0

9 stars 23 forks source link

wk4 2024_jan_epic: ANDON ASAP SIM PROD - Missing 2 teams and saved experiments created on 1/23/24 and 1/24/24 #3065

Closed lijenn closed 8 months ago

lijenn commented 8 months ago

@tejaspesquare @jamesmrollins We found an ANDON/urgent issue today 1/24/24 where we are missing two teams in SIM Prod.

Please message me for the two team names. Team 1 was created on 1/23/24 with saved experiments by @lzim, @dlkibbe, @dlounsbu. @lijenn and @ljmoody created team 2 on 1/24/24 and noticed team 1 and team 2 are both missing in Sim Prod as of 1/24/24.

@lijenn Was unable to find both the team's groups in Epicenter. Is there anyway we can restore these teams? Are any development work being done in Test impacting the functionality in Prod somehow?

tejaspesquare commented 8 months ago

@lijenn - There is a possibility that groups have been deleted by someone accidentally & hence it's not there. By any chance if any flag maintained in database can track recent activities on those particular groups. On lower instances works fine.

Is there anyway we can restore these teams? - Answer to this, if we have any mechanism wherein deleted items are able to restore as record into database it will resolve problem.

lzim commented 8 months ago

Thanks @lijenn & @tejaspesquare

@tejaspesquare Please keep me posted on the progress determining:

[x] Whether any of the work completed on Tuesday in PROD can be regained.
[x] We are particularly concerned about regaining the Q/H/F/D text entered for the experiments run as this was time-consuming to produce.

Then, we'll want to ensure we understand

[x] How/why the work would have been deleted to prevent any loss of work in the future.

FYI: @jamesmrollins @dlkibbe @dlounsbu

lijenn commented 8 months ago

@lzim After discussing with the Forio team, they are not able to recover the experiments. The Forio team spend the weekend to recover the shared, ind, and team worlds. You, @dlkibbe, and @dlounsbu should see this at https://mtl.how/sim now.

Forio confirmed that the worlds were deleted by request in either Sim UI or Epicenter (Jenn is confirming), but the code is unable to show who/what happened exactly.

They plan to update the Epicenter code to track who deletes groups and how to recover experiments over the next year.

Let's review the users who have access to Epicenter and determine if we need to remove anyone.

Talk at 2024_8am_workflow!

FYI @jamesmrollins @ljmoody @tejaspesquare

lzim commented 8 months ago

Thanks for the hard work on this, everyone! I believe we can close this.