Closed olehkorkh-planeks closed 9 months ago
@suhlrich and @AlbertoCasasOrtiz, tagging you both on this one (I don't want us to start deleting data). Can you both review? We can then do some testing on the dev side.
@olehkorkh-planeks two remarks regarding this PR:
This line selects 500 sessions, the problem is that if, after some cleaning maybe, the 500 first selected sessions are good then there will be no more cleaning even if there are still bad sessions in there. That's why I had to bump to 500 to clean the entire dev db. Is there a way to adjust the task to make sure we go through the entire db? The prod db has over 40k sessions. I can see that it is not practical to filter the entire db every time, maybe we can find a way to filter it once and then we only select part of it (as is now). Let me know your thoughts.
We need to use a more conservative criterion here. We noticed that we have sessions with data we want to keep but with no neutral trial. FYI it is possible to directly navigate to step5 and record data even if there is no neutral. Let's use the following:
error
AND older than 7 daysFYI, we also noticed that we might not have a proper backup of the DB and S3 in place, so we might want to implement that before merging this into main
. Regardless, it would be good to address the two points above (then maybe we comment for now). Thanks.
FYI @suhlrich and @AlbertoCasasOrtiz
@antoinefalisse I prepared a fixed implementation. Let me know if it works for you - https://github.com/stanfordnmbl/opencap-api/pull/144
Fix for https://github.com/stanfordnmbl/opencap-api/issues/82