Deleting old session(s) forget pendings

[horkko]

Hi,

I've found a small problem while updating BioMAJ bank and having to delete old sessions because of parameter keep.old.version. Here is my situation. I've updated my bank several time using --from-scratch to be sure to rebuild a session each time and filling the {production} field in MongoDB. Then, after reaching keep.old.version max, BioMAJ starts to delete entries into {sessions} fields in MongoDB which is the expected behavior. Now, I've pusblihsed a release and freeze another one such as follow :

  "production": [
    {
      "freeze": true,
      "release": "2003-11-26__3"
    },
    {
      "freeze": false,
      "release": "2003-11-26__5"
    }
]

I've restarted 2 more sessions with --from-scratch option and --stop-after postprocess. Thus, I've some pending sessions:

  "pending": {
    "2003-11-26__6": 1456129462.64968,
    "2003-11-26__7": 1456129490.114435
  }

If I start another run with --from-scratch``, old sessions (2003-11-266and2003-11-267) are deleted from{sessions}but remain in{pending}`.

  "production": [
    {
      "freeze": true,
      "release": "2003-11-26__3"
    },
    {
      "freeze": false,
      "release": "2003-11-26__5"
    },
    {
      "freeze": false,
      "release": "2003-11-26__8"
    }
  ],
  "sessions": [
    {
      "release": "2003-11-26__3"
    },
    {
      "release": "2003-11-26__5"
    },
    {
      "release": "2003-11-26__8"
    }
  ],
  "pending": {
    "2003-11-26__6": 1456129462.64968,
    "2003-11-26__7": 1456129490.114435
  }

So I think there something missing during the deletion of old sessions. You should also check in the {pending} field. I think that's a small bug.

Emmanuel

[osallou]

we delete only "over" sessions. If you stop after post-process but does not end the workflow, it will not be deleted and remain in pending. We could delete pending sessions, but it is difficult to know which sessions shoud not should not be deleted

[horkko]

What do you mean by "delete only 'over' sessions" ? Because I've run the following: biomaj-cli.py --update --bank alu --from-scratch --stop-after postprocess three times and I've this in the database:

  "sessions": [
    {
      "status": {
        "over": true
      },
      "release": "2003-11-26__3"
    },
    {
      "status": {
        "over": true
      },
      "release": "2003-11-26__5"
    },
    {
      "status": {
        "over": true
      },
      "release": "2003-11-26__9"
    },
    {
      "status": {
        "over": false
      },
      "release": "2003-11-26__10"
    },
    {
      "status": {
        "over": false
      },
      "release": "2003-11-26__11"
    },
    {
      "status": {
        "over": false
      },
      "release": "2003-11-26__12"
    }
  ],
  "current": 1456129428.959047,
  "production": [
    {
      "freeze": true,
      "session": 1455893942.463664,
      "release": "2003-11-26__3"
    },
    {
      "freeze": false,
      "session": 1456129428.959047,
      "release": "2003-11-26__5"
    },
    {
      "freeze": false,
      "session": 1456132781.286973,
      "release": "2003-11-26__9"
    }
  ],
  "pending": {
    "2003-11-26__10": 1456142759.767146,
    "2003-11-26__11": 1456143012.983961,
    "2003-11-26__12": 1456143229.712447
  }

Then, I've run this command biomaj-cli.py --update --bank alu --from-scratch and we can see that "pending" sessions have disappeared from the {sessions} array where as they're still tagged as "pending" ({pending}) Logs

2016-02-22 13:18:10,328 INFO  [root][MainThread] Workflow:wf_delete_old
2016-02-22 13:18:10,328 INFO  [root][MainThread] Workflow:wf_delete_old:Delete:2003-11-26__9
2016-02-22 13:18:10,329 INFO  [root][MainThread] Workflow:Start
2016-02-22 13:18:10,329 INFO  [root][MainThread] Workflow:Start:removeprocess
2016-02-22 13:18:10,329 INFO  [root][MainThread] Workflow:wf_removepreprocess
2016-02-22 13:18:10,329 INFO  [root][MainThread] PROC:REMOVE
2016-02-22 13:18:10,338 INFO  [root][MainThread] Workflow:Start:remove_release
2016-02-22 13:18:10,338 INFO  [root][MainThread] Workflow:wf_remove_release
2016-02-22 13:18:10,428 INFO  [root][MainThread] Workflow:wf_clean_old_sessions
2016-02-22 13:18:10,429 INFO  [root][MainThread] Bank:DeleteOldSessionDir:alu-2003-11-26__10
2016-02-22 13:18:10,509 INFO  [root][MainThread] Bank:DeleteOldSessionDir:alu-2003-11-26__11
2016-02-22 13:18:10,593 INFO  [root][MainThread] Bank:DeleteOldSessionDir:alu-2003-11-26__12

Mongo

  "sessions": [
    {
      "status": {
        "over": true
      },
      "release": "2003-11-26__3"
    },
    {
      "status": {
        "over": true
      },
      "release": "2003-11-26__5"
    },
    {
      "status": {
        "over": true
      },
      "release": "2003-11-26__13"
    }
  ],
  "current": 1456129428.959047,
  "production": [
    {
      "freeze": true,
      "session": 1455893942.463664,
      "release": "2003-11-26__3"
    },
    {
      "freeze": false,
      "session": 1456129428.959047,
      "release": "2003-11-26__5"
    },
    {
      "freeze": false,
      "session": 1456143477.041664,
      "release": "2003-11-26__13"
    }
  ],
  "pending": {
    "2003-11-26__10": 1456142759.767146,
    "2003-11-26__11": 1456143012.983961,
    "2003-11-26__12": 1456143229.712447
  }

From my point of view, deleted sessions should also be check in the {pending} field. It's no sens they are still here.

Emmanuel

[osallou]

ok, I understood things differently. This should be fixed indeed

[osallou]

Can you try the patch? (commit 54bfa8818c5e1eb203ee459592828c59fa084d9a on branch fix_34)

[horkko]

Yep that fixed the problem. However, iteritems (line 550) is not supported in Python3, you should consider using list(self.bank['pending'].items()) :+1:

[osallou]

I was fixing python 3 support in the meanwhile

Le lun. 22 févr. 2016 13:52, Emmanuel Quevillon notifications@github.com a écrit :

Yep that fixed the problem. However, iteritems (line 550) is not supported in Python3, you should consider using list(self.bank['pending'].items()) [image: :+1:]

— Reply to this email directly or view it on GitHub https://github.com/genouest/biomaj/issues/34#issuecomment-187158262.