Tree info deletion on update

snacktavish commented 4 years ago

There is a bug currently where if you update a study, all of the information about the tree in otindex, e.g. @label, or '^ot:branchLengthMode' are replaced by 'null'

curl 'https://devapi.opentreeoflife.org/v3/studies/find_trees' -d '{"property":"ot:studyId","value":"ot_21","exact":true,"verbose":true}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1013  100   944  100    69   6210    453 --:--:-- --:--:-- --:--:--  6664
{
  "matched_studies": [
    {
      "ot:studyId": "ot_21",
      "ot:focalCladeOTTTaxonName": "Postia (genus in family Fomitopsidaceae)",
      "ot:dataDeposit": "http://purl.org/phylo/treebase/phylows/study/TB2:S12727",
      "ot:focalClade": 202387,
      "ot:studyPublicationReference": "Pildain, María Belén, and Mario Rajchenberg. \"The phylogenetic position of Postia sl (Polyporales, Basidiomycota) from Patagonia, Argentina.\" Mycologia 105.2 (2013): 357-367.",
      "ot:studyPublication": "http://dx.doi.org/10.3852/12-088",
      "ot:studyYear": 2013,
      "ot:curatorName": [
        "rgazis"
      ],
      "matched_trees": [
        {
          "ot:studyId": "ot_21",
          "ot:branchLengthDescription": "",
          "@label": "Fig. 2",
          "ot:treeId": "Tr55882",
          "ot:proposedForSynthesis": false,
          "ot:branchLengthMode": "ot:changesCount"
        },
        {
          "ot:studyId": "ot_21",
          "ot:branchLengthDescription": "",
          "@label": "Fig. 1",
          "ot:treeId": "Tr55881",
          "ot:proposedForSynthesis": true,
          "ot:branchLengthMode": "ot:changesCount"
        }
      ],
      "ot:tag": []
    }
  ]
}

and then following an update to the study

curl 'https://devapi.opentreeoflife.org/v3/studies/find_trees' -d '{"property":"ot:studyId","value":"ot_21","exact":true,"verbose":true}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1006  100   937  100    69   5930    436 --:--:-- --:--:-- --:--:--  6367
{
  "matched_studies": [
    {
      "ot:studyId": "ot_21",
      "ot:focalCladeOTTTaxonName": "Postia (genus in family Fomitopsidaceae)",
      "ot:dataDeposit": "http://purl.org/phylo/treebase/phylows/study/TB2:S12727",
      "ot:focalClade": 202387,
      "ot:studyPublicationReference": "Pildain, María Belén, and Mario Rajchenberg. \"The phylogenetic position of Postia sl (Polyporales, Basidiomycota) from Patagonia, Argentina.\" Mycologia 105.2 (2013): 357-367.",
      "ot:studyPublication": "http://dx.doi.org/10.3852/12-088",
      "ot:studyYear": 2013,
      "ot:curatorName": [
        "rgazis",
        "Emily Jane McTavish"
      ],
      "matched_trees": [
        {
          "ot:studyId": "ot_21",
          "ot:branchLengthDescription": null,
          "@label": null,
          "ot:treeId": "Tr55882",
          "ot:proposedForSynthesis": false,
          "ot:branchLengthMode": null
        },
        {
          "ot:studyId": "ot_21",
          "ot:branchLengthDescription": null,
          "@label": null,
          "ot:treeId": "Tr55881",
          "ot:proposedForSynthesis": true,
          "ot:branchLengthMode": null
        }
      ],
      "ot:tag": []
    }
  ]
}

It appears that these tree properties are getting loaded correctly when trees are originally loaded in to the otindex database build: https://github.com/OpenTreeOfLife/otindex/blob/master/otindex/scripts/load_nexson.py#L255

but are not getting updated correctly in https://github.com/OpenTreeOfLife/otindex/blob/master/otindex/add_update_studies.py#L84

potentially due to those properties not being included in the Tree model defined here: https://github.com/OpenTreeOfLife/otindex/blob/master/otindex/models.py#L89

kcranston commented 4 years ago

It looks like a problem with update to the jsonb column and / or index (because these are all properties that aren't explicitly defined in the model, but come from the jsonb column). I can dig more later.

To be clear - the fix is not to add these to the Tree model, but to determine why they aren't going in / being indexed / being found in the jsonb.

kcranston commented 4 years ago

Fixed by PR #49

OpenTreeOfLife / otindex

Tree info deletion on update #48