mozilla / participation-metrics-org

Participation metrics planning repository
4 stars 4 forks source link

author_[min,max]_date fields not generated for Git index #197

Closed canasdiaz closed 5 years ago

canasdiaz commented 5 years ago

The Git index is missing these two fields:

Those fields are created by a study called "demography" which is failing for sotrar. This is the logged error:

Error updating mix and max date for author 50673b0a049f451cb232d51b3bbf8e3e22f26f97. Demography aborted.

The author with this id does exists, so something is happening here not related to the SH database .. at the same time this error is produced I've seen these two:

504 Server Error: Gateway Time-out for url: https://vpc-bitergia-sotrar-es-37o4obgsrpsss3xvycwpdwptzi.eu-central-1.es.amazonaws.com/git_sotrar_181018_enriched_190315/_update_by_query
409 Client Error: Conflict for url: https://vpc-bitergia-sotrar-es-37o4obgsrpsss3xvycwpdwptzi.eu-central-1.es.amazonaws.com/git_sotrar_181018_enriched_190315/_update_by_query
canasdiaz commented 5 years ago

Past week new version of the tool was deployed in sotrar to fix this issue. It updated the git index during the weekend so all items have the fields demography_min_date and demography_max_date.

CC @hmitsch @havardl

We are done here, moving to done.

GET /git/_search
{  
  "aggs": {
    "1": {
      "cardinality": {
        "field": "author_uuid"
      }
    }
  },
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        },
        {
          "range": {
            "demography_min_date": {
              "gte": "01/03/1960",
              "format": "dd/MM/yyyy||yyyy"
              }
          }
        }
      ],
      "filter": [],
      "should": [],
      "must_not": []
    }
  },
  "size": 0
}

Returns

{
  "took": 70,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 3442148,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "1": {
      "value": 22130
    }
  }
}