NNTP is not updated due to mapping errors in Elasticsearch index

[canasdiaz]

The NNTP index need some changes with its mapping due to the following error:

2018-04-04 15:46:58,591 - grimoire_elk.elastic - ERROR - Failed to insert data to ES: {'type': 'illegal_argument_exception', 'reason': 'Limit of total fields [1000] in index [nntp_mozilla_180402] has been exceeded'}, https://****/nntp_mozilla_180402/items/_bulk

Actions:

• apply a new mapping to the nntp raw index
• get nntp index updated

[canasdiaz]

Mapping changed, waiting for the next collection cycle to get updated data. We'll use the original indexes, no need to collect data again (luckily)

   "nntp_mozilla_171125" : {
      "settings" : {
         "index" : {
            "mapping" : {
               "total_fields" : {
                  "limit" : "3000"
               }
            },

...

   "nntp_mozilla_171125_enriched_171126" : {
      "settings" : {
         "index" : {
            "creation_date" : "1511726104380",
            "version" : {
               "created" : "5010199"
            },
            "mapping" : {
               "total_fields" : {
                  "limit" : "3000"
               }
            },

[canasdiaz]

We still have issues with the mapping for the raw index:

2018-04-10 08:08:35,105 - urllib3.connectionpool - DEBUG - https://****:443 "PUT /nntp_mozilla_171125/items/_mapping HTTP/1.1" 400 270
2018-04-10 08:08:35,106 - grimoire_elk.elastic - ERROR - Error creating ES mappings {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[3lI24n-][x.x.x.x:9300][indices:admin/mapping/put]"}],"type":"illegal_argument_exception","reason":"mapper [data.Subject] of different type, current_type [keyword], merged_type [text]"},"status":400}

And we are having a similar error with the enriched index:

2018-04-10 08:15:07,424 - urllib3.connectionpool - DEBUG - https://****:443 "PUT /nntp_mozilla_171125_enriched_171126/items/_mapping HTTP/1.1" 400 293
2018-04-10 08:15:07,426 - grimoire_elk.elastic - ERROR - Error creating ES mappings {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"mapper [Subject] of different type, current_type [keyword], merged_type [text]"}],"
type":"illegal_argument_exception","reason":"mapper [Subject] of different type, current_type [keyword], merged_type [text]"},"status":400}
2018-04-10 08:15:07,426 - grimoire_elk.elastic - ERROR - Mapping: 
            {
                "properties": {
                    "Subject": {
                        "type": "text",
                        "index": true
                    },
                    "Subject_analyzed": {
                        "type": "text",
                        "index": true
                    }
                }
            }

2018-04-10 08:15:07,426 - mordred.task_manager - ERROR - Exception in Task Manager 400 Client Error: Bad Request for url: https://****/nntp_mozilla_171125_enriched_171126/items/_mapping

The impact of this is not having updated data for NTTP. So this is maximum priority.

[canasdiaz]

Indexes are ready, we just need to switch them. Will be done during next two hours.

[canasdiaz]

Data is updated, we are done here.

You can check the updated data at: https://analytics.mozilla.community/goto/da596412f7f5fe03a1cef64bc3e62d78