No matching indices found: No indices match pattern “dmarc_aggregate*”

[valleydon]

Getting the error in Kibana when trying to load the info - No matching indices found: No indices match pattern “dmarc_aggregate*”

Tried the install on two fresh build Ubuntu server 20.04 with same results. Parsedmarc seems to be pulling and parsing emails fine from my mailbox.

[davidande]

Same here, maybe it's because there is no report to analyze? because of RGPD, there is less and less report sent by referer

[leonardo0014]

The Grafana resource names are dmarc-ag and dmarc-fo. The specification is missing in the description of the JSON example.

[variamus]

Hello I've the same problem. Could you please detail about how to solve this ?

Edit: Solution found; you have to had a regex package. pip install regex==2022.3.2 BEFORE running parsedmarc

[RVifian]

I have regex installed and still run into this problem. Ubuntu-20.04.4, parsedmarc 7.1.1, elasticsearch 7.17.1. How exactly were you able to solve it this way?

[valleydon]

Just did a fresh install using the solution from variamus but still having same issue.

[seanthegeek]

Kibana will show this error until data is added in elasticsearch by parsedmarc

[cleellacer]

sorry for the noob question(s). so if I set this up but have not fed the site any reports, I should get this message? while i figure out how to have the reports automatically retrieved, is there a way to place the GZ (or xml) files in a location on the server to be be parsed? I'm extremely new to all of this. but would like to try to parse these reports internally (first) before going to a third party solution to monitor these reports.

thanks in advance.

[msizec]

Hi I facing the same issue No index are created in elasticsearch, only the geoip one But my emails were parsed a while ago

[GIYItalk]

hi I wonder if there is a solution~

[Steltek]

I might be wrong, but my theory is that, at least in Grafana, this is a result of parsedmarc only creating elasticsearch indexes when it tries to save report data. If there is no report to save, no index gets created, and this confuses Kibana and Grafana who expect the index to be there regardless. (create_indexes only gets called by save_aggregate_report_to_elasticsearch or save_forensic_report_to_elasticsearch, which in turn only get called by the main loop if the respective save options are enabled and actual reports are present)

If correct, one potential fix would be to rewrite parsedmarc to create ES indexes in advance (e.g. for today and maybe tomorrow, if run daily), regardless of whether there's data to be put into them or not. (Depending on how often people run parsedmarc, this could require a "past" and "future" range parameter so that you could force it to create the next 15 days for example.)

Alternatively, have parsedmarc set up an index template that will cause ES to create the indexes.

As a workaround, you can manually create the indexes yourself (make sure you match the mappings and settings of the ones created by parsedmarc). Here's me creating today's aggregate reports index:

curl -X PUT "localhost:9200/my-index-000001?pretty" -H 'Content-Type: application/json' -d'
{
      "mappings": {
         "properties": {
            "date_begin": {
               "type": "date"
            },
            "date_end": {
               "type": "date"
            },
            "date_range": {
               "type": "date"
            },
            "disposition": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "dkim_aligned": {
               "type": "boolean"
            },
            "dkim_results": {
               "properties": {
                  "domain": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "result": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "selector": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  }
               }
            },
            "envelope_from": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "envelope_to": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "errors": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "header_from": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "message_count": {
               "type": "long"
            },
            "org_email": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "org_extra_contact_info": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "org_name": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "passed_dmarc": {
               "type": "boolean"
            },
            "policy_overrides": {
               "properties": {
                  "comment": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "type": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  }
               }
            },
            "published_policy": {
               "properties": {
                  "adkim": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "aspf": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "domain": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "fo": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "p": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "pct": {
                     "type": "long"
                  },
                  "sp": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  }
               }
            },
            "report_id": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "source_base_domain": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "source_country": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "source_ip_address": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "source_reverse_dns": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            },
            "spf_aligned": {
               "type": "boolean"
            },
            "spf_results": {
               "properties": {
                  "domain": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "result": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  },
                  "scope": {
                     "fields": {
                        "keyword": {
                           "ignore_above": 256,
                           "type": "keyword"
                        }
                     },
                     "type": "text"
                  }
               }
            },
            "xml_schema": {
               "fields": {
                  "keyword": {
                     "ignore_above": 256,
                     "type": "keyword"
                  }
               },
               "type": "text"
            }
         }
      },
      "settings": {
         "index": {
            "number_of_replicas": "0",
            "number_of_shards": "1"
         }
      }
   }
}
'

For ES/Kibana itself: Keep in mind that parsedmarc only ingests the report data into ES AFTER it has parsed all of the messages. (If you start out with a large chunk of messages, it will spend a lot of time parsing messages before populating the indexes.)