[schema] Update Kafka KIP schema

[canasdiaz]

This patch extends the information schema we have for Kafka KIP following the standard format which includes the fields "name", "type", "aggregatable" and "description" for every field in the enriched index.

Signed-off-by: Luis Cañas-Díaz lcanas@bitergia.com

[coveralls]

Pull Request Test Coverage Report for Build 935523662

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.02%) to 82.253%

---

Totals
Change from base Build 922111005:	0.02%
Covered Lines:	8667
Relevant Lines:	10537

---

💛 - Coveralls

[zhquan]

@sanacl I miss these fields:

• author_gender
• author_gender_acc
• From_gender
• From_gender_acc
• From_user_name

And where does the repository_labels field come from?

[canasdiaz]

@zhquan oh, good catch! you are right about the missing fields.

You can see the code which assign the repository_labels at:

• https://github.com/chaoss/grimoirelab-elk/blob/df179d753ae3196d0143fe836d073e652e99ac36/grimoire_elk/enriched/mbox.py#L185
• https://github.com/chaoss/grimoirelab-elk/blob/df179d753ae3196d0143fe836d073e652e99ac36/grimoire_elk/enriched/utils.py#L42

[canasdiaz]

@zhquan it is "fun" that the repository_labels field is not visible from the index pattern menu but it is from the discover app. I guess this is because of the two different indexes that are linked by the alias (mbox and pipermail indexes)

[zhquan]

@sanacl ok, then we have to update the index pattern to include repository_labels

[canasdiaz]

@zhquan should I add to the documentation all the fields visible from the discovery app? I guess the answer is yes, right?

[canasdiaz]

Hey @zhquan, I compared the fields we have both in the index pattern and the ones visible from the discover app. The ones below were not added as they are undefined and I do not find documentation about them:

• From_multi_org_names
• From_user_name
• offset

I think they should not be part of this patch.

---

UPDATE: I've also excluded From_gender and From_gender_acc because of the same reason.

[zhquan]

@sanacl I see that the following fields are in the index pattern:

• From_user_name
• From_gender
• From_gender_acc

Undefined fields (there may be more):

• From_multi_org_names
• author_multi_org_names
• mbox_author_domain
• metadata__filter_raw
• offset
• repository_labels

We have two options:

1. In this PR update only the available fields on the index pattern. And update again when the index pattern is updated
2. First update the index pattern and then update all fields in this PR

[canasdiaz]

@zhquan in order to avoid being blocked and start "delivering" something I've done what you suggested in option 1. I've only included the ones available in the index pattern.

[zhquan]

@sanacl rebase it and I will merge it thanks.

[canasdiaz]

@zhquan done