search

chaoss / grimoirelab-elk

GNU General Public License v3.0
60 stars 121 forks source link

How to add "new tile" in Grimoirelab dashboard with new name such as "Total stars" ? #911

Closed shikha0208 closed 4 years ago

shikha0208 commented 4 years ago

Hi all,

It would be my pleasure , if someone tells me how to add a new tile in grimoirelab dashboard with new names for example "Total stars for that repository or Total likes anything and it gives the exact count on that tile.

2.Also how to get a graph as shown in the dashboard for Git commits.I would like to know the path for the code.

Thank you.

ria18405 commented 4 years ago

Hello @shikha0208

  1. You can add a new visualisation in the dashboard by going to the "Edit" option at the top right column of your header. There, you can "Add" an existing visualisation or create new visualization and add it here. Once you have added the visualisation of your preference, you could change the title of that visualisation in the dashboard by going to the small settings button on top right corner of each visualisation and do a "Customise panel". Here, you will be able to change the title of the visualisation.

  2. A Bar Graph has been made for representing Git commits. When you make a new visualisation, you would see many options including Pie chart, Dot Plot, and many more, for a bar graph, select "Vertical Bar". You can then select the index, and the fields to be used as X and Y-axis. You can try executing the visualisation by clicking the small play button present. Once you are done, click to save your visualisation (top right).

I hope I understood and answered your question! Please let me know if this solves your issue. Thanks

shikha0208 commented 4 years ago

Hello @ria18405

Thank you for the answer.

I have understood both the points. But if i want create a new tile of table with columns for example: Repository name , forks, stars,contributors etc in the table which will have list of data.

So how to create this and how it will get all the data ?I am not sure , you got what i am trying to convey. If you define the steps for the above,i hope it will reach my point.

Thank you

ria18405 commented 4 years ago

Hi

As far as I understand, you will have to enrich the index containing raw github data. @valeriocos would be able to elaborate on this. For creation of a table, you need to select 'Data Table' under visualisations, select the index to be visualised, and split the row or table with Aggegation set to 'Terms', you will be able to select the required fields for the table.

shikha0208 commented 4 years ago

Thank you @ria18405 . Hello @valeriocos , Kindly if you can elaborate on this subject that would be pretty helpful.

zhquan commented 4 years ago

Hi @shikha0208,

If you want to see data related to forks, stars, and watchers. You have to activate the backend github with the category repository (GitHub repo statistics). Here you have an example how to configure the files (remember to activate the panels in the section phases on your setup.cfg). It will create a dashboard with forks, stars, and watchers info.

If you want to create your own visualization select the index pattern github_repositories to have forks, stars, and watchers info.

PS: If you change some visualizations/dashboard save it as a new visualization/dashboard with a different name due to when the panels is activated when you restart the container it will overwrite your changes otherwise when the dashboards are uploaded you have to deactivate panels and restart the container to avoid overwrite your changes.

I hope it helps you.

Best, Quan

shikha0208 commented 4 years ago

Hi @zhquan ,Thank you for the explanation.

I need a clarification on projects.json format. As mentioned , below is the projects.json format to get repositories in the dashboard. { "Chaoss": { "git": [ "https:/github.com/chaoss/grimoirelab-perceval", "https:/github.com/chaoss/grimoirelab-sirmordred" ], "github": [ "https:/github.com/chaoss/grimoirelab-perceval --filter-no-collection=true", "https:/github.com/chaoss/grimoirelab-sirmordred --labels=[example]" ] } }

But as you said to change the format to https://github.com/chaoss/grimoirelab-sirmordred#repo to retrive the stars,forcks etc . I am not clear with the format yet.Kindly if you can clear this.

  1. And in the setup.cfg file below what does the last line says below :

[github:repo] raw_index = github-repo_raw enriched_index = github-repo_enriched api-token = xxxx category = repository sleep-for-rate = true no-archive = true (suggested) studies = [enrich_extra_data:github]

[enrich_extra_data:github] json_url=https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

Do i have to include all the lines or just the first 5 lines are fit to get the count of stars in the dashboard?

If you want to create your own visualization select the index patterngithub_repositories to have forks, stars, and watchers info. Question: I cannot see github repositories in the index search.How should i select this?How the backend is activated,probably then i could see this.

I am bit new to this ,i would love to hear on this more closely.Thank you.

zhquan commented 4 years ago

Hi @shikha0208,

The files you shared on the projects.json the name of the source is github but on the setup.cfg the name is github:repo, those names must be the same. So if you want to analyze perceval and sirmordred the files should be: projects.json

{
    "Chaoss": {
        "git": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github:repo": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

setup.cfg

[github:repo]
raw_index = github-repo_raw
enriched_index = github-repo_enriched
api-token = xxxx
category = repository
sleep-for-rate = true
no-archive = true

Do i have to include all the lines or just the first 5 lines are fit to get the count of stars in the dashboard?

The last three lines are not needed due to you don't have to execute the study enrich_extra_data

Question: I cannot see github repositories in the index search.How should i select this?How the backend is activated,probably then i could see this.

Fields:

Best, Quan

vchrombie commented 4 years ago

Hi @shikha0208 Just an addition to @zhquan's comment, make sure you declare the below statement in the setup.cfg if you working with github:repo

setup.cfg

[panels]
github-repos = true

@zhquan, feel free to correct me if there is something wrong

shikha0208 commented 4 years ago

Hi @zhquan ,

As you said , both names should be same.But i need from github -PR, commits,issues, repositories (forks,stars).

My setup.cfg file

[general]
short_name = GrimoireLab
update = true
min_update_delay = 60
debug = false
logs_dir = /home/bitergia/logs
aliases_file = /home/bitergia/conf/aliases.json

[projects]
projects_file = /home/bitergia/conf/projects.json

[es_collection]
url = http://elasticsearch:9200

[es_enrichment]
url = http://elasticsearch:9200
autorefresh = true

[sortinghat]
host = mariadb
user = root
password =
database = demo_sh
load_orgs = true
orgs_file = /home/bitergia/conf/organizations.json
autoprofile = [github, pipermail, git]
matching = [email]
sleep_for = 100
unaffiliated_group = Unknown
affiliate = true
strict_mapping = false
reset_on_load = false
identities_file = [/home/bitergia/conf/identities.yml]
identities_format = grimoirelab

[panels]
kibiter_time_from = now-5y
kibiter_default_index = git
kibiter_url = http://kibiter:5601
kibiter_version = 6.1.4-1
github-repos = true

[phases]
collection = true
identities = true
enrichment = true
panels = true

[git]
raw_index = git_demo_raw
enriched_index = git_demo_enriched
latest-items = true
category = commit
studies = [enrich_demography:git, enrich_areas_of_code:git, enrich_onion:git]

[github]
api-token = my API TOKEN
raw_index = github_demo_raw
sleep-for-rate = true
sleep-time = "300"
enriched_index = github_demo_enriched
category = issue

[github:pull]
raw_index = github_pulls_chaoss
enriched_index = github_pulls_chaoss_enriched
api-token = my API TOKEN
sleep-for-rate = true
no-archive = true
category = pull_request
sleep-time = 300
studies = [enrich_onion:github]

[github:repo]
raw_index = github_pulls_chaoss
enriched_index = github_repo_enriched
api-token = my API TOKEN
sleep-for-rate = true
no-archive = true
category = repository
sleep-time = 300
studies = [enrich_onion:github]

[enrich_demography:git]

[enrich_areas_of_code:git]
in_index = git_demo_raw
out_index = git-aoc_demo_enriched

[enrich_onion:git]
in_index = git-aoc_demo_enriched
out_index = git-onion_demo_enriched
contribs_field = hash

projects.json file

{
    "Chaoss": {
        "git": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

Question: What is missing in the file , to get all the data in the dashboard from GitHub PR, commits,issues, repositories (forks,stars).Kindly have a look and correct me.

vchrombie commented 4 years ago

Hi @shikha0208 I understand the problem, I think the projects.json is not fully configured.

projects.json

{
    "Chaoss": {
        "git": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github:pull": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github:repo": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

I think this should be enough. Let us know how it goes.

shikha0208 commented 4 years ago

Hi @vchrombie ,

Thank you for the reply , but i configured the json file same as above and my setup.cfg is also same. But still i am getting ERROR and no data is displayed for github repositories

PFA ((https://user-images.githubusercontent.com/63613092/87191207-409c1100-c2f4-11ea-9af6-c33352f904f9.png))

I do not know where i am going wrong.Kindly help.

ria18405 commented 4 years ago

Hello

I think your files are properly set up, I just want to confirm if you are executing modred with the right parameters. You can have look at some possible executions here.

Let us know if the error is solved?

shikha0208 commented 4 years ago

Hi @ria18405 ,

I am executing from docker-compose , from the this path (https://github.com/chaoss/grimoirelab/tree/master/docker-compose) Using just this command :docker-compose up -d .

But i am not able to see still.

zhquan commented 4 years ago

Hi @shikha0208,

You have to make some changes in your setup.cfg, if you are using this docker-compose.yml you can look at the logs and see if there are any execution errors at /tmp/all.logs

Changes:

[github:repo] studies = [enrich_onion:github]



You can see your indexes and aliases if you go to the `Dev Tools` on kibiter
- index: `GET _cat/indices`
- alias: `GET _cat/aliases`

You can see your index patterns if you go to `Managment` on kibiter
- click on `Index Patterns`

Steps:
1. Stop the mordred container
2. Change the setup.cfg
3. Remove the indexes (these indices are mixing different categories):
    - github_pulls_chaoss
    - github_repo_enriched
    - github_pulls_chaoss_enriched
4. Restart the mordred container
5. Check if the enriched indexes have items
6. Check if the index patterns are created

I hope it helps you.

Best,
Quan
shikha0208 commented 4 years ago

Hi @zhquan , Thank you.

Below is my setup.cfg file:

[general]
short_name = GrimoireLab
update = true
min_update_delay = 60
debug = false
logs_dir = /home/bitergia/logs
aliases_file = /home/bitergia/conf/aliases.json

[projects]
projects_file = /home/bitergia/conf/projects.json

[es_collection]
url = http://elasticsearch:9200

[es_enrichment]
url = http://elasticsearch:9200
autorefresh = true

[sortinghat]
host = mariadb
user = root
password =
database = demo_sh
load_orgs = true
orgs_file = /home/bitergia/conf/organizations.json
autoprofile = [github, pipermail, git]
matching = [email]
sleep_for = 100
unaffiliated_group = Unknown
affiliate = true
strict_mapping = false
reset_on_load = false
identities_file = [/home/bitergia/conf/identities.yml]
identities_format = grimoirelab

[panels]
kibiter_time_from = now-5y
kibiter_default_index = git
kibiter_url = http://kibiter:5601
kibiter_version = 6.1.4-1
github-repos = true

[phases]
collection = true
identities = true
enrichment = true
panels = true

[git]
raw_index = git_demo_raw
enriched_index = git_demo_enriched
latest-items = true
category = commit
studies = [enrich_demography:git, enrich_areas_of_code:git, enrich_onion:git]

[github]
api-token = MYTOKEN
raw_index = github_demo_raw
sleep-for-rate = true
sleep-time = "300"
enriched_index = github_demo_enriched
category = issue

[github:pull]
raw_index = github_pulls_chaoss
enriched_index = github_pulls_chaoss_enriched
api-token = MYTOKEN
sleep-for-rate = true
no-archive = true
category = pull_request
sleep-time = 300

[github:repo]
raw_index = github_repo_raw
enriched_index = github_repo_enriched
api-token = MYTOKEN
sleep-for-rate = true
no-archive = true
category = repository
sleep-time = 300

[enrich_demography:git]

[enrich_areas_of_code:git]
in_index = git_demo_raw
out_index = git-aoc_demo_enriched

[enrich_onion:git]
in_index = git-aoc_demo_enriched
out_index = git-onion_demo_enriched
contribs_field = hash

projects.json format:

{
    "Chaoss": {
        "git": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github:pull": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ],
        "github:repo": [
            "https:/github.com/chaoss/grimoirelab-perceval",
            "https:/github.com/chaoss/grimoirelab-sirmordred"
        ]
    }
}

When i changed the setup.cfg file , and restarted with docker compose . I was asked to create index at kibana. Then dashboard started. 1.Question : Why index has to be created and what name should i create index with?

When dashboard started , i went to GitHub repositories section , there i could not see any data on tiles for "stars,forks.subscribers" but i could see for others such as "Forks over time tile", "stars over time "tile. Questions: I need data for stars ,forks etc, why it is not displayed?

PFA

https://user-images.githubusercontent.com/63613092/87253267-37d54780-c47a-11ea-8a03-ce701f090a7b.png

zhquan commented 4 years ago

Hi @shikha0208,

Are you using this bitergia/mordred:grimoirelab-0.2.42 version of mordred?

1.Question : Why index has to be created and what name should i create index with?

The indexes are created automatically by Mordred so you should have the indexes described at yout setup.cfg and you don't have to create any indexes.

I need data for stars ,forks etc, why it is not displayed?

If the standard visualizations don't fit your needs, you can create your own visualizations. You can check the fields in Discover choosing the index pattern github_repositories. Check out these fields:

https://user-images.githubusercontent.com/63613092/87253267-37d54780-c47a-11ea-8a03-ce701f090a7b.png

The visualizations Forks and Stars showed here is just a title but the configuration of the visualizations seems not correct.

You should see a dashboard like https://github.com/chaoss/grimoirelab-sigils/blob/master/docs/assets/images/screenshots/sigils/github-repositories.png

Best, Quan

shikha0208 commented 4 years ago

Hi @zhquan ,

I am using : image: bitergia/mordred:latest

Ok , so that's just a title , i assumed some count on that tile (stars,forks)

1.But i am getting this count on my dashboard for the same repositories which you gave screenshot ,which is not right and there is no graph on the "Forks over time" and etc other.

PFA of my dashboard:

image 1 https://user-images.githubusercontent.com/63613092/87307432-e2507780-c519-11ea-87a5-edb7564dc50a.png

image 2 https://user-images.githubusercontent.com/63613092/87307577-28a5d680-c51a-11ea-86ce-6f54dc3b981e.png

Correct me , where i am going wrong ?Could you please send me the exact setup.cfg file which will give github (issues,repo,pull request)? I am using the same docker compose yaml file https://github.com/chaoss/grimoirelab/blob/master/docker-compose/docker-compose.yml

2.Problem:Every time i restart my container , i do not find my previous created dashboard, which i saved with different name that is lost. What is the process to save the dashboard, every time i should be able to see my created dashboard.

Kindly address this issue.

3.Why index has to be created and what name should i create index with? PFA https://user-images.githubusercontent.com/63613092/87317414-3dd53200-c527-11ea-84aa-05491e3336d5.png

zhquan commented 4 years ago

Hi @shikha0208,

I am using : image: bitergia/mordred:latest

This is the same as bitergia/mordred:grimoirelab-0.2.42 due to this is the last version.

1.But i am getting this count on my dashboard for the same repositories which you gave screenshot ,which is not right and there is no graph on the "Forks over time" and etc other.

The screenshot that I shared is a demo. The data on your dashboard is correct due to the numbers of the stars, forks, and subscribers are the same as GitHub.

About the visualizations * over time, those metrics are computed every time the process executes. You will see data on those graphs in the next executions.

2.Problem:Every time i restart my container , i do not find my previous created dashboard, which i saved with different name that is lost. What is the process to save the dashboard, every time i should be able to see my created dashboard.

You have to save as a new visualization/dashboard with a different name otherwise Mordred will overwrite your changes. On the other hand, you can deactivate panels on the section phases, this way Mordred will not upload any visualizations/dashboards.

Best, Quan

shikha0208 commented 4 years ago

Hi @zhquan ,

1.Why there in no lines or graph or any data on the "Forks over time"?Is my data right? In the next executions , means when i run container next time , i can get the data?

2.Regarding dashboard , i have saved with different name but i didn't see that in next execution when i restarted the container.

3.I changed short_name in setup.cfg , i think that doesn't impact anything.But now i am getting this PFA https://user-images.githubusercontent.com/63613092/87317414-3dd53200-c527-11ea-84aa-05491e3336d5.png How should i see the dashboard?

Thank you

zhquan commented 4 years ago

Hi @shikha0208,

1.Why there in no lines or graph or any data on the "Forks over time"?Is my data right? In the next executions , means when i run container next time , i can get the data?

You can see dots if you want, but you have to edit the visualization and change the interval to 1m (per minute) and change the time picker to the last 15 min and add this .points(show=true) at the end of the Timelion Expression. By default this visualization shows data per month (1M)

Your data shared at https://user-images.githubusercontent.com/63613092/87307432-e2507780-c519-11ea-87a5-edb7564dc50a.png and https://user-images.githubusercontent.com/63613092/87307577-28a5d680-c51a-11ea-86ce-6f54dc3b981e.png looks good

2.Regarding dashboard , i have saved with different name but i didn't see that in next execution when i restarted the container.

Did you execute docker-compose restart or docker restart <docker_ID or docker_name>?

3.I changed short_name in setup.cfg , i think that doesn't impact anything.But now i am getting this

Yes, changing that value doesn't impact anything.

https://user-images.githubusercontent.com/63613092/87317414-3dd53200-c527-11ea-84aa-05491e3336d5.png How should i see the dashboard?

It seems you removed the index patterns or you removed the elasticsearch docker container. If you make docker-compose down or docker-compose rm -v will remove all indexes, index patterns, dashboards, etc you have.

PS: check your indexes if they are gone too

Best, Quan

shikha0208 commented 4 years ago

Hi @zhquan ,

Thank you , with the above settings i am able to see the dots on the forks over time. https://user-images.githubusercontent.com/63613092/87359840-04251b00-c569-11ea-827a-1eee38064719.png

1.Why name called unknown organisation is displayed in the image below ?Unaffiliated group , so it means it will be always this unknown? In this image , the last tile which is "issue by organization over time" has most of them with green which is unknown org.So it is the right way?Or i should change

https://user-images.githubusercontent.com/63613092/87360165-cb397600-c569-11ea-8dd5-141618e8c826.png

  1. Changes made by me in the dashboard and saved with different name is not seen in the next execution. First i executed docker-compose up -d Made changes and saved the dashboard Then docker-compose down command Last docker-compose up -d command Not able to see the changes

    With docker-compose restart , it is available but why not it is working without restart command? May i know where this data is getting saved for kibana, can i export data into json or is there any way to store the data permanently as if someone uses docker-compose down command then all the data will be vanished.

Kindly address this issue.

zhquan commented 4 years ago

Hi @shikha0208,

You are making different questions I think the issue is resolved.

So if you have other questions please open new issues and close this one.

Best, Quan

shikha0208 commented 4 years ago

Ok sure!

Thank you , i will close this issue.