GSoC Idea: Boosting data processing in GrimoireLab

[valeriocos]

GrimoireLab allows to produce analytics with data extracted from more than 30 tools used for contributing to Open Source development such as version control systems, issue trackers and forums. A common execution of GrimoireLab consists in collecting data from a given repository, processing and enriching the data obtained and finally visualizing it on dynamic Web dashboards. At the core of this process there is a component called ELK, which is in charge of integrating the data finally shown on the dashboards.

The evolution of GrimoireLab requires now to reshape some of the functionalities provided by ELK to improve its maintainability. This project idea is about refactoring and redesigning the core of ELK using popular libraries for data management and processing such as elasticsearch-py and pandas.

The aims of the project are as follows:

• Learning about refactoring software code to improve its functionality.
• Understanding the GrimoireLab components (Perceval, ELK, Mordred) and the corresponding tool-chain.
• Replacing low-level libraries (e.g., requests) with popular ones used to interact with ElasticSearch.
• Enabling the correct working of ELK for different version of ElasticSearch (>=6.1) and Open Distro for ElasticSearch (>= 0.9.0).
• Reorganizing part of the ELK logic into coherent packages.
• Improving the processing of Perceval data.

The aims will require working with Python, ELK and the ElasticSearch database.

• Difficulty: Medium
• Requirements: Python programming. Interest in software analytics. Willingness to understand GrimoireLab internals.
• Recommended: Experience with ElasticSearch and Pandas would be convenient, but can be learned during the project.
• Mentors: @inishchith, @Polaris000, @sduenas, @valeriocos, @zhquan

Microtasks

For becoming familiar with GrimoireLab, you can start by reading some documentation. You can find useful information at:

• GrimoireLab Tutorial
• Perceval: Software Project Data at Your Will
• The GitHub repositories hosting the tools (https://github.com/chaoss/grimoirelab-perceval, https://github.com/chaoss/grimoirelab-sirmordred, https://github.com/chaoss/grimoirelab-...)

Once you're familiar with Grimoirelab, you can have a look at the following microtasks.

• Microtask 0: Download PyCharm and get familiar with it (for instance, you can follow this tutorial).

• Microtask 1: Set up Perceval to be executed from PyCharm.

• Microtask 2: Create a Python script to execute Perceval via its Python interface using the Git and GitHub backends. Feel free to select any target repository.

• Microtask 3: Based on the JSON documents produced by Perceval and its source code, try to answer the following questions:

  • What is the meaning of the JSON attribute timestamp?
  • What is the meaning of the JSON attribute updated_on?
  • What is the meaning of the JSON attribute origin?
  • What is the meaning of the JSON attribute category?
  • How many categories do the Git and GitHub backends have?
  • What is the meaning of the JSON attribute uuid?
  • What is the meaning of the JSON attribute search_fields?
  • What is stored in the attribute data of each JSON document produced by Perceval?
  • Identify the code in charge of dealing with remote APIs and explain its logic.
  • Which is the folder that stores the archives generated by Perceval?

• Microtask 4: Set up a dev environment to work on GrimoireLab. Have a look to https://github.com/chaoss/grimoirelab-sirmordred#setting-up-a-pycharm-dev-environment.

• Microtask 5: Execute micro-mordred to collect and enrich data from any Git repository.

• Microtask 6: Execute micro-mordred to obtain data from the study enrich_areas_of_code for any Git repository.

• Microtask 7: Execute micro-mordred to collect and enrich data from any GitHub repository, making sure that no archives are created by Perceval.

• Microtask 8: In your machine, run the tests for ELK within PyCharm. If you succeed, you can try to run the coverage package on the ELK tests and report the details of each file.

• Microtask 9: Submit at least a PR to one of the GrimoireLab repositories to fix an issue, improve the documentation, etc.

• Microtask 10: Submit a PR to ELK to increase the test coverage of one or more files located at https://github.com/chaoss/grimoirelab-elk/tree/master/grimoire_elk/enriched

[imnitishng]

Congrats to the students who got selected. The results were quite shocking. Anyway, I learnt a lot contributing to the projects under Grimoirelab. With all due respect I would love to get a feedback from the mentors regarding my work, things I did wrong or the areas I should work upon. Thank You @CHAOSS

[kshitij3199]

I would also like to congratulate all the GSoC selected students. Hope you do your best!! This was the first time I was contributing to an open-source community and I have learnt a lot.

I would like to especially thank @valeriocos for your support and all the doubts that you have solved.
Also @valeriocos can you please tell me what things I lacked/missed because of which I was not able to get selected for GSoC. This will help me to improve myself.

And also I am a bit surprised that why no one has been selected for the project "Boosting data processing in GrimoireLab" ??.Is it that all the applicants are not qualified for the project. If so, please tell where we lack so that we can improve ourselves.

Thankyou @valeriocos @chaoss for your time and support

[valeriocos]

I would like to thank all of you for your time and interest to this idea.

The decision process was divided into evaluation and discussion. In the evaluation phase, the mentors checked the proposals based on different dimensions (e.g., clarity, soundness, complexity, microtasks). In the discussion phase, the mentors shared their views on the proposals and democratically made a decision.

The technical skills of the applicants were good, however we realized that the time needed to get confortable with ELK was longer than expected. We considered it as a potential risk for the student, and thus we made a conservative decision and discard this idea.

Said that, I believe that the time, interest and contributions to GrimoireLab of @kshitij3199, @imnitishng and @snack0verflow have been remarkable, and it would be great if you would like to keep contributing to GrimoireLab. At the same time, I totally understand your mood right now.

[valeriocos]

If you have any questions, comments, or concerns about the selection process, feel free to write here or send an email to valcos@bitergia.com. Thanks!

This issue is going to be closed on Friday.

[kshitij3199]

Hi @valeriocos, Thankyou for your reply. 😃 I got your point.

Just a small request to all the mentors, I request all the mentors to please discuss all the projects idea before posting them and remove the risky/not required projects idea. Because Later when the project idea is discarded, all the efforts and time of the students go in vain. And also they feel somewhat discouraged and cheated.

Anyways, Thankyou @valeriocos for all your help and support. Will be contributing to GrimoireLab whenever possible.

[valeriocos]

Thank you @kshitij3199 for the suggestion