chaoss / grimoirelab-elk

GNU General Public License v3.0
59 stars 120 forks source link

Add functionality for producing enriched indexes as csv files #337

Open jgbarah opened 6 years ago

jgbarah commented 6 years ago

Now, GrimoireELK is targeted at producing raw and enriched indexes in Elasticsearch. But in some scenarios, getting a CSV file with the information we usually have in an enriched index would be very convenient, specially if that can be done without accessing Elasticsearch.

Reading the code, it seems this is maybe not straightforward, but not that difficult either. And maybe this could be a first step in decoupling GrimoireELK from Elasticsearch.

Please, feel free to add comments now the idea, including to which extent it makes sense to you, and you would implement it.

alpgarcia commented 6 years ago

When I implemented areas of code and onion studies I tried to use a simple approach for reading and writing: https://github.com/chaoss/grimoirelab-elk/blob/master/grimoire_elk/enriched/ceres_base.py#L36

Then you can use a common code to read: https://github.com/chaoss/grimoirelab-elk/blob/master/grimoire_elk/enriched/ceres_base.py#L90

and write: https://github.com/chaoss/grimoirelab-elk/blob/master/grimoire_elk/enriched/ceres_base.py#L96

I think we should follow an approach similar to this one (maybe not so simple, or maybe even simpler) in such a way that producing new indexes should be as easy as implement the data access layer and use it together the code to process data.

jgbarah commented 6 years ago

Thanks for the links to Ceres, @alpgarcia. Yes, I was thinking about something like that, but I don't know how difficult it would be to implement that in the current GrimoireELK codebase...