gotec
commented
4 years ago Hi Sebastian,
Thank you very much for taking the time to prepare this tutorial! I also heavily use pandas to explore and analyse the repository data both during and after mining so this approach fits very nicely to git2net.
I have a couple of small questions and comments regarding the proposed pandas exporter, as well as the tutorial before integrating it with the rest of the project:
Pandas Exporter:
- How do you envision a pandas exporter to look like, i.e. which functionality would you want it to have? In my research I usually look at large open source projects (e.g. the Linux kernel project) which quickly results in databases with multiple GB in size and tens of millions of rows. Further, the mining process can take multiple weeks. For this reason, I think directly outputting the mining results as a pandas dataframe is not feasible.
- When analysing (large) repositories, I am also rarely interested in all columns of the databases. Further, I usually use the
chunksizeargument to sequentially parse large tables. Thus, all these options would need to be supported by a proper pandas exporter in the way I understand your proposal. - Overall, I think it might be a better path to have users write the sql queries themselves (like you do in the tutorial), rather than providing a wrapper with all the functionality described above. Of course this does not invalidate discussing it in the tutorial. Quite the opposite actually, I think it makes it even more fitting to discuss it there. However, I am very curious and open to your opinion on this.
Tutorial:
- In the diagram in the first box it should read "Ready for analysis :-)", as the mining would have already happened at this point, correct?
- Given that the data generation section is (content wise) identical to the original tutorial, I propose to integrate the data analysis section into the main TUTORIAL.ipynb notebook as an additional (last) section. Then, the tutorial would first describe the cloning and mining process and follow with the (more general) network analysis part. Finally, it could then end with the introduction of the more detailed analysis methods facilitated through pandas.
- I propose to avoid mentioning 136 commits in the text as this number will (already) be wrong in the moment I merge the pull request.
- While seeing the output of the cells was great for reviewing the pull request I propose to remove all outputs from the notebook before integrating it, as otherwise the state of the notebook's output will also be tracked by git.
Again thanks a lot for the very nicely written tutorial and the entertaining examples throughout. I am looking forward to hear your thoughts on the points above!
Cheers, Christoph
Hi,
storing commit data in a sqlite db is nice, but by applying python pandas we are able to explore and analyze the data set more comfortable. In order to support new user, I prepared a small tutorial introducing the tool-chain.
Probably it would be interesting to include a pandas exporter directly?
Sebastian