Gleaning project meta-data

[julianharty]

Context

Code repositories (repos) contain code written in one or more programming languages. There may be connections between the programming language and: unit testing frameworks, programming conventions, and code quality, etc. We don't yet know much about these connections, a pre-requisite is to obtain the information. GitHub provides an API and uses an opensource library to obtain this information from a repo (apparently they also use another project that's not well documented).

Let's start by using the GitHub API as that'll quickly enable us to obtain that information for projects hosted on github.com then we can explore the suitability of various opensource

More info

• About repository languages
• REST API endpoints for repository statistics - e.g. potentially useful for helping assess liveness vs abandonment of a codebase.
• REST API endpoints for commits (already implemented).
  • https://stackoverflow.com/questions/45726013/how-can-i-get-last-commit-from-github-api- is also helpful on ways to obtain the information about the latest commit.
  • https://www.feststelltaste.de/reading-a-git-repos-commit-history-with-pandas-efficiently/ and https://github.com/feststelltaste/software-analytics/blob/master/notebooks/Reading%20a%20Git%20repo's%20commit%20history%20with%20Pandas%20efficiently.ipynb is also really useful

[julianharty]

PS: information gleaned from the Issues of a github hosted repo may also be insightful, https://docs.github.com/en/rest/using-the-rest-api/issue-event-types?apiVersion=2022-11-28 provides one approach, and some work I documented in https://stackoverflow.com/a/56439604/340175 (https://stackoverflow.com/questions/56437562/seeking-a-way-to-clone-the-issues-for-github-projects) another.

[julianharty]

• https://pypi.org/project/guesslang/ is probably a good first start for us as it's a python package so easy to integrate: "Guesslang detects the programming language of a given source code. It supports more than 50 programming languages and detects the correct programming language with more than 90% accuracy." Last release 2021 so not being updated?
  • However as it's not been updated it won't install with python 3.10 or newer (https://stackoverflow.com/questions/75112136/python-unable-to-install-guesslang)
• https://stackoverflow.com/questions/325165/is-there-a-library-that-will-detect-the-source-code-language-of-a-block-of-code
• https://github.blog/2019-07-02-c-or-java-typescript-or-javascript-machine-learning-based-classification-of-programming-languages/
• https://www.scivision.dev/detect-project-primary-languages/
• https://github.com/scivision/linguist-python
• https://highlightjs.org/
• https://github.com/googlearchive/code-prettify (sadly archived at least 5 years ago)
• https://github.com/korsakov/ohcount (archived 16 years ago!)

[julianharty]

I've forked guesslang and updated the requirements.txt so that the updated version will run in python versions 3.8 to 3.11 according to the docs (details in https://github.com/commercetest/guesslang/issues/1)

As I've not published an installable package, the updated version needs to be installed locally from a clone of the fork'd repo, see and read the information about the local install there https://github.com/commercetest/guesslang?tab=readme-ov-file#installation. More completely:

• git clone https://github.com/commercetest/guesslang.git
• cd guesslang
• in the same python environment e.g. in the same conda environment, run pip3 install .
• Check it's installed OK by trying to run guesslang --help