WIP: Upstream Parsers

[jialincheoh]

• we should be parsing the control file and not the package info
• the repo should be git clone and not just browsing through the control file
• fields that are planned with the parsers are: source_name, version, location, author_name, author_email

[jialincheoh]

Tasks accomplish in this commit @SantiagoTorres, @SahithiKasim, @absol27

• [x] I removed the prior script that doesn't git clone the folder at https://salsa.debian.org/python-team/packages
• [x] Uploaded script named clone_salsa.sh that git clone all packages in https://salsa.debian.org/python-team/packages with personal access token after I was given permission by gitlab@salsa.debian.org
• [x] Commented the script on what it currently does
• [x] Created a folder named upstream_data in the tower at the path /data/yellow/guacalytics/raw_data/upstream_data to put all the git clone data there.

[jialincheoh]

Extracted the below list of information with my latest github commit. File name titled get_salsa_repo_info.sh.

{
    "id": 84649,
    "name": "python-trio",
    "ssh_url": "git@salsa.debian.org:python-team/packages/python-trio.git",
    "http_url": "https://salsa.debian.org/python-team/packages/python-trio.git",
    "created_at": "2023-07-02T11:46:23.705Z",
    "statistics": null,
    "star_count": 0,
    "forks_count": 0,
    "last_activity_at": "2023-07-02T11:46:23.705Z",
    "path": "python-trio",
    "default_branch_protection": null,
    "open_issues_count": null,
    "visibility": "public",
    "archived": false,
    "description": null,
    "topics": [],
    "issues_enabled": false,
    "merge_requests_enabled": true,
    "shared_with_groups": [],
    "request_access_enabled": true
  }

[jialincheoh]

The table named upstream_table with the following fields are now in the database in the tower.

{
    "id": 84649,
    "name": "python-trio",
    "ssh_url": "git@salsa.debian.org:python-team/packages/python-trio.git",
    "http_url": "https://salsa.debian.org/python-team/packages/python-trio.git",
    "created_at": "2023-07-02T11:46:23.705Z",
    "statistics": null,
    "star_count": 0,
    "forks_count": 0,
    "last_activity_at": "2023-07-02T11:46:23.705Z",
    "path": "python-trio",
    "default_branch_protection": null,
    "open_issues_count": null,
    "visibility": "public",
    "archived": false,
    "description": null,
    "topics": [],
    "issues_enabled": false,
    "merge_requests_enabled": true,
    "shared_with_groups": [],
    "request_access_enabled": true
  }

[jialincheoh]

• [x] Combined cloning script and get repo info script named clone_extract_salsa_info.sh
• [x] Removed the prior 2 scripts.

[jialincheoh]

Updated the script clone_extract_salsa_info.sh , previously it doesn't clone all 3311 repo, only cloned 100 repos of https://salsa.debian.org/python-team/packages

[jialincheoh]

Latest Scripts @SantiagoTorres

scripts/ingestion/parsers/clone_repos.py

Functionalities

1) If repo exists, pull newest data 2) If repo not exists, clone the repo 3) Check directory if empty, if empty, log error on empty directory 4) If cannot clone repo such as https://packages.debian.org/stable/389-console, log git command error 5) If OSError, log OSError 6) log file named loginfo.log.

scripts/ingestion/parsers/control_to_json.py

Functionalities

1) Output information from control file in json format, such as the below

[
    {
        "Source": "thunar-volman",
        "Section": "xfce",
        "Priority": "optional",
        "Maintainer": "Debian Xfce Maintainers <debian-xfce@lists.debian.org>",
        "Uploaders": "Yves-Alexis Perez <corsac@debian.org>",
        "Build-Depends": "debhelper-compat (= 13), intltool, libexo-2-dev, libgtk-3-dev, libgudev-1.0-dev, libnotify-dev, libxfce4ui-2-dev, libxfce4util-dev, libxfconf-0-dev, xfce4-dev-tools (>= 4.16)",
        "Rules-Requires-Root": "no",
        "Standards-Version": "4.6.1",
        "Vcs-Git": "https://salsa.debian.org/xfce-team/goodies/thunar-volman.git",
        "Vcs-Browser": "https://salsa.debian.org/xfce-team/goodies/thunar-volman",
        "Homepage": "https://docs.xfce.org/xfce/thunar/thunar-volman",
        "Package": "thunar-volman",
        "Architecture": "linux-any",
        "Depends": "exo-utils, thunar, ${misc:Depends}, ${shlibs:Depends}",
        "Description": "Thunar extension for volumes management The Thunar Volume Manager is an extension for the Thunar file manager, which enables automatic management of removable drives and media."
    },
    {
        "Source": "kodi-audioencoder-flac",
        "Priority": "optional",
        "Section": "libs",
        "Maintainer": "Debian Multimedia Maintainers <debian-multimedia@lists.debian.org>",
        "Uploaders": "Vasyl Gello <vasek.gello@gmail.com>",
        "Build-Depends": "debhelper-compat (= 13)",
        "Build-Depends-Arch": "dh-sequence-kodiaddon (>= 2:20~), cmake, libflac-dev, libogg-dev",
        "Standards-Version": "4.6.2",
        "Rules-Requires-Root": "no",
        "Vcs-Browser": "https://salsa.debian.org/multimedia-team/kodi-media-center/kodi-audioencoder-flac",
        "Vcs-Git": "https://salsa.debian.org/multimedia-team/kodi-media-center/kodi-audioencoder-flac.git",
        "Homepage": "https://kodi.tv",
        "Package": "kodi-audioencoder-flac",
        "Architecture": "any",
        "Depends": "${shlibs:Depends},",
        "${misc": "Depends},",
        "${kodi": "AUDIOENCODERAPI}",
        "Description": "FLAC audio encoder add-on for Kodi Kodi, formerly known as XBMC is an award winning free and open source software media-player and entertainment hub for all your digital media. Kodi is available for Linux, Mac OS X (Leopard, Tiger and Apple TV) and Microsoft Windows, as well as the original Xbox game console. Created in 2003 by a group of like minded programmers, Kodi is a non-profit project run and developed by volunteers located around the world. More than 50 software developers have contributed to Kodi, and 100-plus translators have worked to expand its reach, making it available in more than 30 languages. . While Kodi functions very well as a standard media player application for your computer, it has been designed to be the perfect companion for your HTPC. Supporting an almost endless range of remote controls, and combined with its beautiful interface and powerful skinning engine, Kodi feels very natural to use from the couch and is the ideal solution for your home theater. Once installed, your computer will become a fully functional multimedia jukebox. . This package ia an audio encoder add-on for Kodi. It is needed to convert Audio-CD to FLAC."
    }
]

[SahithiKasim]

@absol27 is going to implement these scripts.