Issue 438 - ( rewrited a part of PR #443 ) the logic that manage the commits from github repositories ( OOP )

[raduiman-zz]

This PR will:

• Finished all base functionality explained in Issue #438 -> filter the commits depending of repository type, repository version, repository files that affect CI and keywords.

Public methods:

FICGithub().start(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/bf48b7609f0baff69110c2a2891c096b2d2eba6e

• I have implemented start() public method, the main method to call after the attribute repo_name is initialized in FICDataVault. The method will return a dictionary containing all the new commits that could affect the infrastructure.

How to use:

• for all Github repositories:

  construct = FICGithub()
  for repo in json.load(construct.load(None, "repositories.json")).get("Github"):
  construct.repo_name = repo
  construct.start()

• only for one repository

  construct = FICGithub()
  construct.repo_name = "pushapkscript"
  construct.start()

  it returns:

  pushapkscript
  {1: 
  {'sha': 'a46ca21b8f0fc58058f448efe9f10d9ebbd63765', 
  'url': 'https://api.github.com/repos/mozilla-releng/pushapkscript/commits/a46ca21b8f0fc58058f448efe9f10d9ebbd63765', 
  'author': 'Mitchell Hentges', 
  'author_email': 'mhentges@mozilla.com', 
  'message': '1.0.3', 
  'date': '2019-05-07T20:30:46Z', 
  'files': ['version.txt']}}

FICGithub().get_repo_url(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/7db3e7e3392078d8ed7861ebde9075583959ad41

• method that return the URL of the repository

Private methods:

self._repo_team(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/a1174b56ed252b102c0d0b666dc67ccd9750f6cf

• initialize self.team_name attribute with the owner/team of the repository.

self._repo_files(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/dc752cc5d303ef1e8d1daa507a44654cce27d073

• initialize self.folders_to_check attribute with the folders-to-check from repositories.json

self._extract_repo_type(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/75148feebea8ce95ecc9585d73b1fb250650c0fb

• initialize self.repo_type attribute with the type of the repository

self._local_version(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/1d1ef280fdee399cb380c3dbc0a8c858efe31b23

• initialize self.local_version attribute with the version stored into the element 0 from .json file

self._get_release(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/514b36519fb39bba662c2dd2ec2b3d8031f539d2

• initialize self.release_version attribute with the tag from the remote repository

self._get_version_path() and self._build_puppet_version(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/1cf960211b835317360c7455d4bdb9b9df27dc03

• initialize self.build_puppet_version attribute with the version from build-puppet for the scriptworkers

self._compare_versions(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/1cf960211b835317360c7455d4bdb9b9df27dc03

• returns True if the release version is equal with the version from build-puppet and if those are higher than the local version

self._last_checked(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/708a42cde62b3a3fd00f16dcbc83892ff020cae6

• initialize self.last_check attribute with the datetime from element0/.json file

self._commit_iterator(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/353bc86ff25ec7525030c901119f87436705fe1f

• iterates through the commits of the current repository

self._store_data():, self._get_sha():, self._get_message():, self. _get_date():, self._get_author():, self._get_author_email():, self._get_url():, self._get_files(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/353bc86ff25ec7525030c901119f87436705fe1f

• initialize all the commit information

self._compare_files(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/839dc589e2a9c0c5675f3db748b748798c29d45f

• check if the changed files are affecting the infrastructure

self._construct_commit(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/1b6010573b4c106cb382214594064749e62df2a7

• append the commit to the self.list_of_commits dictionary

self._commit_filter(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/a187b3e342f20591d00b6488f1258b612a3bd7f5

• this method saves only the commits we care about: the commits that contain "deploy" or the release_version in the message, the commits that change the CI's files or the all commits for the repositories that requires this

  self._repo_switcher(): https://github.com/mozilla-releng/firefox-infra-changelog/pull/448/commits/981b869dbf93c29e818e8d2fdf59e303298d14b6

• distributes the repositories depending of their type
  • self._not_tag():, self._build_puppet():, self._tag():, self._commit_keyword():

For testing you can use the following patch:

issue-438-1.txt

[zsoltfay]

Added changes to FIC_Github.py and FIC_FileHandler.py

FIC_Github.py changes 6a2e2cb

_last_checked()

• added import datetime to convert the local "last_checked" string into a datetime object

_commit_filter()

• now accept a new argument commit_content. Because of this change the script will no longer store the data of every commit we verify, however only of the ones we care about saving

_repo_switcher()

• has been renamed to _repo_type_checker() so it is easier to understand its functionality

Removed hard coded argument on line 290

• changed "data" into CHANGELOG_REPO_PATH
• added + ".json" to form the correct path

FIC_FileHandler.py changes 882f5e8

_check_repo_files()

• now appends tupples to _missing_files. This allows for an easier logic when writing the first element into the json files

_create_missing_repo_files()

• now uses the construct_path() method when checking for files
• writes the first element into the newly created json files

_generate_first_element_git()

• NEW METHOD: returns the appropriate dictionary that should be written to the git json files

_generate_first_element_hg()

• NEW METHOD: returns the appropriate dictionary that should be written to the hg json files

_def _extract_repo_type()

• NEW METHOD: returns the type of a git repostitory