Pipeline rebuild

[forstmeier]

Overview

Description

This is a “base of operations” for the rebuild. See the sections below for details on implementation and the overall impetus/structure behind the project.

Flow

As far as the directory structure goes, it will basically be moving conflation/ to live within pipeline/ directly. Data flow would look something like this:

frontend/ -> gateway/ -> ingestor/ -> conflation/ -> backend/

Components

• frontend: hosts website
  • stays as-is
• gateway: fetch GitHub raw data, defined by GraphQL queries, built in "filters"
  • update to APIv4 ( #42 ), specific filters will be built into queries (replacing conflation Scenarios)
  • the transition to APIv4 will likely the be last thing done as part of this pipeline rebuild
• ingestor: populates data into MemSQL, raw form, some metadata additions
  • stays as-is (essentially)
• conflation: identifies object relationships, builds backend training object, identifies dependent variable per object
  • implement dependent-pin logic ( #41 ), conflation manipulation logic moved to backend
  • note: dependent-pin and feature-web are taking the place of the old "normalization"
• backend: necessary conflation manipulation (e.g. "bag of words"), feed into models, learn/predict, generate output
  • add conflation manipulation logic, change struct order ( #18 ) and possibly add fields ( #15 )
  • note: the old "conflation" logic will now be replaced by manipulation logic (which will likely do things like stitch text bodies together)
• general
  • address all golint concerns ( #44 )

Modules

dependent-pin

• Identifies the individual who "owns" a given text body for a GitHub object. Logic here is a series of assumptions presented in “if” logic with regards to selecting Assignees field values. In terms the model, this is identifying the dependent variable in the training process. “Who did what” basically.

  feature-web

• Creates "webs" from related feature "nodes" for relational context and model evaluation and connects them via metadata. Additionally, this would provide an “object hierarchy” which could allow the concept of “partial credit” to enter the system. Implementing this would likely involve the GraphQL GitHub APIv4.

Process

• dependent-pin would find the dependent variables (the User) for the training data
• feature-web would run to build a “web” that could be mutated/weighted when our existing model trains

Breakdown

dependent-pin

• Assumptions:
  • Commit objects have direct relationships to code changes
  • PullRequest(s) hold commits
  • Merged PullRequest(s) and their related commit(s) are value-added signals to who gets credit for fixes
• Logic steps:

Step	Reason	Word bag
A commit maker receives credit for the object	The maker individual actually made and submitted code changes to the code base	Commit message body/title, PullRequest body/title
If the PullRequest or commit closes an Issue, they receive credit also for that Issue	The fixer understood the task and implemented a solution for it	PullRequest body/title, commit message body/title, Issue body/title
“Naked” closed Issue(s) are ignored for the purposes of training the model	There is no concise way of identifying who fixed the issue (one exception may be using the original Assignee field)	N/A

feature-web

• [think in terms of this being the “normalization” / “linking” logic currently implemented]
• Example: in the case of a model like Bhattacharya which relies on text bodies, there would be a directive provided to the feature-web logic which would look to link together text objects (in order to expand the "bag of words" for processing).
  • This could include objects on the issue itself such as title, labels, and text body. Other objects would be things like comments, commit messages, and commit diff comments.

Takeaway

The dependent-pin and feature-web could be features that we build and implement right away as they are “expansions” on current logic/structure.

[taylormike]

"specific filters will be built into queries (replacing conflation Scenarios)" Looks good but this^^ is out of our hands from a technical standpoint. That either works or it doesn't. (Nothing we can really do unless we fork GraphQL and change it) Or have both the GoLang scenarios and GraphQL filters.

[forstmeier]

Agreed; what I meant (and was unclear about) was that the specific data we want from GitHub will be defined in the GraphQL query as opposed to being defined in scenario(s) like we currently have. But yes, it is out of our hands since we would be relying on the current state of GraphQL/GitHub APIv4.

[taylormike]

The good news is Github does respond to feedback and there GraphQL schema is updated everyday.

[forstmeier]

This "parent issue" is being shut down in favor of pursuing the individual sub-issues as updates to the heupr/core repository; some of the larger concepts (e.g. dependent-pin and feature-web) are being moved to the heupr/heupr repository.