search

thoth-station / thoth-application

Thoth-Station ArgoCD Applications
GNU General Public License v3.0
12 stars 22 forks source link

Store GitHub targets more specifically, more scalabe #1630

Open xtuchyna opened 3 years ago

xtuchyna commented 3 years ago

User story

As a mi dataset requester, I want to specify github target and its entities, so that I can simply make PR to the global configmap that will have it registered.

Additional context When aggregating data for mi repositories (either for general mi inspection or a kebechet oriented analysis), the repositories and organization names are stored here https://github.com/thoth-station/thoth-application/blob/master/mi-scheduler/base/configmaps.yaml

However, this is not scalable and user-specific, because everything is stored inside the comma-sep styled string (it is then parsed in mi-scheduler). This also does not allow to specify entity individually for repo/org.

With recent changes to the schedule method in common for mi it is possible to specify repos with specific entities, so the only thing remains to store the list of tuples [ (repo/ord, entities), ... ] to the json file and load it as a configmap.

Acceptance Criteria

Linked epics / issues https://github.com/thoth-station/mi-scheduler/pull/139

xtuchyna commented 3 years ago

/kind feature /kind demo

xtuchyna commented 3 years ago

/assign @xtuchyna

goern commented 3 years ago

Why not create a config file that is read by MI and processed by a method. The config file would be an input the method, the purpose of the method is to create a dict, the dict is what MI should aggregate...

aggregate:
  - "org1/repo1": [ "PR", "issue" ]
  - "org1/repo2": [ "PR, issue" ]
  - "org1/repo3": ["PR"]
  - "org2/repo1: ["PR", "issue"]
  - "org2/repo2": ["PR"]

The config for this could look like:

aggregate:
  orgs:
    - "org1"
    - "org2"
  entities:
    - PR
    - issue
  excludeRepos:
    - repo3
  repos:
    - org1/repo3:
    - org2/repo2
    entities:
      - PR

pretty much like what Tide is doing at https://github.com/thoth-station/thoth-application/blob/master/prow/overlays/cnv-prod/config.yaml#L162

goern commented 3 years ago

/triage needs-information

goern commented 3 years ago

/remove-triage needs-information

goern commented 3 years ago

@xtuchyna are you working on this? if so, please lifecycle it, and set a prio

/priority backlog

sesheta commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

xtuchyna commented 3 years ago

/remove-lifecycle rotten

xtuchyna commented 3 years ago

/priority-important soon

sesheta commented 3 years ago

@xtuchyna: The label(s) priority/soon cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/thoth-station/thoth-application/issues/1630#issuecomment-889826232): >/priority-important soon Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
sesheta commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

sesheta commented 3 years ago

@sesheta: Closing this issue.

In response to [this](https://github.com/thoth-station/thoth-application/issues/1630#issuecomment-912982606): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
xtuchyna commented 3 years ago

/reopen /remove-lifecycle rotten

sesheta commented 3 years ago

@xtuchyna: Reopened this issue.

In response to [this](https://github.com/thoth-station/thoth-application/issues/1630#issuecomment-943241644): >/reopen >/remove-lifecycle rotten Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
codificat commented 2 years ago

/priority-important soon

I will take that as /triage accepted