How to configure backends with multiple repos (in projects.json) and multiple categories (in mordred.cfg)

[lukaszgryglicki]

What if I have multiple repos and categories combinations? Like say for GitHub:

• I have 4 different repos.
• I want to pull issues and pull requests categories.

My understanding would be that I put 4 repos in projects.json, and 2 entries in mordred.cfg - one for issue and the other for pull_request, but why do I need to put full repo paths in projects.json while perceval github expects just org and repo.

It should call Perceval 4 * 2 = 8 times, right? 4 repos with 2 categories?

Can anybody confirm that my data sources configuration is OK in those two code snippets:

• mordred.cfg:

  
  projects_file = onos.json
  (...)
  [git]
  raw_index = git_raw
  enriched_index = git
  category = commit

[github:issue] api-token = xxx raw_index = github_raw enriched_index = github sleep-for-rate = true category = issue

[github:pull_request] api-token = xxx raw_index = github_raw enriched_index = github sleep-for-rate = true category = pull_request

[gerrit] user = xxx raw_index = gerrit_raw enriched_index = gerrit category = review

[slack] raw_index = slack_raw enriched_index = slack category = message api-token = xxx

[jira] raw_index = jira_raw enriched_index = jira category = issue backend-user = xxx backend-password = xxx verify = False

[confluence] raw_index = confluence_raw enriched_index = confluence category = 'historical content'

- `onos.json` (especially take a look at GitHub sources definition - Perceval expects `org` `repo`:

{ "ONOS": { "git": [ "https://gerrit.onosproject.org/onos", "https://gerrit.onosproject.org/onos-felix", "https://gerrit.onosproject.org/spring-open", "https://gerrit.onosproject.org/OnosSystemTest", "https://gerrit.onosproject.org/onos-loxi", "https://gerrit.onosproject.org/onos-yang-tools", "https://gerrit.onosproject.org/fabric-control", "https://gerrit.onosproject.org/onos-vm", "https://gerrit.onosproject.org/onos-app-samples", "https://gerrit.onosproject.org/vm-build", "https://gerrit.onosproject.org/routing", "https://gerrit.onosproject.org/olt-oftest", "https://gerrit.onosproject.org/onos-nemo", "https://gerrit.onosproject.org/spring-open-cli", "https://gerrit.onosproject.org/onos-gerrit-base", "https://gerrit.onosproject.org/manifest", "https://gerrit.onosproject.org/OnosSystemTestJenkins", "https://gerrit.onosproject.org/cord-openwrt" ], "github": [ "opennetworkinglab onos", "opennetworkinglab spring-open", "opennetworkinglab OnosSystemTest", "opennetworkinglab onos-loxi", "opennetworkinglab onos-yang-tools", "opennetworkinglab onos-app-samples", "opennetworkinglab routing", "opennetworkinglab spring-open-cli" ], "gerrit": [ "gerrit.onosproject.org" ], "slack": [ "C095YQBM2", "C095YQBLL" ], "jira": [ "http://jira.onosproject.org" ], "confluence": [ "http://wiki.onap.org" ] } }

Can anybody take a look at other definitions like Arthur etc and confirm that I'm doing this right?
- [mordred.cfg](https://github.com/lukaszgryglicki/bitergia_deployment/blob/master/mordred.cfg.sample)
- [onos.json](https://github.com/lukaszgryglicki/bitergia_deployment/blob/master/onos.json)

And it seems like there is no way to provide SortingHat's autogender API key, tried:

autogender = True autogender_api_token = xxx

Also tried - none works:
- autogender_api_token
- autogender_api-token
- autogender-api_token
- autogender-api-token

My GitHub data source is not working, I've even changed repos list to be (projects.json):

    "github": [
      "https://github.com/httopennetworkinglab/onos",
      "https://github.com/opennetworkinglab/spring-open",
      "https://github.com/opennetworkinglab/OnosSystemTest",
      "https://github.com/opennetworkinglab/onos-loxi",
      "https://github.com/opennetworkinglab/onos-yang-tools",
      "https://github.com/opennetworkinglab/onos-app-samples",
      "https://github.com/opennetworkinglab/routing",
      "https://github.com/opennetworkinglab/spring-open-cli"
    ],

and (mordred.cfg):

[github:issue] api-token = xxx raw_index = github_raw enriched_index = github sleep-for-rate = true category = issue

[github:pull_request] api-token = xxx raw_index = github_raw enriched_index = github sleep-for-rate = true category = pull_request

What can be wrong?

@jgbarah @valeriocos 

[valeriocos]

In the projects.json, you should have:

"github:issue": [ "https://github.com/httopennetworkinglab/onos", "https://github.com/opennetworkinglab/spring-open", "https://github.com/opennetworkinglab/OnosSystemTest", "https://github.com/opennetworkinglab/onos-loxi", "https://github.com/opennetworkinglab/onos-yang-tools", "https://github.com/opennetworkinglab/onos-app-samples", "https://github.com/opennetworkinglab/routing", "https://github.com/opennetworkinglab/spring-open-cli" ], "github:pull_request": [ "https://github.com/httopennetworkinglab/onos", "https://github.com/opennetworkinglab/spring-open", "https://github.com/opennetworkinglab/OnosSystemTest", "https://github.com/opennetworkinglab/onos-loxi", "https://github.com/opennetworkinglab/onos-yang-tools", "https://github.com/opennetworkinglab/onos-app-samples", "https://github.com/opennetworkinglab/routing", "https://github.com/opennetworkinglab/spring-open-cli" ], The names of the sections in the cfg should have a 1-to-1 correspondence with the keys in the projects.json

[lukaszgryglicki]

OK, that makes one fix, thanks. Trying... But can I use "opennetworkinglab onos" and "opennetworkinglab spring-open" as repo names, or I need to specify full paths? This is counterintuitive to what perceval expects...

[valeriocos]

why do I need to put full repo paths in projects.json while perceval github expects just org and repo.

I guess @acs can give you an answer

WRT your mordred.cfg everything looks OK (considering your PR to mordred). Note that the category is not needed if you don't use arthur

[lukaszgryglicki]

I'm using Arthur.

[valeriocos]

WRT bitergia_deployment..onos.json, everything is OK.

WRT bitergia_deployment..mordred.cfg.sample, I'm not sure about some params in the sortinghat section (identities_api_token and autogender_api_token), probably @acs or @jgbarah can answer .

Minor comment about the boolean values in the cfg, we use them in lowercase.

[jgbarah]

why do I need to put full repo paths in projects.json while perceval github expects just org and repo.

AFAIK, urls are the only format. Very likely that's just due to historical reasons, and yes, it would be nice to support also the Perceval format. OTOH, being able of using the same url for issues, pull requests and git repos (in the case of GitHub or GitLab) also makes sense...

[jgbarah]

WRT bitergia_deployment..mordred.cfg.sample, I'm not sure about some params in the sortinghat section (identities_api_token and autogender_api_token)

I think autogender_api_token is for genderize.io, in case you want a gender analysis by name (but I'm not completely sure). I've never used identities_api_token. I think @acs is the person who could know (or otherwise, we should look at the source code).

[lukaszgryglicki]

identities_api_token it is for GitHub afaik, for getting GitHub users data. autogender_api_token should be for genderize.io, but I cannot find any name to be recognized to set this API token. I've looked in the sources and I've only seen the code for identities_api_token, so I guess setting Genderize.io token is unsupported - so we're limited to 1000 calls/day (which is almost always ok, excluding the first run).

[jjmerchante]

I think this issue is a bit old and doesn't match the current status of the project.

Now the autogender is executed in SortingHat, creating a new job from the menu. The token can be defined using the environment variable SORTINGHAT_GENDERIZE_API_KEY. It uses https://genderize.io/ API.

Closing this issue.