search

jenkinsci / gitlab-branch-source-plugin

A Jenkins Plugin for GitLab Multibranch Pipeline jobs and Folder Organization
https://plugins.jenkins.io/gitlab-branch-source
MIT License
121 stars 87 forks source link

Scanner only keeping first 10 detected repositories #180

Closed robross0606 closed 2 years ago

robross0606 commented 2 years ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.319.3 OS: Linux - 4.14.262-135.489.amzn1.x86_64 --- Office-365-Connector:4.15.2 ace-editor:1.1 ansicolor:1.0.1 ant:1.13 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 audit-trail:3.10 authentication-tokens:1.4 aws-bucket-credentials:1.0.0 aws-credentials:189.v3551d5642995 aws-java-sdk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-cloudformation:1.12.163-315.v2b_716ec8e4df aws-java-sdk-codebuild:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ec2:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecr:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-elasticbeanstalk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-iam:1.12.163-315.v2b_716ec8e4df aws-java-sdk-logs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-minimal:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ssm:1.12.163-315.v2b_716ec8e4df azure-commons:1.1.3 badge:1.9.1 basic-branch-build-strategies:1.3.2 bitbucket:223.vd12f2bca5430 blueocean:1.25.3 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.3 blueocean-commons:1.25.3 blueocean-config:1.25.3 blueocean-core-js:1.25.3 blueocean-dashboard:1.25.3 blueocean-display-url:2.4.1 blueocean-events:1.25.3 blueocean-git-pipeline:1.25.3 blueocean-github-pipeline:1.25.3 blueocean-i18n:1.25.3 blueocean-jira:1.25.3 blueocean-jwt:1.25.3 blueocean-personalization:1.25.3 blueocean-pipeline-api-impl:1.25.3 blueocean-pipeline-editor:1.25.3 blueocean-pipeline-scm-api:1.25.3 blueocean-rest:1.25.3 blueocean-rest-impl:1.25.3 blueocean-web:1.25.3 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-6 bouncycastle-api:2.25 branch-api:2.7.0 build-metrics:1.3 build-timeout:1.20 caffeine-api:2.9.2-29.v717aac953ff3 categorized-view:1.12 checks-api:1.7.2 cloudbees-bitbucket-branch-source:757.vddedc5f2589a_ cloudbees-folder:6.708.ve61636eb_65a_5 code-coverage-api:2.0.4 command-launcher:1.6 conditional-buildstep:1.4.1 config-file-provider:3.9.0 copyartifact:1.46.2 credentials:1074.v60e6c29b_b_44b_ credentials-binding:1.27.1 data-tables-api:1.11.4-2 dependency-check-jenkins-plugin:5.1.2 deploy:1.16 disk-usage:0.28 display-url-api:2.3.5 docker-commons:1.19 docker-workflow:1.28 durable-task:493.v195aefbb0ff2 echarts-api:5.3.0-2 email-ext:2.87 envinject:2.839.v52c702c10635 envinject-api:1.180.v98d833b_27470 extra-columns:1.25 favorite:2.4.0 file-leak-detector:1.7 font-awesome-api:6.0.0-1 forensics-api:1.8.1 git:4.10.3 git-client:3.11.0 git-server:1.10 github:1.34.3 github-api:1.301-378.v9807bd746da5 github-branch-source:2.11.4 gitlab-api:1.0.6 gitlab-branch-source:1.5.9 gitlab-plugin:1.5.28 global-build-stats:1.5 gradle:1.38 groovy:2.4 groovy-postbuild:2.5 h2-api:1.4.199 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 htmlpublisher:1.29 http_request:1.14 jackson2-api:2.13.1-246.va8a9f3eaf46a javadoc:217.v905b_86277a_2a_ javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jdk-tool:1.5 jenkins-design-language:1.25.3 jersey2-api:2.35-4 jira:3.7 jjwt-api:0.11.2-9.c8b45b8bb173 jnr-posix-api:3.1.7-2 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.54 kubernetes-cd:2.3.1 ldap:2.8 lockable-resources:2.14 mailer:408.vd726a_1130320 mapdb-api:1.0.9.0 matrix-auth:3.1 matrix-project:1.20 mattermost:3.1.1 maven-plugin:3.16 mercurial:2.16 metrics:4.1.6.1 metrics-diskusage:3.0.0 momentjs:1.1.1 monitoring:1.90.0 nested-view:1.24 nexus-artifact-uploader:2.13 nexus-jenkins-plugin:3.13.20220201-143240.3d657a5 nodejs:1.5.1 okhttp-api:4.9.3-105.vb96869f8ac3a pam-auth:1.7 parameterized-trigger:2.43 pipeline-aws:1.43 pipeline-build-step:2.16 pipeline-github-lib:36.v4c01db_ca_ed16 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:446.vf27b_0b_83500e pipeline-maven:3.10.0 pipeline-milestone-step:100.v60a_03cd446e1 pipeline-model-api:1.9.3 pipeline-model-definition:1.9.3 pipeline-model-extensions:1.9.3 pipeline-npm:0.9.2 pipeline-rest-api:2.23 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:1.9.3 pipeline-stage-view:2.23 pipeline-utility-steps:2.12.0 plain-credentials:1.8 plugin-usage-plugin:2.2 plugin-util-api:2.14.0 popper-api:1.16.1-2 popper2-api:2.11.2-1 pubsub-light:1.16 resource-disposer:0.17 rich-text-publisher-plugin:1.4 run-condition:1.5 run-condition-extras:0.3 s3:0.12.1 saferestart:0.3 scm-api:595.vd5a_df5eb_0e39 script-security:1138.v8e727069a_025 shelve-project-plugin:3.2 shiningpanda:0.24 simple-theme-plugin:103.va_161d09c38c7 snakeyaml-api:1.29.1 sonar:2.14 sse-gateway:1.25 ssh-agent:1.24.1 ssh-credentials:1.19 ssh-slaves:1.806.v2253cedd3295 ssh-steps:2.0.0 sshd:3.1.0 structs:308.v852b473a2b8c subversion:2.15.2 support-core:2.80 timestamper:1.17 token-macro:280.v97a_82642793c toolenv:1.2 trilead-api:1.0.13 variant:1.4 view-job-filters:2.3 violation-comments-to-gitlab:2.52 violation-comments-to-stash:1.128 windows-slaves:1.8 workflow-aggregator:2.7 workflow-api:1138.v619fd5201b_2f workflow-basic-steps:2.24 workflow-cps:2660.vb_c0412dc4e6d workflow-cps-global-lib:564.ve62a_4eb_b_e039 workflow-durable-task-step:1121.va_65b_d2701486 workflow-job:1145.v7f2433caa07f workflow-multibranch:711.vdfef37cda_816 workflow-scm-step:2.13 workflow-step-api:622.vb_8e7c15b_c95a_ workflow-support:813.vb_d7c3d2984a_0 ws-cleanup:0.40 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

NAME="Amazon Linux AMI" VERSION="2018.03" ID="amzn" ID_LIKE="rhel fedora" VERSION_ID="2018.03" PRETTY_NAME="Amazon Linux AMI 2018.03" ANSI_COLOR="0;33" CPE_NAME="cpe:/o:amazon:linux:2018.03:ga" HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

Reproduction steps

  1. Create a GitLab Group
  2. Create more than 10 repositories in the group
  3. Create a Jenkins Organizational Folder job pointing to the group
  4. Scan for repositories

Expected Results

It should find all the repositories and create them as jobs in the organizational folder.

Actual Results

It scans and finds all the repositories. Then is seems to intentionally disregard anything more than the first 10 and actively removes them.

Scan log:

[Wed Mar 09 20:50:07 UTC 2022] Finished organization scan. Scan took 6.9 sec
Started
[Wed Mar 09 21:19:42 UTC 2022] Starting organization scan...
[Wed Mar 09 21:19:42 UTC 2022] Updating actions...
Group URL: [Nurocor Library](https://git.mycompany.com/groups/nl)
[Wed Mar 09 21:19:43 UTC 2022] Consulting GitLab Group
Checking projects...

Checking project [UI Components](https://git.mycompany.com/nl/ui-components)
Web hook already created
Ignoring UI Components

Checking project [NCP UI Module](https://git.mycompany.com/nl/ncp-ui-module)
Web hook already created
Ignoring NCP UI Module

Checking project [GraphQL Voyager](https://git.mycompany.com/nl/graphql-voyager)
Web hook already created
Ignoring GraphQL Voyager

Checking project [ESlint Config](https://git.mycompany.com/nl/eslint-config)
Web hook already created
Ignoring ESlint Config

Checking project [Cypress Nurocor Reporter](https://git.mycompany.com/nl/cypress-nurocor-reporter)
Web hook already created
Ignoring Cypress Nurocor Reporter

Checking project [Resource Info - NodeJS](https://git.mycompany.com/nl/resource-info-nodejs)
Web hook already created
Ignoring Resource Info - NodeJS

Checking project [Property Validation - NodeJS](https://git.mycompany.com/nl/property-validation-nodejs)
Web hook already created
Ignoring Property Validation - NodeJS

Checking project [PostgreSQL Client - NodeJS](https://git.mycompany.com/nl/postgresql-client-nodejs)
Web hook already created
Ignoring PostgreSQL Client - NodeJS

Checking project [OSS License Reporter for NPM](https://git.mycompany.com/nl/oss-license-reporter-for-npm)
Web hook already created
Ignoring OSS License Reporter for NPM

Checking project [Objective and Endpoint Template UI](https://git.mycompany.com/nl/object-and-endpoint-template-ui)
Web hook already created
Ignoring Objective and Endpoint Template UI

Checking project [Node Server Utils](https://git.mycompany.com/nl/node-server-utils)
Web hook already created
Ignoring Node Server Utils

Checking project [Node GraphQL Utils](https://git.mycompany.com/nl/node-graphql-utils)
Web hook already created
Ignoring Node GraphQL Utils

Checking project [Node Error Utils](https://git.mycompany.com/nl/node-error-utils)
Web hook already created
Ignoring Node Error Utils

Checking project [Node Arango Utils](https://git.mycompany.com/nl/node-arango-utils)
Web hook already created
Ignoring Node Arango Utils

Checking project [NCP Service Info](https://git.mycompany.com/nl/ncp-service-info)
Web hook already created
Ignoring NCP Service Info

Checking project [NCP Metadata - NodeJS](https://git.mycompany.com/nl/ncp-metadata-nodejs)
Web hook already created
Ignoring NCP Metadata - NodeJS

Checking project [NCP Context - NodeJS](https://git.mycompany.com/nl/ncp-context-nodejs)
Web hook already created
Ignoring NCP Context - NodeJS

Checking project [MDR Config Tool](https://git.mycompany.com/nl/mdr-config-tool)
Web hook already created
Ignoring MDR Config Tool

Checking project [Log Client - NodeJS](https://git.mycompany.com/nl/log-client-nodejs)
Web hook already created
Ignoring Log Client - NodeJS

Checking project [Log Client - Java](https://git.mycompany.com/nl/log-client-java)
Web hook already created
Ignoring Log Client - Java

Checking project [Cypress to Test Manager](https://git.mycompany.com/nl/cypress-to-test-manager)
Web hook already created
Ignoring Cypress to Test Manager

Checking project [AMQP Utils - NodeJS](https://git.mycompany.com/nl/amqp-utils-nodejs)
Web hook already created
Ignoring AMQP Utils - NodeJS

Checking project [AMQP Mocks](https://git.mycompany.com/nl/amqp-mocks)
Web hook already created
Ignoring AMQP Mocks

Checking project [Add Copyright Comment](https://git.mycompany.com/nl/add-copyright-comment)
Web hook already created
Ignoring Add Copyright Comment

Checking project [Access Utils - NodeJS](https://git.mycompany.com/nl/access-utils-nodejs)
Web hook already created
Ignoring Access Utils - NodeJS

25 projects were processed
[Wed Mar 09 21:19:45 UTC 2022] Finished organization scan. Scan took 3.7 sec
Evaluating orphaned items in Nurocor Library
Will not remove objective-and-endpoint-template-ui as it is only #1 in the list
Will not remove ui-components as it is only #2 in the list
Will not remove cypress-nurocor-reporter as it is only #3 in the list
Will not remove property-validation-nodejs as it is only #4 in the list
Will not remove postgresql-client-nodejs as it is only #5 in the list
Will not remove oss-license-reporter-for-npm as it is only #6 in the list
Will not remove node-server-utils as it is only #7 in the list
Will not remove node-graphql-utils as it is only #8 in the list
Will not remove node-error-utils as it is only #9 in the list
Will not remove node-arango-utils as it is only #10 in the list
Will remove ncp-ui-module as it is #11 in the list
Will remove ncp-metadata-nodejs as it is #12 in the list
Will remove ncp-context-nodejs as it is #13 in the list
Will remove mdr-config-tool as it is #14 in the list
Will remove log-client-nodejs as it is #15 in the list
Will remove log-client-java as it is #16 in the list
Will remove graphql-voyager as it is #17 in the list
Will remove eslit-config as it is #18 in the list
Will remove amqp-utils-nodejs as it is #19 in the list
Will remove amqp-mocks as it is #20 in the list
Will remove als-compare-tool as it is #21 in the list
Will remove temp_log-client-java_consumer as it is #22 in the list

Anything else?

No response

robross0606 commented 2 years ago

Turns out this was an issue with filtering by project name accidentally instead of by branch name.