Export CVE is not using CVE ID filter

[jorisjumanne]

Expected behavior and actual behavior: Expexted: when using a CVE ID, e.g. CVE-2022-42889, while exporting a CVE list, I expect only projects to be exported which have the CVE I used for filtering. I used the following flavors of CVE ID: CVE-2022-42889, 2022-42889,42889, *2022-42889, *42889 and **42889

See also this screenshot:

Actual result: an export file is generated with size 0 bytes.

Steps to reproduce the problem:

• Log in as a user with enough permissions.
• Go to project
• Select a project
• From the dropdown select "Action" --> "Export CVEs"
• Fill in a CVE using any of the formats as described above

Versions:

• harbor version: 2.6.1

[JayKus]

Hi Daojun,

Thank you for the quick response! For some reason I don't see it in the ticket though.

As far as I know we started out with the accurate CVE ID, it's why we raised the ticket :)

Hoping to hear from you,

Jimmy

On Mon, Oct 24, 2022 at 10:18 AM stonezdj(Daojun Zhang) < @.***> wrote:

Current the CVE ID should be the accurate CVE ID, doesn't support wildcard.

— Reply to this email directly, view it on GitHub https://github.com/goharbor/harbor/issues/17678#issuecomment-1288612992, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALJY5PBT3PUGTZ3B6DV3MJLWEZA7DANCNFSM6AAAAAARLD4WAA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[AllForNothing]

@jorisjumanne For the CVE filter, you can only input accurate CVE IDs or IDs, regex, fuzzy match, and double star rules are not supported.

And please check if the selected project has the CVE you input in the CVE filter. If not, you are expected to get an empty file

[jorisjumanne]

Dear @AllForNothing ,

• The CVE is present in one of the repositories under the project I selected
• You can see in the original issue how I submitted the CVEs. The first thing I tried was the entire CVE, including CVE-. That didn't work and then I tried some other possibilities.

Hope it helps.

Joris

[AllForNothing]

@jorisjumanne

Can you get the right CVE file with the CVE ID filter empty?

[AllForNothing]

@jorisjumanne And you can check the log in /var/log/harbor/jobservice.log to see if there are any errors

[JayKus]

@jorisjumanne Can you get the right CVE file with the CVE ID filter empty?

Yes, without any filter we get a file that's tens of MBs in size, the CVE in question is in there.

We checked the logging.

Using a regular account without any additional access (Role on project says -) results in the "Export CVEs" button being inaccessible.

Using the same regular account with a Developer role on a project, I tag that project, start the export with the filter, and the logging generates the following:

2022-10-27T07:38:58Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"SCAN_DATA_EXPORT","id":"17e0000d841836f21347f6b8","t":1666856338,"args":null}
2022-10-27T07:38:58Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T07:38:58Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T07:38:58Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:78]: Scan data export job started in mode : export
2022-10-27T07:38:58Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:161]: Created CSV export file /var/scandata_exports/scandata_export_216.csv
2022-10-27T07:38:58Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:167]: Request for export : map[CVEIds:CVE-2022-42889 JobName: Labels:[] Projects:[2] Repositories: Tags: UserID:30 UserName:bla-user]
2022-10-27T07:38:59Z [INFO] [/pkg/scan/export/filter_processor.go:53]: Retrieved user id :30 for user name : bla-user
2022-10-27T07:38:59Z [INFO] [/pkg/scan/export/filter_processor.go:234]: User bla-user is not sys admin. Selecting projects with admin roles for export.
2022-10-27T07:38:59Z [INFO] [/pkg/scan/export/filter_processor.go:65]: Selected 0 projects administered by user bla-user 
2022-10-27T07:38:59Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:95]: Export Job Id = 216, FileName = /var/scandata_exports/scandata_export_216.csv, Hash = sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-10-27T07:38:59Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:105]: Creating repository for CSV file with blob : scandata_export_216
2022-10-27T07:38:59Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:111]: Export Job Id = 216. CSV file size: 0
2022-10-27T07:38:59Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T07:38:59Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:120]: Export Job Id = 216. Created system artifact: 12 for report file /var/scandata_exports/scandata_export_216.csv to persistent storage: <nil>
2022-10-27T07:38:59Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:127]: Scan data export job completed
2022-10-27T07:38:59Z [INFO] [/jobservice/runner/redis.go:152]: Job 'SCAN_DATA_EXPORT:17e0000d841836f21347f6b8' exit with success

A zerobyte file is generated. Log entry User bla is not sys admin. Selecting projects with admin roles for export. and Selected 0 projects administered does not seem correct behaviour?

Anyway, we then gave that same regular account the Project Admin role on the same project, with the same endresult, but at least there was 1 project to select. :)

Logging:

2022-10-27T08:28:14Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"SCAN_DATA_EXPORT","id":"1b5ea471b76b9de3673e7fbb","t":1666859292,"args":null}
2022-10-27T08:28:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T08:28:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T08:28:14Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:78]: Scan data export job started in mode : export
2022-10-27T08:28:14Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:161]: Created CSV export file /var/scandata_exports/scandata_export_222.csv
2022-10-27T08:28:14Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:167]: Request for export : map[CVEIds:CVE-2022-42889 JobName: Labels:[] Projects:[2] Repositories: Tags: UserID:30 UserName:bla-user]
2022-10-27T08:28:14Z [INFO] [/pkg/scan/export/filter_processor.go:53]: Retrieved user id :30 for user name : bla-user
2022-10-27T08:28:14Z [INFO] [/pkg/scan/export/filter_processor.go:234]: User bla-user is not sys admin. Selecting projects with admin roles for export.
2022-10-27T08:28:14Z [INFO] [/pkg/scan/export/filter_processor.go:65]: Selected 1 projects administered by user bla-user 
2022-10-27T08:28:19Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:248]: No more data to fetch. Exiting...
2022-10-27T08:28:19Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:95]: Export Job Id = 222, FileName = /var/scandata_exports/scandata_export_222.csv, Hash = sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-10-27T08:28:19Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:105]: Creating repository for CSV file with blob : scandata_export_222
2022-10-27T08:28:19Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:111]: Export Job Id = 222. CSV file size: 0
2022-10-27T08:28:19Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T08:28:20Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:120]: Export Job Id = 222. Created system artifact: 18 for report file /var/scandata_exports/scandata_export_222.csv to persistent storage: <nil>
2022-10-27T08:28:20Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:127]: Scan data export job completed
2022-10-27T08:28:20Z [INFO] [/jobservice/runner/redis.go:152]: Job 'SCAN_DATA_EXPORT:1b5ea471b76b9de3673e7fbb' exit with success

As a last check I ran the export again on the same project, but now without the filter. Logging:

2022-10-27T08:29:09Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"SCAN_DATA_EXPORT","id":"0e46db5e0b7d81ca1f4dacef","t":1666859346,"args":null}
2022-10-27T08:29:09Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T08:29:09Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T08:29:09Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:78]: Scan data export job started in mode : export
2022-10-27T08:29:09Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:161]: Created CSV export file /var/scandata_exports/scandata_export_223.csv
2022-10-27T08:29:09Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:167]: Request for export : map[CVEIds: JobName: Labels:[] Projects:[2] Repositories: Tags: UserID:30 UserName:bla-user]
2022-10-27T08:29:09Z [INFO] [/pkg/scan/export/filter_processor.go:53]: Retrieved user id :30 for user name : bla-user
2022-10-27T08:29:09Z [INFO] [/pkg/scan/export/filter_processor.go:234]: User bla-user is not sys admin. Selecting projects with admin roles for export.
2022-10-27T08:29:09Z [INFO] [/pkg/scan/export/filter_processor.go:65]: Selected 1 projects administered by user bla-user 
2022-10-27T08:29:14Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:251]: Export Group Id = 0, Job Id = 223, Page Number = 1, Page Size = 100000 Num Records = 100000
2022-10-27T08:29:19Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:251]: Export Group Id = 0, Job Id = 223, Page Number = 2, Page Size = 100000 Num Records = 100000
2022-10-27T08:29:24Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:251]: Export Group Id = 0, Job Id = 223, Page Number = 3, Page Size = 100000 Num Records = 79951
2022-10-27T08:29:25Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:95]: Export Job Id = 223, FileName = /var/scandata_exports/scandata_export_223.csv, Hash = sha256:3982f0b7e80153a6b607105eabb4957fa658b0a0b6a6cc3112ba45a06136c1b3
2022-10-27T08:29:25Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:105]: Creating repository for CSV file with blob : scandata_export_223
2022-10-27T08:29:25Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:111]: Export Job Id = 223. CSV file size: 115196784
2022-10-27T08:29:25Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T08:29:28Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:120]: Export Job Id = 223. Created system artifact: 19 for report file /var/scandata_exports/scandata_export_223.csv to persistent storage: <nil>
2022-10-27T08:29:28Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:127]: Scan data export job completed
2022-10-27T08:29:28Z [INFO] [/jobservice/runner/redis.go:152]: Job 'SCAN_DATA_EXPORT:0e46db5e0b7d81ca1f4dacef' exit with success

Switching to an admin level account, the export with the filter generates the following logging:

2022-10-27T07:43:56Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"SCAN_DATA_EXPORT","id":"fb4b63898f98545748a61173","t":1666856635,"args":null}
2022-10-27T07:43:56Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T07:43:56Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T07:43:56Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:78]: Scan data export job started in mode : export
2022-10-27T07:43:56Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:161]: Created CSV export file /var/scandata_exports/scandata_export_221.csv
2022-10-27T07:43:56Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:167]: Request for export : map[CVEIds:CVE-2022-42889 JobName: Labels:[] Projects:[2] Repositories: Tags: UserID:1 UserName:admin]
2022-10-27T07:43:56Z [INFO] [/pkg/scan/export/filter_processor.go:53]: Retrieved user id :1 for user name : admin
2022-10-27T07:43:56Z [INFO] [/pkg/scan/export/filter_processor.go:230]: User admin is sys admin. Selecting all projects for export.
2022-10-27T07:43:56Z [INFO] [/pkg/scan/export/filter_processor.go:65]: Selected 5 projects administered by user admin 
2022-10-27T07:44:01Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:248]: No more data to fetch. Exiting...
2022-10-27T07:44:01Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:95]: Export Job Id = 221, FileName = /var/scandata_exports/scandata_export_221.csv, Hash = sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-10-27T07:44:01Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:105]: Creating repository for CSV file with blob : scandata_export_221
2022-10-27T07:44:01Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:111]: Export Job Id = 221. CSV file size: 0
2022-10-27T07:44:01Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-10-27T07:44:02Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:120]: Export Job Id = 221. Created system artifact: 17 for report file /var/scandata_exports/scandata_export_221.csv to persistent storage: <nil>
2022-10-27T07:44:02Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:127]: Scan data export job completed
2022-10-27T07:44:02Z [INFO] [/jobservice/runner/redis.go:152]: Job 'SCAN_DATA_EXPORT:fb4b63898f98545748a61173' exit with success

A zerobyte file is generated. Why does the logging say all 5 projects are selected if I only put the tag in 1 of them (it's being shown in the screenshot)?

Hoping to hear from you :)

[AllForNothing]

@chlins Can you have a look at this?

[AllForNothing]

@JayKus How did you tag a project? in Harbor, we can only tag an artifact

[JayKus]

@JayKus How did you tag a project? in Harbor, we can only tag an artifact

Sorry, with "tag" I meant "tick the box" 😄

[chlins]

@JayKus Hi, just want to confirm with your cases.

case 1: Account with developer role for the project with CVE filter got empty csv file. case 2: Account with project admin role for the project with CVE filter got empty csv file. case 3: Account with project admin role for the project without CVE filter got right csv file. case 4: Account with system admin role for the project without CVE filter got empty csv file.

[JayKus]

Hi @chlins sorry for the late reply!

Case 1 to 3 is correct, case 4 is incorrect.

Case 4 should be "System admin without any specific role for the project without CVE filter gives the right csv file". Case 5 would be "System admin without any specific role for the project with CVE filter gives an empty csv file".

[chlins]

@JayKus Hi, I've tried the same scenarios as you provided, but still can not reproduce the issue which you ran into.

[robinkb]

@chlins I am the one who deployed the Harbor installation that this ticket is about. Maybe we misconfigured something? Harbor is installed through Helm and running on AKS. Here are the sanitized values that we used:

cache:
  enabled: true
chartmuseum:
  enabled: false
core:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: harbor
            component: core
        topologyKey: topology.kubernetes.io/zone
  replicas: 2
  resources:
    limits:
      memory: 256Mi
    requests:
      cpu: 100m
      memory: 256Mi
  secret: <snip>
  xsrfKey: <snip>
database:
  external:
    coreDatabase: registry
    host: <snip>
    password: <snip>
    sslmode: verify-full
    username: harbor@<snip>
  type: external
existingSecretSecretKey: harbor-encryption-key
exporter:
  resources:
    limits:
      memory: 32Mi
    requests:
      cpu: 10m
      memory: 32Mi
expose:
  ingress:
    annotations:
      cert-manager.io/cluster-issuer: lets-encrypt-production
      cert-manager.io/private-key-algorithm: ECDSA
      external-dns.alpha.kubernetes.io/hostname: <snip>
      external-dns.alpha.kubernetes.io/ingress-hostname-source: annotation-only
    className: nginx-public
    hosts:
      core: <snip>
  tls:
    certSource: secret
    secret:
      secretName: <snip>
externalURL: https://<snip>
jobservice:
  jobLoggers:
  - database
  - stdout
  replicas: 1
  resources:
    limits:
      memory: 384Mi
    requests:
      cpu: 100m
      memory: 384Mi
  secret: <snip>
metrics:
  enabled: true
  serviceMonitor:
    enabled: true
notary:
  enabled: false
persistence:
  imageChartStorage:
    azure:
      accountkey: <snip>
      accountname: <snip>
      container: <snip>
      realm: core.windows.net
    type: azure
  persistentVolumeClaim:
    jobservice:
      scanDataExports:
        storageClass: standard-ssd-zrs
    trivy:
      storageClass: standard-ssd-zrs
portal:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: harbor
            component: portal
        topologyKey: topology.kubernetes.io/zone
  replicas: 2
  resources:
    limits:
      memory: 16Mi
    requests:
      cpu: 10m
      memory: 16Mi
redis:
  external:
    addr: <snip>:6379
    password: <snip>
  type: external
registry:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: harbor
            component: registry
        topologyKey: topology.kubernetes.io/zone
  controller:
    resources:
      limits:
        memory: 32Mi
      requests:
        cpu: 10m
        memory: 32Mi
  credentials:
    htpasswdString: harbor_registry_user:$2a$10$bulyvmcoqAOUoiU9bhDci.vWpRBbbPmzmWKvcM4Vh7ZtLKbMJfLHe
    password: <snip>
    username: harbor_registry_user
  registry:
    resources:
      limits:
        memory: 128Mi
      requests:
        cpu: 300m
        memory: 128Mi
  replicas: 2
  secret: <snip>
trivy:
  resources:
    limits:
      cpu: null
      memory: 1Gi
    requests:
      cpu: 1
      memory: 1Gi
updateStrategy:
  type: Recreate

The chart version is harbor-1.10.2. Please let us know if you require any more information.

[robinkb]

Another thing: Is there more logging that we can provide? Maybe the filtering mechanism is breaking on data that is present in our specific installation? Or maybe the size is the problem? We have quite a few images present in Harbor.

[robinkb]

Hi @chlins , is there anything more that we can do to help get this issue resolved?

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

[chlins]

@robinkb Which account did you use when export cve and did you use OIDC or LDAP for harbor authorization?

[robinkb]

Hi @chlins, I tried it again today with the Admin account on Harbor version 2.7.1. I copied a CVE from one of our scanning reports, and ran the export function with that CVE as input. It returned zero results:

[jorisjumanne]

Hi @chlins . We have version v2.11.0-70255684 deployed at the moment, but we are still unable to export a filtered list of projects/artifacts where a CVE was found. I turned to the swagger UI this time, logged in as an "admin" user.

Request which is working, returning a CSV with all vulnerabilities in all artifacts:

{ "job_name": "string", "projects": [2], "labels": [], "repositories": null, "cveIds": null, "tags": null }

Request which is not working, because it doesn't return a file at all:

{ "job_name": "string", "projects": [2], "labels": [], "repositories": "**", "cveIds": "CVE-2024-6387", "tags": "**" }

[chlins]

Hi @chlins . We have version v2.11.0-70255684 deployed at the moment, but we are still unable to export a filtered list of projects/artifacts where a CVE was found. I turned to the swagger UI this time, logged in as an "admin" user.

Request which is working, returning a CSV with all vulnerabilities in all artifacts:

{ "job_name": "string", "projects": [2], "labels": [], "repositories": null, "cveIds": null, "tags": null }

Request which is not working, because it doesn't return a file at all:

{ "job_name": "string", "projects": [2], "labels": [], "repositories": "**", "cveIds": "CVE-2024-6387", "tags": "**" }

@jorisjumanne Hi, could you see the CVE-2024-6387 in the CSV when you not set any filter?

[jorisjumanne]

Hi Chlins,

Yes, I can confirm the CVE is there in multiple projects/artifacts.

Joris

On Wed, Jul 3, 2024 at 5:01 AM Chlins Zhang @.***> wrote:

Hi @chlins https://github.com/chlins . We have version v2.11.0-70255684 deployed at the moment, but we are still unable to export a filtered list of projects/artifacts where a CVE was found. I turned to the swagger UI this time, logged in as an "admin" user.

Request which is working, returning a CSV with all vulnerabilities in all artifacts:

{ "job_name": "string", "projects": [2], "labels": [], "repositories": null, "cveIds": null, "tags": null }

Request which is not working, because it doesn't return a file at all:

{ "job_name": "string", "projects": [2], "labels": [], "repositories": "", "cveIds": "CVE-2024-6387", "tags": "" }

@jorisjumanne https://github.com/jorisjumanne Hi, could you see the CVE-2024-6387 in the CSV when you not set any filter?

— Reply to this email directly, view it on GitHub https://github.com/goharbor/harbor/issues/17678#issuecomment-2204975055, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEY47WACHFQQDLK4XEXEIN3ZKNSRDAVCNFSM6AAAAABKHO6HT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBUHE3TKMBVGU . You are receiving this because you were mentioned.Message ID: @.***>