search

whitesource-ps / ws-nexus-integration

WhiteSource Nexus integration tool
Apache License 2.0
15 stars 8 forks source link

[BUG] [ws_nexus_integration] byte indices must be integers or slices, not str #31

Closed danielnbalasoiu closed 2 years ago

danielnbalasoiu commented 2 years ago

Bug Description
Running ws_nexus_integration param.config w/ the following params.config config:

[Nexus Settings]
NexusBaseUrl=http://nexus.host:8080
NexusAuthToken=
NexusUser=REDACTED
NexusPassword=REDACTED
NexusRepositories=docker-internal
NexusAltDockerRegistryAddress=

[WhiteSource Settings]
WSUserKey=REDACTED
WSApiKey=REDACTED
WSProductName=Nexus
WSCheckPolicies=False
WSUrl=https://saas-eu.whitesourcesoftware.com
WSLang=

[General Settings]
ThreadCount=1
WorkDir=
JavaBin=

fails w/ 

root@4960da47faaa:~# ws_nexus_integration param.config
INFO 22-01-17 15:46:56 281472833163280 root: Nexus Version: Nexus/3.XXX (OSS)
INFO 22-01-17 15:46:56 281472833163280 root: Validate specified repositories
INFO 22-01-17 15:46:56 281472833163280 root: Getting region parameters has finished
INFO 22-01-17 15:46:56 281472833163280 root: Repository: docker-internal
INFO 22-01-17 15:46:56 281472833163280 root: Validate artifact list
Traceback (most recent call last):
  File "/usr/local/bin/ws_nexus_integration", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/ws_nexus_integration/nexus_integration.py", line 461, in main
    download_components_from_repositories(selected_repositories)
  File "/usr/local/lib/python3.8/site-packages/ws_nexus_integration/nexus_integration.py", line 221, in download_components_from_repositories
    for item in cur_comp_response['items']:
TypeError: byte indices must be integers or slices, not str

Steps to Reproduce
Steps to reproduce the behavior:

  1. Install ws-nexus-integration: pip install ws-nexus-integration[DOCKER]
  2. Update params.config file as shown above.
  3. Run the scan: ws_nexus_integration param.config
  4. See error above.

Environment Details

tamari-oz commented 2 years ago

Something odd is returning in from: _/service/rest/v1/components?repository=_ endpoint.

danielnbalasoiu commented 2 years ago 
root@4960da47faaa:~# DEBUG=1 ws_nexus_integration param.config
DEBUG 22-01-18 07:40:29 281473656692752 root: Using configuration file: 'param.config'
DEBUG 22-01-18 07:40:29 281473656692752 root: Converting user and password to basic string
DEBUG 22-01-18 07:40:29 281473656692752 ws_sdk.ws_utilities: Executing command: ['java', '-version']
DEBUG 22-01-18 07:40:29 281473656692752 ws_sdk.ws_utilities: Java version: '11.0.13'
DEBUG 22-01-18 07:40:29 281473656692752 ws_sdk.ws_utilities: Executing command: ['java', '-Djava.io.tmpdir=/tmp/ws-nexus-integration', '-jar', '/tmp/ws-nexus-integration/wss-unified-agent.jar', '-v', '-noConfig', 'True']
DEBUG 22-01-18 07:40:31 281473656692752 ws_sdk.client: UA executed successfully. Return Code 0. Message: 21.12.2

DEBUG 22-01-18 07:40:31 281473656692752 ws_sdk.client: Local WhiteSource Unified Agent version 21.12.2
DEBUG 22-01-18 07:40:31 281473656692752 ws_sdk.ws_utilities: Calling url: https://api.github.com/repos/whitesource/unified-agent-distribution/releases/latest
DEBUG 22-01-18 07:40:31 281473656692752 ws_sdk.ws_utilities: Latest Unified Agent version: v21.12.2
DEBUG 22-01-18 07:40:31 281473656692752 ws_sdk.client: Skipping WhiteSource Unified Agent update
DEBUG 22-01-18 07:40:31 281473656692752 root: Sending request for retrieving Nexus repository list
DEBUG 22-01-18 07:40:31 281473656692752 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/repositories
INFO 22-01-18 07:40:31 281473656692752 root: Nexus Version: Nexus/3.37.3-02 (OSS)
INFO 22-01-18 07:40:31 281473656692752 root: Validate specified repositories
INFO 22-01-18 07:40:31 281473656692752 root: Getting region parameters has finished
DEBUG 22-01-18 07:40:31 281473656692752 root: Using repository: /service/rest/v1/repositorySettings
INFO 22-01-18 07:40:31 281473656692752 root: Repository: docker-internal
INFO 22-01-18 07:40:31 281473656692752 root: Validate artifact list
DEBUG 22-01-18 07:40:31 281473656692752 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal
DEBUG 22-01-18 07:40:32 281473656692752 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=REDACTED
DEBUG 22-01-18 07:40:32 281473656692752 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=REDACTED
Traceback (most recent call last):
  File "/usr/local/bin/ws_nexus_integration", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/ws_nexus_integration/nexus_integration.py", line 461, in main
    download_components_from_repositories(selected_repositories)
  File "/usr/local/lib/python3.8/site-packages/ws_nexus_integration/nexus_integration.py", line 221, in download_components_from_repositories
    for item in cur_comp_response['items']:
TypeError: byte indices must be integers or slices, not str

tamari-oz commented 2 years ago

I think that there is a discrepancy between the actual repositories and the ones configured to be scanned (in params.conf: NexusRepositories=).

I improved the mechanism a bit with additional prints. Can you test pre release 0.2.10a1 (pip install ws-nexus-integration==0.2.10a1) and upload the output (DEBUG=1)?

Thanks

danielnbalasoiu commented 2 years ago

params.config:

root@4960da47faaa:~# grep NexusRepositories param.config
NexusRepositories=docker-internal

Note: Some Nexus repositories were removed but the relevant one (docker-internal) was kept and a few more for the sake of debugging.

root@4960da47faaa:~# DEBUG=1 ws_nexus_integration param.config
DEBUG 22-01-18 11:15:11 281472821805072 root: Using configuration file: 'param.config'
DEBUG 22-01-18 11:15:11 281472821805072 root: Converting user and password to basic string
DEBUG 22-01-18 11:15:11 281472821805072 ws_sdk.ws_utilities: Executing command: ['java', '-version']
DEBUG 22-01-18 11:15:11 281472821805072 ws_sdk.ws_utilities: Java version: '11.0.13'
DEBUG 22-01-18 11:15:11 281472821805072 ws_sdk.ws_utilities: Executing command: ['java', '-Djava.io.tmpdir=/tmp/ws-nexus-integration', '-jar', '/tmp/ws-nexus-integration/wss-unified-agent.jar', '-v', '-noConfig', 'True']
DEBUG 22-01-18 11:15:12 281472821805072 ws_sdk.client: UA executed successfully. Return Code 0. Message: 21.12.2

DEBUG 22-01-18 11:15:12 281472821805072 ws_sdk.client: Local WhiteSource Unified Agent version 21.12.2
DEBUG 22-01-18 11:15:12 281472821805072 ws_sdk.ws_utilities: Calling url: https://api.github.com/repos/whitesource/unified-agent-distribution/releases/latest
DEBUG 22-01-18 11:15:12 281472821805072 ws_sdk.ws_utilities: Latest Unified Agent version: v21.12.2
DEBUG 22-01-18 11:15:12 281472821805072 ws_sdk.client: Skipping WhiteSource Unified Agent update
DEBUG 22-01-18 11:15:12 281472821805072 root: Sending request for retrieving Nexus repository list
DEBUG 22-01-18 11:15:12 281472821805072 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/repositories
INFO 22-01-18 11:15:13 281472821805072 root: Nexus Version: Nexus/3.37.3-02 (OSS)
DEBUG 22-01-18 11:15:13 281472821805072 root: The following repositories were found: ['docker-internal', 'maven-central', 'npm-registry']
DEBUG 22-01-18 11:15:13 281472821805072 root: Repository: 'docker-internal' was added to scan
DEBUG 22-01-18 11:15:13 281472821805072 root: Using repository: /service/rest/v1/repositorySettings
INFO 22-01-18 11:15:13 281472821805072 root: Repository: docker-internal
INFO 22-01-18 11:15:13 281472821805072 root: Validate artifact list
DEBUG 22-01-18 11:15:13 281472821805072 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal
DEBUG 22-01-18 11:15:14 281472821805072 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=REDACTED
DEBUG 22-01-18 11:15:14 281472821805072 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=REDACTED
Traceback (most recent call last):
  File "/usr/local/bin/ws_nexus_integration", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/ws_nexus_integration/nexus_integration.py", line 447, in main
    download_components_from_repositories(selected_repositories)
  File "/usr/local/lib/python3.8/site-packages/ws_nexus_integration/nexus_integration.py", line 221, in download_components_from_repositories
    for item in cur_comp_response['items']:
TypeError: byte indices must be integers or slices, not str
tamari-oz commented 2 years ago

OK I think it might be due to the continuationToken (the end URL is empty). The code now will bypass that.

Can you test please test ws-nexus-integration==0.2.10a2?

danielnbalasoiu commented 2 years ago

The continuationToken was not empty. It had a string which I replaced with REDACTED.

root@4960da47faaa:~# DEBUG=1 ws_nexus_integration param.config
DEBUG 22-01-18 14:35:10 281473747169296 root: Using configuration file: 'param.config'
DEBUG 22-01-18 14:35:10 281473747169296 root: Converting user and password to basic string
DEBUG 22-01-18 14:35:10 281473747169296 ws_sdk.ws_utilities: Executing command: ['java', '-version']
DEBUG 22-01-18 14:35:10 281473747169296 ws_sdk.ws_utilities: Java version: '11.0.13'
DEBUG 22-01-18 14:35:10 281473747169296 ws_sdk.ws_utilities: Executing command: ['java', '-Djava.io.tmpdir=/tmp/ws-nexus-integration', '-jar', '/tmp/ws-nexus-integration/wss-unified-agent.jar', '-v', '-noConfig', 'True']
DEBUG 22-01-18 14:35:11 281473747169296 ws_sdk.client: UA executed successfully. Return Code 0. Message: 21.12.2

DEBUG 22-01-18 14:35:11 281473747169296 ws_sdk.client: Local WhiteSource Unified Agent version 21.12.2
DEBUG 22-01-18 14:35:11 281473747169296 ws_sdk.ws_utilities: Calling url: https://api.github.com/repos/whitesource/unified-agent-distribution/releases/latest
DEBUG 22-01-18 14:35:12 281473747169296 ws_sdk.ws_utilities: Latest Unified Agent version: v21.12.2
DEBUG 22-01-18 14:35:12 281473747169296 ws_sdk.client: Skipping WhiteSource Unified Agent update
DEBUG 22-01-18 14:35:12 281473747169296 root: Sending request for retrieving Nexus repository list
DEBUG 22-01-18 14:35:12 281473747169296 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/repositories
INFO 22-01-18 14:35:12 281473747169296 root: Nexus Version: Nexus/3.37.3-02 (OSS)
DEBUG 22-01-18 14:35:12 281473747169296 root: The following repositories were found: ['docker-internal', 'maven-central', 'npm-registry']
DEBUG 22-01-18 14:35:12 281473747169296 root: Repository: 'docker-internal' was added to scan
DEBUG 22-01-18 14:35:12 281473747169296 root: Using repository: /service/rest/v1/repositorySettings
INFO 22-01-18 14:35:12 281473747169296 root: Repository: docker-internal
INFO 22-01-18 14:35:12 281473747169296 root: Validate artifact list
DEBUG 22-01-18 14:35:12 281473747169296 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal
DEBUG 22-01-18 14:35:13 281473747169296 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b37REDACTED
DEBUG 22-01-18 14:35:13 281473747169296 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b37REDACTED
Traceback (most recent call last):
  File "/usr/local/bin/ws_nexus_integration", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/ws_nexus_integration/nexus_integration.py", line 451, in main
    download_components_from_repositories(selected_repositories)
  File "/usr/local/lib/python3.8/site-packages/ws_nexus_integration/nexus_integration.py", line 221, in download_components_from_repositories
    for item in cur_comp_response.get('items'):
AttributeError: 'bytes' object has no attribute 'get'
root@4960da47faaa:~# pip show ws_nexus_integration
Name: ws-nexus-integration
Version: 0.2.10a2
Summary: WS Nexus Integration
Home-page: https://github.com/whitesource-ps/ws-nexus-integration
Author: WhiteSource Professional Services
Author-email: ps@whitesourcesoftware.com
License: LICENSE
Location: /usr/local/lib/python3.8/site-packages
Requires: requests, wget, ws-sdk
Required-by:
danielnbalasoiu commented 2 years ago

Hi,

I checked version 0.2.10:

Traceback (most recent call last):
  File "/usr/bin/ws_nexus_integration", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 451, in main
    download_components_from_repositories(selected_repositories)
  File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 221, in download_components_from_repositories
    for item in cur_comp_response.get('items'):
AttributeError: 'bytes' object has no attribute 'get'

and v0.2.11 which fails from the start

/opt/whitesource # DEBUG=1 ws_nexus_integration
DEBUG 22-01-25 08:39:11 281473604283464 root: Using configuration file: 'params.config'
DEBUG 22-01-25 08:39:11 281473604283464 root: Converting user and password to basic string
Traceback (most recent call last):
  File "/usr/bin/ws_nexus_integration", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 450, in main
    config = Configuration()
  File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 146, in __init__
    self.ws_conn = WS(user_key=conf['WhiteSource Settings']['WSUserKey'],
  File "/usr/lib/python3.9/site-packages/ws_sdk/web.py", line 12, in __init__
    self.ws_app = WSApp.__init__(self, **kwargs)
  File "/usr/lib/python3.9/site-packages/ws_sdk/app.py", line 120, in __init__
    raise WsSdkTokenError(self.user_key)
ws_sdk.ws_errors.WsSdkTokenError: Invalid token: ''
tamari-oz commented 2 years ago

Seems that the token is missing (in 'params.conf' under ['WhiteSource Settings'] -> 'WSUserKey').

danielnbalasoiu commented 2 years ago

Indeed, sorry about that. I missed that config parameter.

I'm having the same error with the latest version:

/opt/whitesource # pip list | grep ws
ws-nexus-integration 0.2.11
ws-sdk               0.9a3
Traceback (most recent call last):
  File "/usr/bin/ws_nexus_integration", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 453, in main
    download_components_from_repositories(selected_repositories)
  File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 221, in download_components_from_repositories
    for item in cur_comp_response.get('items'):
AttributeError: 'bytes' object has no attribute 'get'
tamari-oz commented 2 years ago

Thanks.

I create 0.2.12a1. Can you please test? And upload the debug output, if it fails

danielnbalasoiu commented 2 years ago 
/opt/whitesource # pip list | grep ws
ws-nexus-integration 0.2.12a1
ws-sdk               0.9.0.2
Debug log ```shell /opt/whitesource # DEBUG=1 ws_nexus_integration DEBUG 22-01-27 15:26:08 281473758039112 root: Using configuration file: 'params.config' DEBUG 22-01-27 15:26:08 281473758039112 root: Converting user and password to basic string DEBUG 22-01-27 15:26:08 281473758039112 ws_sdk.ws_utilities: Executing command: ['java', '-version'] DEBUG 22-01-27 15:26:08 281473758039112 ws_sdk.ws_utilities: Java version: '1.8.0_302' DEBUG 22-01-27 15:26:08 281473758039112 ws_sdk.ws_utilities: Executing command: ['java', '-Djava.io.tmpdir=/tmp/ws-nexus-integration', '-jar', '/tmp/ws-nexus-integration/wss-unified-agent.jar', '-v', '-noConfig', 'True '] DEBUG 22-01-27 15:26:10 281473758039112 ws_sdk.client: UA executed successfully. Return Code 0. Message: 22.1.1 DEBUG 22-01-27 15:26:10 281473758039112 ws_sdk.client: Local WhiteSource Unified Agent version 22.1.1 DEBUG 22-01-27 15:26:10 281473758039112 ws_sdk.ws_utilities: Calling url: https://api.github.com/repos/whitesource/unified-agent-distribution/releases/latest DEBUG 22-01-27 15:26:10 281473758039112 ws_sdk.ws_utilities: Latest Unified Agent version: v22.1.1 DEBUG 22-01-27 15:26:10 281473758039112 ws_sdk.client: Skipping WhiteSource Unified Agent update DEBUG 22-01-27 15:26:10 281473758039112 root: Sending request for retrieving Nexus repository list DEBUG 22-01-27 15:26:10 281473758039112 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/repositories INFO 22-01-27 15:26:11 281473758039112 root: Nexus Version: Nexus/3.37.3-02 (OSS) DEBUG 22-01-27 15:26:11 281473758039112 root: The following repositories were found: ['docker-internal', 'maven-central', 'npm-registry'] DEBUG 22-01-27 15:26:11 281473758039112 root: Repository: 'docker-internal' was added to scan DEBUG 22-01-27 15:26:11 281473758039112 root: Using repository: /service/rest/v1/repositorySettings INFO 22-01-27 15:26:11 281473758039112 root: Repository: docker-internal INFO 22-01-27 15:26:11 281473758039112 root: Validate artifact list DEBUG 22-01-27 15:26:11 281473758039112 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal DEBUG 22-01-27 15:26:11 281473758039112 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c6722b1b075c0426200c3 DEBUG 22-01-27 15:26:12 281473758039112 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 Traceback (most recent call last): File "/usr/bin/ws_nexus_integration", line 8, in sys.exit(main()) File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 454, in main download_components_from_repositories(selected_repositories) File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 222, in download_components_from_repositories for item in cur_comp_response.get('items'): AttributeError: 'bytes' object has no attribute 'get' ```
tamari-oz commented 2 years ago

OK so something returns from the call :) Can you test 0.2.12a2 and upload the debug again? Thanks

danielnbalasoiu commented 2 years ago 
/opt/whitesource # pip list | grep ws
ws-nexus-integration 0.2.12a2
ws-sdk               0.9.0.2
Debug log ```shell /opt/whitesource # DEBUG=1 ws_nexus_integration DEBUG 22-01-27 17:50:30 281473486548040 root: Using configuration file: 'params.config' DEBUG 22-01-27 17:50:30 281473486548040 root: Converting user and password to basic string DEBUG 22-01-27 17:50:30 281473486548040 ws_sdk.ws_utilities: Executing command: ['java', '-version'] DEBUG 22-01-27 17:50:30 281473486548040 ws_sdk.ws_utilities: Java version: '1.8.0_302' DEBUG 22-01-27 17:50:30 281473486548040 ws_sdk.ws_utilities: Executing command: ['java', '-Djava.io.tmpdir=/tmp/ws-nexus-integration', '-jar', '/tmp/ws-nexus-integration/wss-unified-agent.jar', '-v', '-noConfig', 'True '] DEBUG 22-01-27 17:50:32 281473486548040 ws_sdk.client: UA executed successfully. Return Code 0. Message: 22.1.1 DEBUG 22-01-27 17:50:32 281473486548040 ws_sdk.client: Local WhiteSource Unified Agent version 22.1.1 DEBUG 22-01-27 17:50:32 281473486548040 ws_sdk.ws_utilities: Calling url: https://api.github.com/repos/whitesource/unified-agent-distribution/releases/latest DEBUG 22-01-27 17:50:32 281473486548040 ws_sdk.ws_utilities: Latest Unified Agent version: v22.1.1.1 INFO 22-01-27 17:50:32 281473486548040 ws_sdk.client: A new WhiteSource Unified Agent exists. Downloading the latest DEBUG 22-01-27 17:50:32 281473486548040 ws_sdk.ws_utilities: Backing up previous wss-unified-agent.jar DEBUG 22-01-27 17:50:32 281473486548040 ws_sdk.ws_utilities: Calling url: https://api.github.com/repos/whitesource/unified-agent-distribution/releases/latest DEBUG 22-01-27 17:50:32 281473486548040 ws_sdk.ws_utilities: Latest Unified Agent version: v22.1.1.1 DEBUG 22-01-27 17:50:32 281473486548040 ws_sdk.ws_utilities: Downloading WS Unified Agent (version: v22.1.1.1) to /tmp/ws-nexus-integration/wss-unified-agent.jar DEBUG 22-01-27 17:50:50 281473486548040 root: Sending request for retrieving Nexus repository list DEBUG 22-01-27 17:50:50 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/repositories DEBUG 22-01-27 17:50:51 281473486548040 root: Response return type: INFO 22-01-27 17:50:51 281473486548040 root: Nexus Version: Nexus/3.37.3-02 (OSS) DEBUG 22-01-27 17:50:51 281473486548040 root: The following repositories were found: ['docker-internal', 'maven-central', 'npm-registry'] DEBUG 22-01-27 17:50:51 281473486548040 root: Repository: 'docker-internal' was added to scan DEBUG 22-01-27 17:50:51 281473486548040 root: Using repository: /service/rest/v1/repositorySettings INFO 22-01-27 17:50:51 281473486548040 root: Repository: docker-internal INFO 22-01-27 17:50:51 281473486548040 root: Validate artifact list DEBUG 22-01-27 17:50:51 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal DEBUG 22-01-27 17:50:51 281473486548040 root: Response return type: DEBUG 22-01-27 17:50:51 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c6722b1b075c0426200c3 DEBUG 22-01-27 17:50:52 281473486548040 root: Response return type: DEBUG 22-01-27 17:50:52 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 17:52:50 281473486548040 root: Response is not JSON DEBUG 22-01-27 17:52:50 281473486548040 root: Response return type: DEBUG 22-01-27 17:52:50 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 17:54:39 281473486548040 root: Response is not JSON DEBUG 22-01-27 17:54:39 281473486548040 root: Response return type: DEBUG 22-01-27 17:54:39 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 17:56:36 281473486548040 root: Response is not JSON DEBUG 22-01-27 17:56:36 281473486548040 root: Response return type: DEBUG 22-01-27 17:56:36 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 17:59:30 281473486548040 root: Response is not JSON DEBUG 22-01-27 17:59:30 281473486548040 root: Response return type: DEBUG 22-01-27 17:59:30 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:01:43 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:01:43 281473486548040 root: Response return type: DEBUG 22-01-27 18:01:43 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:03:36 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:03:36 281473486548040 root: Response return type: DEBUG 22-01-27 18:03:36 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:05:29 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:05:29 281473486548040 root: Response return type: DEBUG 22-01-27 18:05:29 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:08:10 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:08:10 281473486548040 root: Response return type: DEBUG 22-01-27 18:08:10 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:10:53 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:10:53 281473486548040 root: Response return type: DEBUG 22-01-27 18:10:53 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:13:14 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:13:14 281473486548040 root: Response return type: DEBUG 22-01-27 18:13:14 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:15:31 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:15:31 281473486548040 root: Response return type: DEBUG 22-01-27 18:15:31 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:17:35 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:17:35 281473486548040 root: Response return type: DEBUG 22-01-27 18:17:35 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:20:06 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:20:06 281473486548040 root: Response return type: DEBUG 22-01-27 18:20:06 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:22:10 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:22:10 281473486548040 root: Response return type: DEBUG 22-01-27 18:22:10 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:23:59 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:23:59 281473486548040 root: Response return type: DEBUG 22-01-27 18:23:59 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:25:49 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:25:49 281473486548040 root: Response return type: DEBUG 22-01-27 18:25:49 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:27:41 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:27:41 281473486548040 root: Response return type: DEBUG 22-01-27 18:27:41 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:30:07 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:30:07 281473486548040 root: Response return type: DEBUG 22-01-27 18:30:07 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:33:37 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:33:37 281473486548040 root: Response return type: DEBUG 22-01-27 18:33:37 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:35:29 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:35:29 281473486548040 root: Response return type: DEBUG 22-01-27 18:35:29 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:37:26 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:37:26 281473486548040 root: Response return type: DEBUG 22-01-27 18:37:26 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:39:22 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:39:22 281473486548040 root: Response return type: DEBUG 22-01-27 18:39:22 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:42:13 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:42:13 281473486548040 root: Response return type: DEBUG 22-01-27 18:42:13 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:44:42 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:44:42 281473486548040 root: Response return type: DEBUG 22-01-27 18:44:42 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:47:00 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:47:00 281473486548040 root: Response return type: DEBUG 22-01-27 18:47:00 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:48:50 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:48:50 281473486548040 root: Response return type: DEBUG 22-01-27 18:48:50 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:50:38 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:50:38 281473486548040 root: Response return type: DEBUG 22-01-27 18:50:38 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:53:34 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:53:34 281473486548040 root: Response return type: DEBUG 22-01-27 18:53:34 281473486548040 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=docker-internal&continuationToken=4b378653591c67227180318a3f096283 DEBUG 22-01-27 18:56:04 281473486548040 root: Response is not JSON DEBUG 22-01-27 18:56:04 281473486548040 root: Response return type: ``` And it goes like this on and on... For container image found in the registry I guess.
tamari-oz commented 2 years ago

Thanks for the update. It seems that the continuationToken (4b378653591c67227180318a3f096283) is not replaced between calls. So the 1st and 2nd calls are mapped with the same token which generate the same URL.

AFAIK the max number of artifacts in a call is 50.

I would like to replicate similar env before providing a fix for this.

danielnbalasoiu commented 2 years ago

There are a loooot of docker images in this repository (> 10k).

image

I'll check with another one which has fewer images and get back to you.

tamari-oz commented 2 years ago

Thanks. I'll make sure the code can handle the continuationToken.

tamari-oz commented 2 years ago

I just created v0.2.12a3. In this release I refactored the continuationToken mechanism. There was also a bug that caused it to endlessly call the same token.

Can you please re-test and send me the logs? Thanks

danielnbalasoiu commented 2 years ago

Yes. I'll test the new version (v0.2.12a3). I was about to paste the output from the other run but nothing changed.

danielnbalasoiu commented 2 years ago 
/opt/whitesource # pip list | grep ws
ws-nexus-integration 0.2.12a3
ws-sdk               0.9.1a1
/opt/whitesource # DEBUG=1 ws_nexus_integration
DEBUG 22-01-31 15:53:00 281473411271752 root: Using configuration file: 'params.config'
DEBUG 22-01-31 15:53:00 281473411271752 root: Converting user and password to basic string
DEBUG 22-01-31 15:53:00 281473411271752 ws_sdk.ws_utilities: Executing command: ['java', '-version']
DEBUG 22-01-31 15:53:00 281473411271752 ws_sdk.ws_utilities: Java version: '1.8.0_302'
DEBUG 22-01-31 15:53:00 281473411271752 ws_sdk.ws_utilities: Executing command: ['java', '-Djava.io.tmpdir=/tmp/ws-nexus-integration', '-jar', '/tmp/ws-nexus-integration/wss-unified-agent.jar', '-v', '-noConfig', 'True']
DEBUG 22-01-31 15:53:01 281473411271752 ws_sdk.client: UA executed successfully. Return Code 0. Message: 22.1.1.1

DEBUG 22-01-31 15:53:01 281473411271752 ws_sdk.client: Local WhiteSource Unified Agent version 22.1.1.1
DEBUG 22-01-31 15:53:01 281473411271752 ws_sdk.ws_utilities: Calling url: https://api.github.com/repos/whitesource/unified-agent-distribution/releases/latest
DEBUG 22-01-31 15:53:04 281473411271752 ws_sdk.ws_utilities: Latest Unified Agent version: v22.1.1.1
DEBUG 22-01-31 15:53:04 281473411271752 ws_sdk.client: Skipping WhiteSource Unified Agent update
DEBUG 22-01-31 15:53:04 281473411271752 root: Sending request for retrieving Nexus repository list
DEBUG 22-01-31 15:53:04 281473411271752 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/repositories
DEBUG 22-01-31 15:53:05 281473411271752 root: Response return type: <class 'list'>
INFO 22-01-31 15:53:05 281473411271752 root: Nexus Version: Nexus/3.37.3-02 (OSS)
DEBUG 22-01-31 15:53:05 281473411271752 root: The following repositories were found: ['ax-docker']
DEBUG 22-01-31 15:53:05 281473411271752 root: Repository: 'ax-docker' was added to scan
DEBUG 22-01-31 15:53:05 281473411271752 root: Using repository: /service/rest/v1/repositorySettings
INFO 22-01-31 15:53:05 281473411271752 root: Handling repository: ax-docker
DEBUG 22-01-31 15:53:05 281473411271752 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=ax-docker
DEBUG 22-01-31 15:55:56 281473411271752 root: Response is not JSON
DEBUG 22-01-31 15:55:56 281473411271752 root: Response return type: <class 'bytes'>
Traceback (most recent call last):
  File "/usr/bin/ws_nexus_integration", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 469, in main
    download_components_from_repositories(selected_repositories)
  File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 230, in download_components_from_repositories
    all_repo_items = get_items_from_repo(repo_name)
  File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 219, in get_items_from_repo
    continuation_token = cur_comp_resp['continuationToken']
TypeError: byte indices must be integers or slices, not str
tamari-oz commented 2 years ago

Still unsure why it happens in the Nexus you are scanning (doesn't happen here). According to Sonatype this shouldn't happen

Can you please try 0.2.12a4? It will not process binary responses. Also did you install manually install pre-release 0.9.1a1? This version contains new caching feature but still has some bugs and missing behaviour). IMO Better stick with 0.9.0.3 for now.

danielnbalasoiu commented 2 years ago

ws-sdk 0.9.1a1 installs automatically as a dependency for either 0.2.12a3 or 0.2.12.a4:

ws-nexus-integration==0.2.12a4 installation log ```shell /opt/whitesource # pip install ws-nexus-integration==0.2.12a4 Collecting ws-nexus-integration==0.2.12a4 Using cached ws_nexus_integration-0.2.12a4-py3-none-any.whl (13 kB) Requirement already satisfied: requests~=2.27.1 in /usr/lib/python3.9/site-packages (from ws-nexus-integration==0.2.12a4) (2.27.1) Requirement already satisfied: wget~=3.2 in /usr/lib/python3.9/site-packages (from ws-nexus-integration==0.2.12a4) (3.2) Collecting ws-sdk==0.9.1a1 Using cached ws_sdk-0.9.1a1-py3-none-any.whl (45 kB) Requirement already satisfied: DateTime~=4.3 in /usr/lib/python3.9/site-packages (from ws-sdk==0.9.1a1->ws-nexus-integration==0.2.12a4) (4.3) Requirement already satisfied: cachetools~=5.0.0 in /usr/lib/python3.9/site-packages (from ws-sdk==0.9.1a1->ws-nexus-integration==0.2.12a4) (5.0.0) Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests~=2.27.1->ws-nexus-integration==0.2.12a4) (3.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.9/site-packages (from requests~=2.27.1->ws-nexus-integration==0.2.12a4) (2021.10.8) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests~=2.27.1->ws-nexus-integration==0.2.12a4) (1.26.8) Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/lib/python3.9/site-packages (from requests~=2.27.1->ws-nexus-integration==0.2.12a4) (2.0.10) Requirement already satisfied: pytz in /usr/lib/python3.9/site-packages (from DateTime~=4.3->ws-sdk==0.9.1a1->ws-nexus-integration==0.2.12a4) (2021.3) Requirement already satisfied: zope.interface in /usr/lib/python3.9/site-packages (from DateTime~=4.3->ws-sdk==0.9.1a1->ws-nexus-integration==0.2.12a4) (5.4.0) Requirement already satisfied: setuptools in /usr/lib/python3.9/site-packages (from zope.interface->DateTime~=4.3->ws-sdk==0.9.1a1->ws-nexus-integration==0.2.12a4) (60.5.0) Installing collected packages: ws-sdk, ws-nexus-integration Attempting uninstall: ws-sdk Found existing installation: ws-sdk 0.9.0.3 Uninstalling ws-sdk-0.9.0.3: Successfully uninstalled ws-sdk-0.9.0.3 Attempting uninstall: ws-nexus-integration Found existing installation: ws-nexus-integration 0.2.12a3 Uninstalling ws-nexus-integration-0.2.12a3: Successfully uninstalled ws-nexus-integration-0.2.12a3 Successfully installed ws-nexus-integration-0.2.12a4 ws-sdk-0.9.1a1 ```

And I get an error when I install the stable version of the SDK:

ws-sdk==0.9.0.3 installation log ```shell /opt/whitesource # pip install ws-sdk==0.9.0.3 Collecting ws-sdk==0.9.0.3 Using cached ws_sdk-0.9.0.3-py3-none-any.whl (45 kB) Requirement already satisfied: requests~=2.27.1 in /usr/lib/python3.9/site-packages (from ws-sdk==0.9.0.3) (2.27.1) Requirement already satisfied: DateTime~=4.3 in /usr/lib/python3.9/site-packages (from ws-sdk==0.9.0.3) (4.3) Requirement already satisfied: zope.interface in /usr/lib/python3.9/site-packages (from DateTime~=4.3->ws-sdk==0.9.0.3) (5.4.0) Requirement already satisfied: pytz in /usr/lib/python3.9/site-packages (from DateTime~=4.3->ws-sdk==0.9.0.3) (2021.3) Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/lib/python3.9/site-packages (from requests~=2.27.1->ws-sdk==0.9.0.3) (2.0.10) Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.9/site-packages (from requests~=2.27.1->ws-sdk==0.9.0.3) (2021.10.8) Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests~=2.27.1->ws-sdk==0.9.0.3) (3.3) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests~=2.27.1->ws-sdk==0.9.0.3) (1.26.8) Requirement already satisfied: setuptools in /usr/lib/python3.9/site-packages (from zope.interface->DateTime~=4.3->ws-sdk==0.9.0.3) (60.5.0) Installing collected packages: ws-sdk Attempting uninstall: ws-sdk Found existing installation: ws-sdk 0.9.1a1 Uninstalling ws-sdk-0.9.1a1: Successfully uninstalled ws-sdk-0.9.1a1 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. ws-nexus-integration 0.2.12a4 requires ws-sdk==0.9.1a1, but you have ws-sdk 0.9.0.3 which is incompatible. Successfully installed ws-sdk-0.9.0.3 ``` ```shell /opt/whitesource # pip list | grep ws ws-nexus-integration 0.2.12a4 ws-sdk 0.9.0.3 ```

I have tested 0.2.12a4:

Nexus Scanning log ```shell /opt/whitesource # DEBUG=1 ws_nexus_integration DEBUG 22-02-01 06:38:48 281473884994632 root: Using configuration file: 'params.config' DEBUG 22-02-01 06:38:48 281473884994632 root: Converting user and password to basic string DEBUG 22-02-01 06:38:48 281473884994632 ws_sdk.ws_utilities: Executing command: ['java', '-version'] DEBUG 22-02-01 06:38:48 281473884994632 ws_sdk.ws_utilities: Java version: '1.8.0_302' DEBUG 22-02-01 06:38:48 281473884994632 ws_sdk.ws_utilities: Executing command: ['java', '-Djava.io.tmpdir=/tmp/ws-nexus-integration', '-jar', '/tmp/ws-nexus-integration/wss-unified-agent.jar', '-v', '-noConfig', 'True '] DEBUG 22-02-01 06:38:49 281473884994632 ws_sdk.client: UA executed successfully. Return Code 0. Message: 22.1.1.1 DEBUG 22-02-01 06:38:49 281473884994632 ws_sdk.client: Local WhiteSource Unified Agent version 22.1.1.1 DEBUG 22-02-01 06:38:49 281473884994632 ws_sdk.ws_utilities: Calling url: https://api.github.com/repos/whitesource/unified-agent-distribution/releases/latest DEBUG 22-02-01 06:38:50 281473884994632 ws_sdk.ws_utilities: Latest Unified Agent version: v22.1.1.1 DEBUG 22-02-01 06:38:50 281473884994632 ws_sdk.client: Skipping WhiteSource Unified Agent update DEBUG 22-02-01 06:38:50 281473884994632 root: Sending request for retrieving Nexus repository list DEBUG 22-02-01 06:38:50 281473884994632 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/repositories DEBUG 22-02-01 06:38:50 281473884994632 root: Response return type: INFO 22-02-01 06:38:50 281473884994632 root: Nexus Version: Nexus/3.37.3-02 (OSS) DEBUG 22-02-01 06:38:50 281473884994632 root: The following repositories were found: ['ax-docker'] DEBUG 22-02-01 06:38:50 281473884994632 root: Repository: 'axiros-docker' was added to scan DEBUG 22-02-01 06:38:50 281473884994632 root: Using repository: /service/rest/v1/repositorySettings INFO 22-02-01 06:38:50 281473884994632 root: Handling repository: axiros-docker DEBUG 22-02-01 06:38:50 281473884994632 root: Calling Nexus URL: http://nexus.host:8080/service/rest/v1/components?repository=axiros-docker DEBUG 22-02-01 06:41:23 281473884994632 root: Response is not JSON DEBUG 22-02-01 06:41:23 281473884994632 root: Response return type: DEBUG 22-02-01 06:41:23 281473884994632 root: Found 0 items in repository: 'axiros-docker' DEBUG 22-02-01 06:41:23 281473884994632 root: No artifacts found in axiros-docker INFO 22-02-01 06:41:23 281473884994632 ws_sdk.client: Scanning Dir(s): /tmp/ws-nexus-integration/_wstemp to product: 'Nexus' DEBUG 22-02-01 06:41:23 281473884994632 ws_sdk.ws_utilities: Executing command: ['java', '-Djava.io.tmpdir=/tmp/ws-nexus-integration', '-jar', '/tmp/ws-nexus-integration/wss-unified-agent.jar', '-d', '/tmp/ws-nexus-integration/_wstemp', '-product', 'Nexus', '-project', 'IRRELEVANT', '-noConfig', 'True '] Traceback (most recent call last): File "/usr/bin/ws_nexus_integration", line 8, in sys.exit(main()) File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 471, in main return_code = execute_scan() File "/usr/lib/python3.9/site-packages/ws_nexus_integration/nexus_integration.py", line 433, in execute_scan ret = config.ws_conn.scan(scan_dir=config.scan_dir, File "/usr/lib/python3.9/site-packages/ws_sdk/client.py", line 215, in scan ret = self._execute_ua(options=f"-d {existing_dirs} {dest_scope.to_execute()}", ua_conf=local_ua_all_conf) File "/usr/lib/python3.9/site-packages/ws_sdk/client.py", line 96, in _execute_ua _handle_ws_client_errors() File "/usr/lib/python3.9/site-packages/ws_sdk/client.py", line 78, in _handle_ws_client_errors raise ws_errors.WsSdkClientGenericError(output) ws_sdk.ws_errors.WsSdkClientGenericError: Generic error running Unified Agent: [INFO] [2022-02-01 06:41:25,240 +0000] - UnifiedAgent version (pluginVersion) : 22.1.1.1 [INFO] [2022-02-01 06:41:25,246 +0000] - ##### general ##### wss.url=https://saas-eu.whitesourcesoftware.com/agent whiteSourceFolderPath=/tmp/ws-nexus-integration userKey=****** showProgressBar=false scanComment=agent:ps-nexus-integration;agentVersion:0.2.12a4;comment:None projectTag= projectPerFolder=true projectName=IRRELEVANT productName=Nexus offline=false includes=**/*.7z,**/*.a,**/*.aar,**/*.air,**/*.apk,**/*.ar,**/*.bz2,**/*.car,**/*.conda,**/*.cpa,**/*.crate,**/*.deb,**/*.dll,**/*.dmg,**/*.docker,**/*.drpm,**/*.ear,**/*.egg,**/*.epk,**/*.exe,**/*.gal,**/*.gem,**/*.gzip,**/*.har,**/*.hpi,**/*.jar,**/*.jar.pack.gz,**/*.jpi,**/*.js,**/*.ko,**/*.min.js,**/*.msi,**/*.nupkg,**/*.pkg.tar.xz,**/*.rpm,**/*.sar,**/*.sit,**/*.so,**/*.swc,**/*.swf,**/*.tar,**/*.tar.bz2,**/*.tar.gz,**/*.tbz,**/*.tgz,**/*.udeb,**/*.war,**/*.whl,**/*.zip,**/*.4th,**/*.6pl,**/*.6pm,**/*.8xk,**/*.8xk.txt,**/*.8xp,**/*.8xp.txt,**/*.E,**/*.ML,**/*._coffee,**/*._js,**/*._ls,**/*.abap,**/*.ada,**/*.adb,**/*.ado,**/*.adp,**/*.ads,**/*.agda,**/*.ahk,**/*.ahkl,**/*.aidl,**/*.aj,**/*.al,**/*.als,**/*.ampl,**/*.apl,**/*.app.src,**/*.applescript,**/*.arc,**/*.as,**/*.asax,**/*.asc,**/*.ascx,**/*.asd,**/*.ash,**/*.ashx,**/*.asmx,**/*.asp,**/*.aspx,**/*.au3,**/*.aug,**/*.auk,**/*.aw,**/*.awk,**/*.axd,**/*.axi,**/*.axi.erb,**/*.axs,**/*.axs.erb,**/*.b,**/*.bas,**/*.bash,**/*.bat,**/*.bats,**/*.bb,**/*.befunge,**/*.bf,**/*.bison,**/*.bmx,**/*.bones,**/*.boo,**/*.boot,**/*.brd,**/*.bro,**/*.brs,**/*.bsl,**/*.bsv,**/*.builder,**/*.bzl,**/*.c,**/*.c++,**/*.cake,**/*.capnp,**/*.cats,**/*.cbl,**/*.cc,**/*.ccp,**/*.cdf,**/*.ceylon,**/*.cfc,**/*.cfm,**/*.cfml,**/*.cgi,**/*.ch,**/*.chpl,**/*.chs,**/*.cirru,**/*.cjsx,**/*.ck,**/*.cl,**/*.cl2,**/*.click,**/*.clj,**/*.cljc,**/*.cljs,**/*.cljs.hl,**/*.cljscm,**/*.cljx,**/*.clp,**/*.cls,**/*.clw,**/*.cmd,**/*.cob,**/*.cobol,**/*.coffee,**/*.com,**/*.command,**/*.coq,**/*.cp,**/*.cpp,**/*.cps,**/*.cpy,**/*.cr,**/*.cs,**/*.csd,**/*.cshtml,**/*.csx,**/*.ctp,**/*.cu,**/*.cuh,**/*.cw,**/*.cxx,**/*.cy,**/*.d,**/*.dart,**/*.dats,**/*.db2,**/*.dcl,**/*.decls,**/*.dfm,**/*.di,**/*.djs,**/*.dlm,**/*.dm,**/*.do,**/*.doh,**/*.dpr,**/*.druby,**/*.duby,**/*.dyalog,**/*.dyl,**/*.dylan,**/*.e,**/*.ec,**/*.ecl,**/*.eclxml,**/*.eh,**/*.el,**/*.eliom,**/*.eliomi,**/*.elm,**/*.em,**/*.emacs,**/*.emacs.desktop,**/*.emberscript,**/*.eq,**/*.erl,**/*.es,**/*.es6,**/*.escript,**/*.ex,**/*.exs,**/*.eye,**/*.f,**/*.f03,**/*.f08,**/*.f77,**/*.f90,**/*.f95,**/*.factor,**/*.fan,**/*.fancypack,**/*.fcgi,**/*.feature,**/*.flex,**/*.flux,**/*.for,**/*.forth,**/*.fp,**/*.fpp,**/*.fr,**/*.frag,**/*.frg,**/*.frm,**/*.frt,**/*.frx,**/*.fs,**/*.fsh,**/*.fshader,**/*.fsi,**/*.fsx,**/*.fth,**/*.ftl,**/*.fun,**/*.fx,**/*.fxh,**/*.fy,**/*.g,**/*.g4,**/*.gap,**/*.gawk,**/*.gd,**/*.gdb,**/*.gdbinit,**/*.gemspec,**/*.geo,**/*.geom,**/*.gf,**/*.gi,**/*.glf,**/*.glsl,**/*.glslv,**/*.gml,**/*.gms,**/*.gnu,**/*.gnuplot,**/*.go,**/*.god,**/*.golo,**/*.gp,**/*.grace,**/*.groovy,**/*.grt,**/*.gs,**/*.gshader,**/*.gsp,**/*.gst,**/*.gsx,**/*.gtpl,**/*.gvy,**/*.gyp,**/*.gypi,**/*.h,**/*.h++,**/*.hats,**/*.hb,**/*.hcl,**/*.hh,**/*.hic,**/*.hlean,**/*.hlsl,**/*.hlsli,**/*.hpp,**/*.hqf,**/*.hrl,**/*.hs,**/*.hsc,**/*.hx,**/*.hxsl,**/*.hxx,**/*.hy,**/*.i7x,**/*.iced,**/*.icl,**/*.idc,**/*.idr,**/*.ihlp,**/*.ijs,**/*.ik,**/*.ily,**/*.inc,**/*.inl,**/*.ino,**/*.intr,**/*.io,**/*.ipf,**/*.ipp,**/*.irbrc,**/*.iss,**/*.j,**/*.jake,**/*.java,**/*.jbuilder,**/*.jflex,**/*.ji,**/*.jison,**/*.jisonlex,**/*.jl,**/*.jq,**/*.js,**/*.jsb,**/*.jscad,**/*.jsfl,**/*.jsm,**/*.jsp,**/*.jss,**/*.jsx,**/*.kicad_pcb,**/*.kid,**/*.krl,**/*.ksh,**/*.kt,**/*.ktm,**/*.kts,**/*.l,**/*.lagda,**/*.las,**/*.lasso,**/*.lasso8,**/*.lasso9,**/*.ldml,**/*.lean,**/*.lex,**/*.lfe,**/*.lgt,**/*.lhs,**/*.lid,**/*.lidr,**/*.lisp,**/*.litcoffee,**/*.ll,**/*.lmi,**/*.logtalk,**/*.lol,**/*.lookml,**/*.lpr,**/*.ls,**/*.lsl,**/*.lslp,**/*.lsp,**/*.lua,**/*.lvproj,**/*.ly,**/*.m,**/*.m4,**/*.ma,**/*.mak,**/*.make,**/*.mako,**/*.mao,**/*.mata,**/*.matah,**/*.mathematica,**/*.matlab,**/*.mawk,**/*.maxhelp,**/*.maxpat,**/*.maxproj,**/*.mcr,**/*.metal,**/*.minid,**/*.mir,**/*.mirah,**/*.mk,**/*.mkfile,**/*.ml,**/*.ml4,**/*.mli,**/*.mll,**/*.mly,**/*.mm,**/*.mmk,**/*.mms,**/*.mo,**/*.mod,**/*.model.lkml,**/*.monkey,**/*.moo,**/*.moon,**/*.mq4,**/*.mq5,**/*.mqh,**/*.ms,**/*.mspec,**/*.mss,**/*.mt,**/*.mu,**/*.muf,**/*.mumps,**/*.mxt,**/*.myt,**/*.n,**/*.nawk,**/*.nb,**/*.nbp,**/*.nc,**/*.ncl,**/*.ni,**/*.nim,**/*.nimrod,**/*.nit,**/*.nix,**/*.njs,**/*.nl,**/*.nlogo,**/*.nqp,**/*.nse,**/*.nsh,**/*.nsi,**/*.nu,**/*.numpy,**/*.numpyw,**/*.numsc,**/*.nut,**/*.ny,**/*.omgrofl,**/*.ooc,**/*.opa,**/*.opal,**/*.opencl,**/*.orc,**/*.os,**/*.ox,**/*.oxh,**/*.oxo,**/*.oxygene,**/*.oz,**/*.p,**/*.p4,**/*.p6,**/*.p6l,**/*.p6m,**/*.pac,**/*.pan,**/*.parrot,**/*.pas,**/*.pascal,**/*.pasm,**/*.pat,**/*.pb,**/*.pbi,**/*.pbt,**/*.pck,**/*.pd,**/*.pd_lua,**/*.pde,**/*.perl,**/*.ph,**/*.php,**/*.php3,**/*.php4,**/*.php5,**/*.phps,**/*.phpt,**/*.pig,**/*.pike,**/*.pir,**/*.pkb,**/*.pks,**/*.pl,**/*.pl6,**/*.plb,**/*.plot,**/*.pls,**/*.plsql,**/*.plt,**/*.pluginspec,**/*.plx,**/*.pm,**/*.pm6,**/*.pmod,**/*.pod,**/*.podsl,**/*.podspec,**/*.pogo,**/*.pony,**/*.pov,**/*.pp,**/*.pprx,**/*.prg,**/*.pri,**/*.pro,**/*.prolog,**/*.prw,**/*.ps1,**/*.psc,**/*.psd1,**/*.psgi,**/*.psm1,**/*.purs,**/*.pwn,**/*.pxd,**/*.pxi,**/*.py,**/*.py3,**/*.pyde,**/*.pyp,**/*.pyt,**/*.pyw,**/*.pyx,**/*.qbs,**/*.qml,**/*.r,**/*.r2,**/*.r3,**/*.rabl,**/*.rake,**/*.rb,**/*.rbbas,**/*.rbfrm,**/*.rbmnu,**/*.rbres,**/*.rbtbar,**/*.rbuild,**/*.rbuistate,**/*.rbw,**/*.rbx,**/*.rbxs,**/*.rd,**/*.re,**/*.reb,**/*.rebol,**/*.red,**/*.reds,**/*.rei,**/*.rex,**/*.rexx,**/*.rg,**/*.rkt,**/*.rktd,**/*.rktl,**/*.rl,**/*.rpy,**/*.rs,**/*.rs.in,**/*.rsc,**/*.rsh,**/*.rsx,**/*.ru,**/*.ruby,**/*.sage,**/*.sagews,**/*.sas,**/*.sats,**/*.sbt,**/*.sc,**/*.scad,**/*.scala,**/*.scd,**/*.sce,**/*.sch,**/*.sci,**/*.scm,**/*.sco,**/*.scpt,**/*.scrbl,**/*.self,**/*.sexp,**/*.sh,**/*.sh-session,**/*.sh.in,**/*.shader,**/*.shen,**/*.sig,**/*.sj,**/*.sjs,**/*.sl,**/*.sld,**/*.sls,**/*.sma,**/*.smali,**/*.sml,**/*.smt,**/*.smt2,**/*.sp,**/*.spec,**/*.spin,**/*.sps,**/*.sqf,**/*.sql,**/*.sra,**/*.sru,**/*.srw,**/*.ss,**/*.ssjs,**/*.st,**/*.stan,**/*.sthlp,**/*.sv,**/*.svh,**/*.swift,**/*.t,**/*.tac,**/*.tcc,**/*.tcl,**/*.tf,**/*.thor,**/*.thrift,**/*.thy,**/*.tla,**/*.tm,**/*.tmux,**/*.tool,**/*.tpl,**/*.tpp,**/*.ts,**/*.tst,**/*.tsx,**/*.tu,**/*.txl,**/*.uc,**/*.udo,**/*.uno,**/*.upc,**/*.ur,**/*.urs,**/*.v,**/*.vala,**/*.vapi,**/*.vark,**/*.vb,**/*.vba,**/*.vbhtml,**/*.vbs,**/*.vcl,**/*.veo,**/*.vert,**/*.vh,**/*.vhd,**/*.vhdl,**/*.vhf,**/*.vhi,**/*.vho,**/*.vhs,**/*.vht,**/*.vhw,**/*.view.lkml,**/*.vim,**/*.volt,**/*.vrx,**/*.vsh,**/*.vshader,**/*.w,**/*.watchr,**/*.webidl,**/*.wisp,**/*.wl,**/*.wlt,**/*.wlua,**/*.wsgi,**/*.x,**/*.x10,**/*.xc,**/*.xi,**/*.xm,**/*.xojo_code,**/*.xojo_menu,**/*.xojo_report,**/*.xojo_script,**/*.xojo_toolbar,**/*.xojo_window,**/*.xpl,**/*.xproc,**/*.xpy,**/*.xq,**/*.xql,**/*.xqm,**/*.xquery,**/*.xqy,**/*.xrl,**/*.xs,**/*.xsjs,**/*.xsjslib,**/*.xsl,**/*.xslt,**/*.xsp-config,**/*.xsp.metadata,**/*.xtend,**/*.y,**/*.yacc,**/*.yap,**/*.yrl,**/*.yy,**/*.zep,**/*.zimpl,**/*.zmpl,**/*.zpl,**/*.zsh configFilePath=DEFAULT apiKey=****** #updateType=OVERRIDE #updateInventory=true #updateEmptyProject=true #scanReportTimeoutMinutes=10 #scanReportFilenameFormat=project_with_timestamp #scanPackageManager=false #resolveAllDependencies=true #requireKnownSha1=true #requesterEmail= #projectVersion= #projectToken= #projectPerFolderIncludes=* #projectPerFolderExcludes= #productVersion= #productToken= #offline.zip=false #offline.prettyJson=true #log.level=info #ignoreEuaNotices=informational #iaLanguage= #generateScanReport=false #generateProjectDetailsJson=false #forceUpdate.failBuildOnPolicyViolation=false #forceUpdate=false #forceCheckAllDependencies=false #fileSystemScan=true #failErrorLevel=DEFAULT #excludes=**/.*,**/node_modules,**/src/test,**/testdata,**/*sources.jar,**/*javadoc.jar #excludeDependenciesFromNodes= #euaVul=euaVul.json #euaRes=euaRes.json #euaOffline= #euaDep=euaDep.json #enableImpactAnalysis=false #commandTimeout=900 #checkPolicies=false #appPath= #analyzeMultiModule= #analyzeFrameworksReference=analyzeFrameworks.txt #analyzeFrameworks=false ##### ANT ##### #ant.resolveDependencies=true #ant.pathIdIncludes= #ant.external.parameters= ##### archive ##### #archiveIncludes= #archiveExtractionDepth=0 #archiveExcludes= ##### artifactory ##### #artifactory.userPassword= #artifactory.userName= #artifactory.url= #artifactory.repoKeys= #artifactory.includes=.*.* #artifactory.excludes= #artifactory.enableScan=false #artifactory.accessToken= ##### bazel ##### #bazel.runPreStep=false #bazel.resolveDependencies=true ##### bower ##### #bower.runPreStep=false #bower.resolveDependencies=true #bower.ignoreSourceFiles=false ##### cabal ##### #haskell.runPreStep=false #haskell.resolveDependencies=true #haskell.ignoreSourceFiles=true #haskell.ignorePreStepErrors=false ##### cargo ##### #cargo.runPreStep=false #cargo.resolveDependencies=true #cargo.ignoreSourceFiles=true ##### cocoapods ##### #cocoapods.runPreStep=false #cocoapods.resolveDependencies=true #cocoapods.ignoreSourceFiles=true ##### conda ##### #conda.resolveDependencies=true ##### docker ##### #docker.scanTarFiles=false #docker.scanImages=false #docker.scanContainers=false #docker.pull.tags= #docker.pull.maxImages=10 #docker.pull.images= #docker.pull.force=false #docker.pull.enable=false #docker.pull.digest= #docker.projectNameFormat=default #docker.login.sudo=true #docker.layers=false #docker.includes=.*.* #docker.includeSingleScan= #docker.excludes= #docker.delete.force=false #docker.containerIncludes=.*.* #docker.containerExcludes= ##### docker artifactory ##### #docker.artifactory.userPassword= #docker.artifactory.userName= #docker.artifactory.url= #docker.artifactory.repositoriesNames= #docker.artifactory.pullUrl= #docker.artifactory.enable=false #docker.artifactory.dockerAccessMethod= ##### docker aws ##### #docker.aws.registryIds= #docker.aws.enable=false ##### docker azure ##### #docker.azure.userPassword= #docker.azure.userName= #docker.azure.registryNames= #docker.azure.registryAuthenticationParameters= #docker.azure.enable=false #docker.azure.authenticationType=UserAccount ##### docker google containers ##### #docker.gcr.repositories= #docker.gcr.enable=false #docker.gcr.account= ##### docker hub ##### #docker.hub.userPassword= #docker.hub.userName= #docker.hub.organizationsNames= #docker.hub.enabled=false #docker.hub.accessToken= ##### environment ##### #followSymbolicLinks=true #case.sensitive.glob=false ##### go ##### #go.resolveDependencies=true #go.modules.resolveDependencies=true #go.modules.removeDuplicateDependencies=true #go.modules.includeTestDependencies=false #go.modules.ignoreSourceFiles=true #go.ignoreSourceFiles=false #go.gogradle.enableTaskAlias=false #go.glide.ignoreTestPackages=true #go.dependencyManager= #go.collectDependenciesAtRuntime=false ##### gradle ##### #gradle.resolveDependencies=true #gradle.preferredEnvironment=wrapper #gradle.innerModulesAsDependencies=true #gradle.includedConfigurations= #gradle.includeModules= #gradle.ignoredConfigurations= #gradle.ignoreSourceFiles=false #gradle.excludeModules= #gradle.aggregateModules=false #gradle.additionalArguments= ##### hex ##### #hex.runPreStep=false #hex.resolveDependencies=true #hex.ignoreSourceFiles=true #hex.aggregateModules=false ##### html ##### #html.resolveDependencies=true ##### maven ##### #maven.runPreStep=false #maven.resolveDependencies=true #maven.projectNameFromDependencyFile=false #maven.maxCacheFolderSize= #maven.m2RepositoryPath= #maven.ignoredScopes=provided,test #maven.ignoreSourceFiles=false #maven.ignorePomModules=true #maven.ignoreMvnTreeErrors=false #maven.environmentPath= #maven.downloadMissingDependencies=true #maven.aggregateModules=false #maven.additionalArguments= ##### npm ##### #npm.yarnProject=false #npm.yarn.frozenLockfile=false #npm.runPreStep=false #npm.resolveMainPackageJsonOnly=false #npm.resolveLockFile=true #npm.resolveGlobalPackages=false #npm.resolveDependencies=true #npm.resolveAdditionalDependencies=false #npm.removeDuplicateDependencies=true #npm.projectNameFromDependencyFile=false #npm.includeDevDependencies=false #npm.ignoreSourceFiles=true #npm.ignoreScripts=false #npm.ignoreNpmLsErrors=false #npm.ignoreDirectoryPatterns=example,examples,.ws_bower,test #npm.identifyByNameAndVersion=false #npm.failOnNpmLsErrors= #npm.accessToken= ##### nuget ##### #nuget.runPreStep=false #nuget.resolvePackagesConfigFiles=true #nuget.resolveNuspecFiles=true #nuget.resolveDependencies=true #nuget.resolveCsProjFiles=true #nuget.resolveAssetsFiles=true #nuget.preferredEnvironment= #nuget.packagesDirectory= #nuget.ignoreSourceFiles=true ##### ocaml ##### #ocaml.switchName= #ocaml.runPreStep=false #ocaml.resolveDependencies=true #ocaml.ignoredScopes=with-test,with-doc #ocaml.ignoreSourceFiles=false #ocaml.aggregateModules=false ##### paket ##### #paket.runPreStep=false #paket.resolveDependencies=true #paket.ignoredGroups= #paket.ignoreSourceFiles=true #paket.exePath= ##### php ##### #php.runPreStep=false #php.resolveDependencies=true #php.removeDuplicateDependencies=true #php.includeDevDependencies=false #php.ignoreSourceFiles=true ##### python ##### #python.runPoetryPreStep=false #python.runPipenvPreStep=false #python.resolveSetupPyFiles=false #python.resolvePipEditablePackages=false #python.resolveHierarchyTree=true #python.resolveGlobalPackages=false #python.resolveDependencies=true #python.requirementsFileIncludes= #python.pipenvDevDependencies=false #python.pipPath=pip #python.path=python #python.localPackagePathsToInstall= #python.installVirtualenv=false #python.indexUrl= #python.includePoetryDevDependencies=false #python.ignoreSourceFiles=true #python.ignorePipInstallErrors=false #python.IgnorePipenvInstallErrors=false ##### r_language ##### #r.runPreStep=false #r.resolveDependencies=true #r.packageManager=packrat #r.ignoreSourceFiles=true #r.cranMirrorUrl= ##### ruby ##### #ruby.runBundleInstall=false #ruby.resolveDependencies=true #ruby.overwriteGemFile=false #ruby.installMissingGems=false #ruby.ignoreSourceFiles=true ##### sbt ##### #sbt.runPreStep=false #sbt.resolveDependencies=true #sbt.includedScopes=compile,runtime #sbt.ignoreSourceFiles=false #sbt.aggregateModules=false ##### scm ##### #scm.user= #scm.url= #scm.tag= #scm.repositoriesFile= #scm.ppk= #scm.pass= #scm.branch= ##### serverless ##### #serverless.scanFunctions=false #serverless.region= #serverless.provider=aws-lambda #serverless.maxFunctions=10 #serverless.includes= #serverless.excludes= [WARN] [2022-02-01 06:41:25,317 +0000] - ------------------------------------------------------------------------ -------------- START - File System Permissions Validations ------------- ------------------------------------------------------------------------ [WARN] [2022-02-01 06:41:25,317 +0000] - Logs - File or Directory does not exists: /tmp/ws-nexus-integration/./whitesource [WARN] [2022-02-01 06:41:25,317 +0000] - ------------------------------------------------------------------------ -------------- END - File System Permissions Validations --------------- ------------------------------------------------------------------------ [INFO] [2022-02-01 06:41:25,999 +0000] - Starting analysis [INFO] [2022-02-01 06:41:25,999 +0000] - Scanning directories [/tmp/ws-nexus-integration/_wstemp] for package dependencies (may take a few minutes) [INFO] [2022-02-01 06:41:25,999 +0000] - Included file types: **/*.7z,**/*.a,**/*.aar,**/*.air,**/*.apk,**/*.ar,**/*.bz2,**/*.car,**/*.conda,**/*.cpa,**/*.crate,**/*.deb,**/*.dll,**/*.dmg,**/*.docker,**/*.drpm,**/*.ear,**/*.egg,**/*.epk,**/*.exe,**/*.gal,**/*.gem,**/*.gzip,**/*.har,**/*.hpi,**/*.jar,**/*.jar.pack.gz,**/*.jpi,**/*.js,**/*.ko,**/*.min.js,**/*.msi,**/*.nupkg,**/*.pkg.tar.xz,**/*.rpm,**/*.sar,**/*.sit,**/*.so,**/*.swc,**/*.swf,**/*.tar,**/*.tar.bz2,**/*.tar.gz,**/*.tbz,**/*.tgz,**/*.udeb,**/*.war,**/*.whl,**/*.zip,**/*.4th,**/*.6pl,**/*.6pm,**/*.8xk,**/*.8xk.txt,**/*.8xp,**/*.8xp.txt,**/*.E,**/*.ML,**/*._coffee,**/*._js,**/*._ls,**/*.abap,**/*.ada,**/*.adb,**/*.ado,**/*.adp,**/*.ads,**/*.agda,**/*.ahk,**/*.ahkl,**/*.aidl,**/*.aj,**/*.al,**/*.als,**/*.ampl,**/*.apl,**/*.app.src,**/*.applescript,**/*.arc,**/*.as,**/*.asax,**/*.asc,**/*.ascx,**/*.asd,**/*.ash,**/*.ashx,**/*.asmx,**/*.asp,**/*.aspx,**/*.au3,**/*.aug,**/*.auk,**/*.aw,**/*.awk,**/*.axd,**/*.axi,**/*.axi.erb,**/*.axs,**/*.axs.erb,**/*.b,**/*.bas,**/*.bash,**/*.bat,**/*.bats,**/*.bb,**/*.befunge,**/*.bf,**/*.bison,**/*.bmx,**/*.bones,**/*.boo,**/*.boot,**/*.brd,**/*.bro,**/*.brs,**/*.bsl,**/*.bsv,**/*.builder,**/*.bzl,**/*.c,**/*.c++,**/*.cake,**/*.capnp,**/*.cats,**/*.cbl,**/*.cc,**/*.ccp,**/*.cdf,**/*.ceylon,**/*.cfc,**/*.cfm,**/*.cfml,**/*.cgi,**/*.ch,**/*.chpl,**/*.chs,**/*.cirru,**/*.cjsx,**/*.ck,**/*.cl,**/*.cl2,**/*.click,**/*.clj,**/*.cljc,**/*.cljs,**/*.cljs.hl,**/*.cljscm,**/*.cljx,**/*.clp,**/*.cls,**/*.clw,**/*.cmd,**/*.cob,**/*.cobol,**/*.coffee,**/*.com,**/*.command,**/*.coq,**/*.cp,**/*.cpp,**/*.cps,**/*.cpy,**/*.cr,**/*.cs,**/*.csd,**/*.cshtml,**/*.csx,**/*.ctp,**/*.cu,**/*.cuh,**/*.cw,**/*.cxx,**/*.cy,**/*.d,**/*.dart,**/*.dats,**/*.db2,**/*.dcl,**/*.decls,**/*.dfm,**/*.di,**/*.djs,**/*.dlm,**/*.dm,**/*.do,**/*.doh,**/*.dpr,**/*.druby,**/*.duby,**/*.dyalog,**/*.dyl,**/*.dylan,**/*.e,**/*.ec,**/*.ecl,**/*.eclxml,**/*.eh,**/*.el,**/*.eliom,**/*.eliomi,**/*.elm,**/*.em,**/*.emacs,**/*.emacs.desktop,**/*.emberscript,**/*.eq,**/*.erl,**/*.es,**/*.es6,**/*.escript,**/*.ex,**/*.exs,**/*.eye,**/*.f,**/*.f03,**/*.f08,**/*.f77,**/*.f90,**/*.f95,**/*.factor,**/*.fan,**/*.fancypack,**/*.fcgi,**/*.feature,**/*.flex,**/*.flux,**/*.for,**/*.forth,**/*.fp,**/*.fpp,**/*.fr,**/*.frag,**/*.frg,**/*.frm,**/*.frt,**/*.frx,**/*.fs,**/*.fsh,**/*.fshader,**/*.fsi,**/*.fsx,**/*.fth,**/*.ftl,**/*.fun,**/*.fx,**/*.fxh,**/*.fy,**/*.g,**/*.g4,**/*.gap,**/*.gawk,**/*.gd,**/*.gdb,**/*.gdbinit,**/*.gemspec,**/*.geo,**/*.geom,**/*.gf,**/*.gi,**/*.glf,**/*.glsl,**/*.glslv,**/*.gml,**/*.gms,**/*.gnu,**/*.gnuplot,**/*.go,**/*.god,**/*.golo,**/*.gp,**/*.grace,**/*.groovy,**/*.grt,**/*.gs,**/*.gshader,**/*.gsp,**/*.gst,**/*.gsx,**/*.gtpl,**/*.gvy,**/*.gyp,**/*.gypi,**/*.h,**/*.h++,**/*.hats,**/*.hb,**/*.hcl,**/*.hh,**/*.hic,**/*.hlean,**/*.hlsl,**/*.hlsli,**/*.hpp,**/*.hqf,**/*.hrl,**/*.hs,**/*.hsc,**/*.hx,**/*.hxsl,**/*.hxx,**/*.hy,**/*.i7x,**/*.iced,**/*.icl,**/*.idc,**/*.idr,**/*.ihlp,**/*.ijs,**/*.ik,**/*.ily,**/*.inc,**/*.inl,**/*.ino,**/*.intr,**/*.io,**/*.ipf,**/*.ipp,**/*.irbrc,**/*.iss,**/*.j,**/*.jake,**/*.java,**/*.jbuilder,**/*.jflex,**/*.ji,**/*.jison,**/*.jisonlex,**/*.jl,**/*.jq,**/*.js,**/*.jsb,**/*.jscad,**/*.jsfl,**/*.jsm,**/*.jsp,**/*.jss,**/*.jsx,**/*.kicad_pcb,**/*.kid,**/*.krl,**/*.ksh,**/*.kt,**/*.ktm,**/*.kts,**/*.l,**/*.lagda,**/*.las,**/*.lasso,**/*.lasso8,**/*.lasso9,**/*.ldml,**/*.lean,**/*.lex,**/*.lfe,**/*.lgt,**/*.lhs,**/*.lid,**/*.lidr,**/*.lisp,**/*.litcoffee,**/*.ll,**/*.lmi,**/*.logtalk,**/*.lol,**/*.lookml,**/*.lpr,**/*.ls,**/*.lsl,**/*.lslp,**/*.lsp,**/*.lua,**/*.lvproj,**/*.ly,**/*.m,**/*.m4,**/*.ma,**/*.mak,**/*.make,**/*.mako,**/*.mao,**/*.mata,**/*.matah,**/*.mathematica,**/*.matlab,**/*.mawk,**/*.maxhelp,**/*.maxpat,**/*.maxproj,**/*.mcr,**/*.metal,**/*.minid,**/*.mir,**/*.mirah,**/*.mk,**/*.mkfile,**/*.ml,**/*.ml4,**/*.mli,**/*.mll,**/*.mly,**/*.mm,**/*.mmk,**/*.mms,**/*.mo,**/*.mod,**/*.model.lkml,**/*.monkey,**/*.moo,**/*.moon,**/*.mq4,**/*.mq5,**/*.mqh,**/*.ms,**/*.mspec,**/*.mss,**/*.mt,**/*.mu,**/*.muf,**/*.mumps,**/*.mxt,**/*.myt,**/*.n,**/*.nawk,**/*.nb,**/*.nbp,**/*.nc,**/*.ncl,**/*.ni,**/*.nim,**/*.nimrod,**/*.nit,**/*.nix,**/*.njs,**/*.nl,**/*.nlogo,**/*.nqp,**/*.nse,**/*.nsh,**/*.nsi,**/*.nu,**/*.numpy,**/*.numpyw,**/*.numsc,**/*.nut,**/*.ny,**/*.omgrofl,**/*.ooc,**/*.opa,**/*.opal,**/*.opencl,**/*.orc,**/*.os,**/*.ox,**/*.oxh,**/*.oxo,**/*.oxygene,**/*.oz,**/*.p,**/*.p4,**/*.p6,**/*.p6l,**/*.p6m,**/*.pac,**/*.pan,**/*.parrot,**/*.pas,**/*.pascal,**/*.pasm,**/*.pat,**/*.pb,**/*.pbi,**/*.pbt,**/*.pck,**/*.pd,**/*.pd_lua,**/*.pde,**/*.perl,**/*.ph,**/*.php,**/*.php3,**/*.php4,**/*.php5,**/*.phps,**/*.phpt,**/*.pig,**/*.pike,**/*.pir,**/*.pkb,**/*.pks,**/*.pl,**/*.pl6,**/*.plb,**/*.plot,**/*.pls,**/*.plsql,**/*.plt,**/*.pluginspec,**/*.plx,**/*.pm,**/*.pm6,**/*.pmod,**/*.pod,**/*.podsl,**/*.podspec,**/*.pogo,**/*.pony,**/*.pov,**/*.pp,**/*.pprx,**/*.prg,**/*.pri,**/*.pro,**/*.prolog,**/*.prw,**/*.ps1,**/*.psc,**/*.psd1,**/*.psgi,**/*.psm1,**/*.purs,**/*.pwn,**/*.pxd,**/*.pxi,**/*.py,**/*.py3,**/*.pyde,**/*.pyp,**/*.pyt,**/*.pyw,**/*.pyx,**/*.qbs,**/*.qml,**/*.r,**/*.r2,**/*.r3,**/*.rabl,**/*.rake,**/*.rb,**/*.rbbas,**/*.rbfrm,**/*.rbmnu,**/*.rbres,**/*.rbtbar,**/*.rbuild,**/*.rbuistate,**/*.rbw,**/*.rbx,**/*.rbxs,**/*.rd,**/*.re,**/*.reb,**/*.rebol,**/*.red,**/*.reds,**/*.rei,**/*.rex,**/*.rexx,**/*.rg,**/*.rkt,**/*.rktd,**/*.rktl,**/*.rl,**/*.rpy,**/*.rs,**/*.rs.in,**/*.rsc,**/*.rsh,**/*.rsx,**/*.ru,**/*.ruby,**/*.sage,**/*.sagews,**/*.sas,**/*.sats,**/*.sbt,**/*.sc,**/*.scad,**/*.scala,**/*.scd,**/*.sce,**/*.sch,**/*.sci,**/*.scm,**/*.sco,**/*.scpt,**/*.scrbl,**/*.self,**/*.sexp,**/*.sh,**/*.sh-session,**/*.sh.in,**/*.shader,**/*.shen,**/*.sig,**/*.sj,**/*.sjs,**/*.sl,**/*.sld,**/*.sls,**/*.sma,**/*.smali,**/*.sml,**/*.smt,**/*.smt2,**/*.sp,**/*.spec,**/*.spin,**/*.sps,**/*.sqf,**/*.sql,**/*.sra,**/*.sru,**/*.srw,**/*.ss,**/*.ssjs,**/*.st,**/*.stan,**/*.sthlp,**/*.sv,**/*.svh,**/*.swift,**/*.t,**/*.tac,**/*.tcc,**/*.tcl,**/*.tf,**/*.thor,**/*.thrift,**/*.thy,**/*.tla,**/*.tm,**/*.tmux,**/*.tool,**/*.tpl,**/*.tpp,**/*.ts,**/*.tst,**/*.tsx,**/*.tu,**/*.txl,**/*.uc,**/*.udo,**/*.uno,**/*.upc,**/*.ur,**/*.urs,**/*.v,**/*.vala,**/*.vapi,**/*.vark,**/*.vb,**/*.vba,**/*.vbhtml,**/*.vbs,**/*.vcl,**/*.veo,**/*.vert,**/*.vh,**/*.vhd,**/*.vhdl,**/*.vhf,**/*.vhi,**/*.vho,**/*.vhs,**/*.vht,**/*.vhw,**/*.view.lkml,**/*.vim,**/*.volt,**/*.vrx,**/*.vsh,**/*.vshader,**/*.w,**/*.watchr,**/*.webidl,**/*.wisp,**/*.wl,**/*.wlt,**/*.wlua,**/*.wsgi,**/*.x,**/*.x10,**/*.xc,**/*.xi,**/*.xm,**/*.xojo_code,**/*.xojo_menu,**/*.xojo_report,**/*.xojo_script,**/*.xojo_toolbar,**/*.xojo_window,**/*.xpl,**/*.xproc,**/*.xpy,**/*.xq,**/*.xql,**/*.xqm,**/*.xquery,**/*.xqy,**/*.xrl,**/*.xs,**/*.xsjs,**/*.xsjslib,**/*.xsl,**/*.xslt,**/*.xsp-config,**/*.xsp.metadata,**/*.xtend,**/*.y,**/*.yacc,**/*.yap,**/*.yrl,**/*.yy,**/*.zep,**/*.zimpl,**/*.zmpl,**/*.zpl,**/*.zsh [INFO] [2022-02-01 06:41:25,999 +0000] - Excluded file types: **/.*,**/node_modules,**/src/test,**/testdata,**/*sources.jar,**/*javadoc.jar [INFO] [2022-02-01 06:41:26,003 +0000] - ------------------------------------------------------------------------ -------------------- Start: Scan Files Matching Includes Pattern ------- ------------------------------------------------------------------------ [INFO] [2022-02-01 06:41:26,006 +0000] - Scanning directories [/tmp/ws-nexus-integration/_wstemp] for matching source/binary file types (may take a few minutes) [INFO] [2022-02-01 06:41:26,016 +0000] - Total files found according to the includes/excludes pattern: 0 [INFO] [2022-02-01 06:41:26,016 +0000] - ------------------------------------------------------------------------ -------------------- End: Scan Files Matching Includes Pattern --------- ------------------------------------------------------------------------ [INFO] [2022-02-01 06:41:26,017 +0000] - Finished analyzing Files [INFO] [2022-02-01 06:41:26,068 +0000] - Initializing WhiteSource Client [INFO] [2022-02-01 06:41:26,070 +0000] - Service URL is https://saas-eu.whitesourcesoftware.com/agent [INFO] [2022-02-01 06:41:26,094 +0000] - UpdateType set to OVERRIDE [INFO] [2022-02-01 06:41:26,094 +0000] - Sending Update [INFO] [2022-02-01 06:41:26,094 +0000] - ------------------------------------------------------------------------ -------------------- Start: Update Inventory --------------------------- ------------------------------------------------------------------------ [ERROR] [2022-02-01 06:41:26,787 +0000] - Failed to send request to WhiteSource server: Illegal arguments: Bad org token [INFO] [2022-02-01 06:41:26,789 +0000] - Support token: 25a26af9910424cf98bfc1829d862e30d1643697686893 [INFO] [2022-02-01 06:41:26,791 +0000] - Process finished with exit code SERVER_FAILURE (Failed to send request to WhiteSource server: Illegal arguments: Bad org token Support token: 25a26af9910424cf98bfc1829d862e30d1643697686893) [INFO] [2022-02-01 06:41:26,791 +0000] - Log files are found at: ./whitesource [INFO] [2022-02-01 06:41:26,795 +0000] - ------------------------------------------------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------- WhiteSource Scan Summary: -------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------ ====================================================================================================================================================== Scan Origin: Local File System ====================================================================================================================================================== Step Completion Status Elapsed Comments ====================================================================================================================================================== Fetch Configuration COMPLETED 00:00:00.131 -------- Scan Files Matching Includes Pattern COMPLETED 00:00:00.013 0 source/binary files ====================================================================================================================================================== Elapsed running time: 00:00:00.144 ====================================================================================================================================================== Process finished with exit code SERVER_FAILURE (-5) . Return Code: 251 ```
tamari-oz commented 2 years ago

Got it. The last log looks better. The failure now looks like incorrect org token/ user key or permissions.

danielnbalasoiu commented 2 years ago

Can you double check that the user key is at least "Product Integrator" on the designated product ('Nexus'), permission of org administrator is also fine. and the org token is correct?

  • I've been using this key for other type of scans and it's working fine. I can see the scan reports in WhiteSource Dashboard.

Also it doesn't find any Images 'axiros-docker' (log: Found 0 items in repository: 'axiros-docker'). Is this the actual case?

  • No, there are a lot of container images in there.

Can you provide more information on your Nexus setup? How did you install it? Where it is deployed?

  • The Nexus server was installed by a colleague in a VM using the Nexus Repository tar.gz file provided by sonatype.
danielnbalasoiu commented 2 years ago

I created a new docker registry just for testing ws-nexus (and avoid waiting to check hundreds of container images) it's working:

ws-nexus-integration 0.2.12a4
ws-sdk               0.9.0.3
[INFO] [2022-02-14 10:14:16,125 +0000] - Project name: IRRELEVANT, URL: https://saas-eu.whitesourcesoftware.com/Wss/WSS.html#!project;id=1195087
[INFO] [2022-02-14 10:14:16,125 +0000] - Support Token: 3b500228c6aa84a71a4f3ce76a16815171644833654373
[INFO] [2022-02-14 10:14:16,130 +0000] -
------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------- WhiteSource Scan Summary: --------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------
======================================================================================================================================================
Scan Origin: Local File System
======================================================================================================================================================
Step                                              Completion Status               Elapsed                  Comments
======================================================================================================================================================
Fetch Configuration                                  COMPLETED                  00:00:00.107               --------
Scan Files Matching Includes Pattern                 COMPLETED                  00:00:00.013               0 source/binary files
Update Inventory                                     COMPLETED                  00:00:02.605               1 updated projects

======================================================================================================================================================
Elapsed running time:                                                           00:00:02.725
======================================================================================================================================================
Process finished with exit code SUCCESS (0)

DEBUG 22-02-14 10:14:16 281473675177032 ws_sdk.app: Calling: https://saas-eu.whitesourcesoftware.com/api/v1.3 with requestType: getRequestState
DEBUG 22-02-14 10:14:16 281473675177032 ws_sdk.app: API getRequestState call on orgToken 20a**** succeeded
NatalyaDalid commented 2 years ago

@danielnbalasoiu Thank you for the update!