OutOfMemoryError while generating 'SpdxDocument' report

[harshadsinkar]

Describe the bug

The error occurs while attempting to generate a 'SpdxDocument' report, resulting in an OutOfMemoryError. The specific error message states: "UTF16 String size is 1310357935, should be less than 1073741823", indicating that an attempt to create a UTF16 string exceeded the maximum allowed size, leading to the error.

To Reproduce

Steps to reproduce the behavior:

1. scan-result.json file must be around ~800Mb
2. Try to use report tool to generate 'SpdxDocument'

Expected behavior

SPDX report is generated.

Console / log output

Add console and / or log output that shows the error and additional context. No screenshots of plain text please, to keep text searchable.

07:19:27.769 [main] INFO  org.ossreviewtoolkit.plugins.commands.api.utils.ExtensionsKt - Read ORT result from 'scan-result.json' (786.76 MiB) in 9.969460793s.
Generating the 'SpdxDocument' report in thread 'DefaultDispatcher-worker-2'...
07:22:44.570 [main] ERROR org.ossreviewtoolkit.plugins.commands.reporter.ReporterCommand - Could not create 'SpdxDocument' report in 2m 42.163936651s: OutOfMemoryError: UTF16 String size is 1310357935, should be less than 1073741823
Created 0 of 1 report(s) in 2m 42.346035902s.

Environment

Output of the ort requirements command:

 ______________________________
/        \_______   \__    ___/ The OSS Review Toolkit, version 25.0.0-026.sha.5886773,
|    |   | |       _/ |    |    built with JDK 11.0.23+9-post-Ubuntu-1ubuntu122.04.1, running under Java 11.0.23.
|    |   | |    |   \ |    |    Executing 'requirements' as 'ospouser' on Linux
\________/ |____|___/ |____|    with 4 CPUs and a maximum of 12288 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = /home/ospouser/.ort/config
ORT_DATA_DIR = /home/ospouser/.ort
HOME = /home/ospouser
SHELL = /bin/bash
TERM = xterm-256color
JAVA_HOME = /usr/lib/jvm/default-java

Looking for ORT configuration in the following file:
        /home/ospouser/.ort/config/config.yml (does not exist)

AdviceProviderFactory plugins:
        * GitHubDefects
        * NexusIQ
        * OssIndex
        * OSV
        * VulnerableCode

OrtCommand plugins:
        * advise
        * analyze
        * compare
        * config
        * download
        * evaluate
        * migrate
        * notify
        * report
        * requirements
        * scan
        * upload-curations
        * upload-result-to-postgres
        * upload-result-to-sw360

PackageConfigurationProviderFactory plugins:
        * DefaultDir
        * Dir
        * DOS
        * OrtConfig

PackageCurationProviderFactory plugins:
        * ClearlyDefined
        * DefaultDir
        * DefaultFile
        * File
        * OrtConfig
        * SW360

PackageManagerFactory plugins:
        * Bazel
        * Bower
        * Bundler
        * Cargo
        * Carthage
        * CocoaPods
        * Composer
        * Conan
        * GoMod
        * Gradle
        * GradleInspector
        * Maven
        * NPM
        * NuGet
        * PIP
        * Pipenv
        * PNPM
        * Poetry
        * Pub
        * SBT
        * SpdxDocumentFile
        * Stack
        * SwiftPM
        * Unmanaged
        * Yarn
        * Yarn2

Reporter plugins:
        * CtrlXAutomation
        * CycloneDx
        * DocBookTemplate
        * EvaluatedModel
        * FossId
        * FossIdSnippet
        * GitLabLicenseModel
        * HtmlTemplate
        * ManPageTemplate
        * Opossum
        * PdfTemplate
        * PlainTextTemplate
        * SpdxDocument
        * StaticHtml
        * TrustSource
        * WebApp

ScannerWrapperFactory plugins:
        * Askalono
        * BoyterLc
        * FossId
        * Licensee
        * ScanCode
        * SCANOSS

VersionControlSystem plugins:
        * Git
        * GitRepo
        * Mercurial
        * Subversion

Scanners:
        - Askalono: Requires 'askalono' in no specific version. Tool not found.
        - BoyterLc: Requires 'lc' in no specific version. Tool not found.
        - Licensee: Requires 'licensee' in no specific version. Tool not found.
        * ScanCode: Requires 'scancode' in version >=3.0.0. Found version 32.2.0.

PackageManagers:
        - Bazel: Requires 'bazel' in version >=7.0.0. Tool not found.
        - Bower: Requires 'bower' in version >=1.8.8. Tool not found.
        - Cargo: Requires 'cargo' in no specific version. Tool not found.
        - CocoaPods: Requires 'pod' in version >=1.11.0. Tool not found.
        - Composer: Requires 'composer' in version >=1.5.0. Tool not found.
        - Conan: Requires 'conan' in version >=1.44.0 and <2.0.0. Tool not found.
        - GoMod: Requires 'go' in version >=1.21.1. Tool not found.
        * Npm: Requires 'npm' in version >=6.0.0 and <11.0.0. Found version 8.5.1.
        + NuGetInspector: Requires 'nuget-inspector' in no specific version. Could not determine the version.
        - Pipenv: Requires 'pipenv' in version >=2018.10.9. Tool not found.
        - Pnpm: Requires 'pnpm' in version >=5.0.0 and <9.0.0. Tool not found.
        - Poetry: Requires 'poetry' in no specific version. Tool not found.
        - Pub: Requires 'dart' in version >=2.10.0. Tool not found.
        - PythonInspector: Requires 'python-inspector' in version >=0.9.2. Tool not found.
        - Sbt: Requires 'sbt' in version >=0.13.0. Tool not found.
        - Stack: Requires 'stack' in version >=2.1.1. Tool not found.
        - SwiftPm: Requires 'swift' in no specific version. Tool not found.
        - Yarn: Requires 'yarn' in version >=1.3.0 and <1.23.0. Tool not found.

VersionControlSystems:
        * Git: Requires 'git' in version >=2.29.0. Found version 2.34.1.
        - GitRepo: Requires 'repo' in no specific version. Tool not found.
        - Mercurial: Requires 'hg' in no specific version. Tool not found.

Prefix legend:
        - The tool was not found in the PATH environment.
        + The tool was found in the PATH environment, but not in the required version.
        * The tool was found in the PATH environment in the required version.

ScanCode license texts found in '/home/ospouser/Downloads/scancode-toolkit-v32.2.0/venv/lib/python3.10/site-packages/licensedcode/data/licenses'.

Not all tools requirements were satisfied:
        ! Some tools were not found at all.

Additional context

As stated here https://github.com/oss-review-toolkit/ort?tab=readme-ov-file#system-requirements, following option was also exported and same outcome was observed:

export JAVA_OPTS="$JAVA_OPTS -Xmx8g"

[fviernau]

@harshadsinkar - I cannot reproduce, so I made a speculative fix. Can you please try if it works? (#8832)

[harshadsinkar]

@fviernau - Thank you for quick fix-#8832. Tested and it works. 👍

[fviernau]

Thanks @harshadsinkar for testing.