search

aboutcode-org / scancode-toolkit

:mag: ScanCode detects licenses, copyrights, dependencies by "scanning code" ... to discover and inventory open source and third-party packages used in your code. Sponsored by NLnet project https://nlnet.nl/project/vulnerabilitydatabase, the Google Summer of Code, Azure credits, nexB and others generous sponsors!
https://github.com/aboutcode-org/scancode-toolkit/releases/
2.09k stars 541 forks source link

ScanCode 3.2.1rc2 fails to load the 'scan:licenses' plugin #2286

Closed fviernau closed 2 years ago

fviernau commented 3 years ago

I installed ScanCode 3.2.1rc2 in Docker using ubuntu:18.04 as base image. Therefore I downgraded Python from version 3.7 to version 3.6.9 as I understood this to be required. In the docker file I configured scancode as follows:

    # Install ScanCode, configure it for Python 3 and re-index the licenses for performace.
    curl -ksSL https://github.com/nexB/scancode-toolkit/archive/v$SCANCODE_VERSION.tar.gz | tar -zxC /usr/local && \
    PYTHON_EXE=/usr/bin/python3 /usr/local/scancode-toolkit-$SCANCODE_VERSION/scancode --reindex-licenses && \
    chmod -R o=u /usr/local/scancode-toolkit-$SCANCODE_VERSION && \
    ln -s /usr/local/scancode-toolkit-$SCANCODE_VERSION/scancode /usr/local/bin/scancode

When executing ScanCode it runs into the following issue. It's not yet understood whether it reproduces reliably:

Error: Invalid value: ERROR: failed to setup plugin: scan:licenses:
Traceback (most recent call last):
File "/usr/local/scancode-toolkit-3.2.1rc2/src/licensedcode/cache.py", line 280, in load_index
return LicenseIndex.load(ifc)
File "/usr/local/scancode-toolkit-3.2.1rc2/src/licensedcode/index.py", line 977, in load
idx = pickler.load(fn)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfd in position 0: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/scancode-toolkit-3.2.1rc2/src/scancode/cli.py", line 774, in run_scan
plugin.setup(**requested_options)
File "/usr/local/scancode-toolkit-3.2.1rc2/src/licensedcode/plugin_license.py", line 130, in setup
get_index(return_value=False)
File "/usr/local/scancode-toolkit-3.2.1rc2/src/licensedcode/cache.py", line 68, in get_index
_LICENSES_BY_KEY_INDEX = get_cached_index(cache_dir, check_consistency)
File "/usr/local/scancode-toolkit-3.2.1rc2/src/licensedcode/cache.py", line 216, in get_cached_index
return load_index(cache_file)
File "/usr/local/scancode-toolkit-3.2.1rc2/src/licensedcode/cache.py", line 289, in load_index
raise ex_type(message).with_traceback(ex_traceback)
TypeError: function takes exactly 5 arguments (1 given)
pombredanne commented 3 years ago

@fviernau Thanks! Do you have a link to the Dockerfile?

pombredanne commented 3 years ago

Using the Dockerfile below (derived from ort's and essentially the same) :

So for now I cannot reproduce this as is

(thanks to @daniel-eder for the docker instructions ;) ...)

FROM adoptopenjdk:11-jre-hotspot-bionic

ENV \
    # Package manager versions.
    BOWER_VERSION=1.8.8 \
    BUNDLER_VERSION=1.16.1-1 \
    CARGO_VERSION=0.44.1-0ubuntu1~18.04.1 \
    COMPOSER_VERSION=1.6.3-1 \
    CONAN_VERSION=1.18.0 \
    FLUTTER_VERSION=v1.12.13+hotfix.9-stable \
    GO_DEP_VERSION=0.5.4 \
    GO_VERSION=1.13.4 \
    HASKELL_STACK_VERSION=2.1.3 \
    NPM_VERSION=6.14.2 \
    PYTHON_PIP_VERSION=9.0.1-2.3~ubuntu1.18.04.3 \
    PYTHON_PIPENV_VERSION=2018.11.26 \
    PYTHON_VIRTUALENV_VERSION=15.1.0 \
    SBT_VERSION=1.3.8 \
    YARN_VERSION=1.22.4 \
    # SDK versions.
    ANDROID_SDK_VERSION=6609375 \
    # Scanner versions.
    SCANCODE_VERSION=3.2.1rc2 \
    # Installation directories.
    ANDROID_HOME=/opt/android-sdk \
    FLUTTER_HOME=/opt/flutter \
    GOPATH=$HOME/go

ENV DEBIAN_FRONTEND=noninteractive \
    PATH="$PATH:$HOME/.local/bin:$FLUTTER_HOME/bin:$FLUTTER_HOME/bin/cache/dart-sdk/bin:$GOPATH/bin:/opt/go/bin"

# Apt install commands.
RUN  \
    apt-get update && \
    apt-get install -y --no-install-recommends gnupg && \
    echo 'Acquire::https::dl.bintray.com::Verify-Peer "false";' | tee -a /etc/apt/apt.conf.d/00sbt && \
    echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list && \
    curl -ksS "https://keyserver.ubuntu.com/pks/lookup?op=get&options=mr&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | apt-key adv --import - && \
    curl -sL https://deb.nodesource.com/setup_12.x | bash - && \
    apt-get update && \
    apt-get install -y --no-install-recommends \
        # Install general tools required by this Dockerfile.
        lib32stdc++6 \
        libffi-dev \
        libgmp-dev \
        libxext6 \
        libxi6 \
        libxrender1 \
        libxtst6 \
        make \
        netbase \
        openssh-client \
        unzip \
        xz-utils \
        zlib1g-dev \
        # Install VCS tools (no specific versions required here).
        cvs \
        git \
        mercurial \
        subversion \
        # Install package managers (in versions known to work).
        bundler=$BUNDLER_VERSION \
        cargo=$CARGO_VERSION \
        composer=$COMPOSER_VERSION \
        nodejs \
        python-pip=$PYTHON_PIP_VERSION \
        python-setuptools \
        python3-pip=$PYTHON_PIP_VERSION \
        python3-setuptools \
        sbt=$SBT_VERSION \
    && \
    rm -rf /var/lib/apt/lists/*

# Custom install commands.
RUN \
    # Install VCS tools (no specific versions required here).
    curl -ksS https://storage.googleapis.com/git-repo-downloads/repo > /usr/local/bin/repo && \
    chmod a+x /usr/local/bin/repo && \
    # Install package managers (in versions known to work).
    npm install --global npm@$NPM_VERSION bower@$BOWER_VERSION yarn@$YARN_VERSION && \
    pip install wheel && \
    pip install conan==$CONAN_VERSION pipenv==$PYTHON_PIPENV_VERSION virtualenv==$PYTHON_VIRTUALENV_VERSION && \
    curl -ksSO https://storage.googleapis.com/flutter_infra/releases/stable/linux/flutter_linux_$FLUTTER_VERSION.tar.xz && \
    tar xf flutter_linux_$FLUTTER_VERSION.tar.xz -C $(dirname $FLUTTER_HOME) && \
    rm flutter_linux_$FLUTTER_VERSION.tar.xz && \
    chmod -R a+rw $FLUTTER_HOME && \
    flutter config --no-analytics && \
    flutter doctor && \
    # Install golang in order to have `go mod` as package manager.
    curl -ksSO https://dl.google.com/go/go$GO_VERSION.linux-amd64.tar.gz && \
    tar -C /opt -xzf go$GO_VERSION.linux-amd64.tar.gz && \
    rm go$GO_VERSION.linux-amd64.tar.gz && \
    mkdir -p $GOPATH/bin && \
    curl -ksS https://raw.githubusercontent.com/golang/dep/v$GO_DEP_VERSION/install.sh | sh && \
    curl -ksS https://raw.githubusercontent.com/commercialhaskell/stack/v$HASKELL_STACK_VERSION/etc/scripts/get-stack.sh | sh && \
    # Install SDKs required for analysis.
    curl -Os https://dl.google.com/android/repository/commandlinetools-linux-${ANDROID_SDK_VERSION}_latest.zip && \
    unzip -q commandlinetools-linux-${ANDROID_SDK_VERSION}_latest.zip -d $ANDROID_HOME && \
    rm commandlinetools-linux-${ANDROID_SDK_VERSION}_latest.zip && \
    PROXY_HOST_AND_PORT=${https_proxy#*://} && \
    if [ -n "$PROXY_HOST_AND_PORT" ]; then \
        # While sdkmanager uses HTTPS by default, the proxy type is still called "http".
        SDK_MANAGER_PROXY_OPTIONS="--proxy=http --proxy_host=${PROXY_HOST_AND_PORT%:*} --proxy_port=${PROXY_HOST_AND_PORT##*:}"; \
    fi && \
    yes | $ANDROID_HOME/tools/bin/sdkmanager $SDK_MANAGER_PROXY_OPTIONS --sdk_root=$ANDROID_HOME "platform-tools" && \
    # Add scanners (in versions known to work).
    curl -ksSL https://github.com/nexB/scancode-toolkit/archive/v$SCANCODE_VERSION.tar.gz | \
        tar -zxC /usr/local && \
        # Trigger ScanCode configuration for Python 3 and reindex licenses initially.
        PYTHON_EXE=/usr/bin/python3 /usr/local/scancode-toolkit-$SCANCODE_VERSION/scancode --reindex-licenses && \
        chmod -R o=u /usr/local/scancode-toolkit-$SCANCODE_VERSION && \
        ln -s /usr/local/scancode-toolkit-$SCANCODE_VERSION/scancode /usr/local/bin/scancode

ENTRYPOINT ["/usr/local/bin/scancode"]
pombredanne commented 3 years ago

Do you have an example of scan options and a file that fails?

fviernau commented 3 years ago

Scanner command / options: /usr/local/bin/scancode --copyright --license --ignore *.ort.yml --info --strip-root --timeout 300 --ignore HERE_NOTICE --ignore META-INF/DEPENDENCIES --processes 15 /scan-dir --json-pp /result.json

I actually do not have the file that fails. Could it be that it does not fail on any specific file but before scanning any file? I believe the below bit was missing initially.

Setup plugins...
Removing temporary files...done.
Usage: scancode [OPTIONS] <OUTPUT FORMAT OPTION(s)> <input>...
Try 'scancode --help' for help on options and arguments.

Error: Invalid value: ERROR: failed to setup plugin: scan:licenses:
Traceback (most recent call last):

Unfortunately I don't have the reproduce instructions available yet, and the docker image used is company internal. I can further investigate that and come back with the results. But that'd be next week then.

pombredanne commented 3 years ago

I tried with this : docker run -v $PWD/:/project scancode-toolkit --copyright --license --ignore "*.ort.yml" --info --strip-root --timeout 300 --ignore HERE_NOTICE --ignore META-INF/DEPENDENCIES --processes 15 --json-pp /project/result.json /project/NOTICE and the the command runs fine too.

It could be that somehow your container may not have been built correctly and should be re-built?

66Ton99 commented 3 years ago

Have similar problem

scancode-toolkit git:(master) virtualenv -p /usr/bin/python3.6 venv-scancode
created virtual environment CPython3.6.10.final.0-64 in 203ms
  creator CPython3Posix(dest=/srv/src/scancode-toolkit/venv-scancode, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ton/.local/share/virtualenv)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
➜  scancode-toolkit git:(master) source venv-scancode/bin/activate
(venv-scancode) ➜  scancode-toolkit git:(master) ./scancode --html opencv.html --copyright --package /srv/src/ofsc/opencv
Setup plugins...
Removing temporary files...done.
Usage: scancode [OPTIONS] <OUTPUT FORMAT OPTION(s)> <input>...
Try the 'scancode --help' option for help on options and arguments.

Error: Invalid value: ERROR: failed to setup plugin: scan:licenses:
Traceback (most recent call last):
  File "/srv/src/scancode-toolkit/src/scancode/cli.py", line 702, in run_scan
    plugin.setup(**requested_options)
  File "/srv/src/scancode-toolkit/src/licensedcode/plugin_license.py", line 130, in setup
    get_index(return_value=False)
  File "/srv/src/scancode-toolkit/src/licensedcode/cache.py", line 68, in get_index
    _LICENSES_BY_KEY_INDEX = get_cached_index(cache_dir, check_consistency)
  File "/srv/src/scancode-toolkit/src/licensedcode/cache.py", line 249, in get_cached_index
    idx = LicenseIndex(rules, _spdx_tokens=spdx_tokens)
  File "/srv/src/scancode-toolkit/src/licensedcode/index.py", line 245, in __init__
    rules, _legalese=_legalese, _spdx_tokens=_spdx_tokens)
  File "/srv/src/scancode-toolkit/src/licensedcode/index.py", line 533, in _add_rules
    raise AssertionError(msg)
AssertionError: Duplicate rules: 
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/rpsl-1.0_2.RULE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/rpsl-1.0_7.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/licenses/mpl-1.1.LICENSE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/mpl-1.1_28.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/licenses/liberation-font-exception.LICENSE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/liberation-font-exception_1.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/licenses/nosl-3.0.LICENSE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/nosl-3.0.SPDX.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/licenses/afl-2.1.LICENSE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/afl-2.1_2.RULE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/afl-2.1_6.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/lgpl-3.0.SPDX.RULE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/lgpl-3.0_23.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/cddl-1.0.SPDX.RULE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/cddl-1.0_9.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/licenses/cecill-1.1.LICENSE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/cecill-1.1.SPDX.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/afl-3.0.SPDX.RULE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/afl-3.0_6.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/licenses/cpal-1.0.LICENSE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/cpal-1.0.SPDX.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/rpsl-1.0.SPDX.RULE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/rpsl-1.0_1.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/licenses/npl-1.1.LICENSE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/npl-1.1_7.RULE

file:///srv/src/scancode-toolkit/src/licensedcode/data/licenses/osl-3.0.LICENSE
file:///srv/src/scancode-toolkit/src/licensedcode/data/rules/osl-3.0.SPDX.RULE

(venv-scancode) ➜  scancode-toolkit git:(master)
pombredanne commented 3 years ago

@66Ton99 I think yours may be a different problem entirely... let me move this to a new ticket :)

pombredanne commented 2 years ago

@fviernau I am closing this for now, as this is for an ancient version. Please create a new ticket if this happens again.