search

DataDog / guarddog

:snake: :mag: GuardDog is a CLI tool to Identify malicious PyPI and npm packages
https://securitylabs.datadoghq.com/articles/guarddog-identify-malicious-pypi-packages/
Apache License 2.0
585 stars 43 forks source link

Received unsupported operand type(s) for &: 'set' and 'tuple'% #404

Closed fjmnav-nudge closed 2 months ago

fjmnav-nudge commented 2 months ago

After updating to version v1.11.0 we have started seeing these errors:

DEBUG: Extracting archive /var/folders/jd/x0bfj3wj39zdjyc_jws_xl680000gn/T/tmpzazeoekn/validators.tar.gz to directory /var/folders/jd/x0bfj3wj39zdjyc_jws_xl680000gn/T/tmpzazeoekn/validators
DEBUG: Successfully extracted files to /var/folders/jd/x0bfj3wj39zdjyc_jws_xl680000gn/T/tmpzazeoekn/validators
DEBUG: Removing temporary archive file /var/folders/jd/x0bfj3wj39zdjyc_jws_xl680000gn/T/tmpzazeoekn/validators.tar.gz
DEBUG: Retrieving PyPI package metadata from https://pypi.org/pypi/validators/json
DEBUG: Running metadata rules against package 'validators'
Received unsupported operand type(s) for &: 'set' and 'tuple'%
DEBUG: Successfully extracted files to /var/folders/jd/x0bfj3wj39zdjyc_jws_xl680000gn/T/tmpqb4rze_2/yamale
DEBUG: Removing temporary archive file /var/folders/jd/x0bfj3wj39zdjyc_jws_xl680000gn/T/tmpqb4rze_2/yamale.tar.gz
DEBUG: Retrieving PyPI package metadata from https://pypi.org/pypi/yamale/json
DEBUG: Running metadata rules against package 'yamale'
Received unsupported operand type(s) for &: 'set' and 'tuple'%

Reverting back to v1.10.0 fixes the issue.

sobregosodd commented 2 months ago

Hello @fjmnav-nudge thanks for reporting.

Can you provide which python version are you using, arch, OS, and full command line, I'm not able to reproduce your error (tried both docker and pypi package)

> docker run --rm ghcr.io/datadog/guarddog:v1.11.0 --log-level=DEBUG pypi scan yamale
DEBUG: Considering that 'yamale' is a remote target
DEBUG: Retrieving PyPI package metadata from https://pypi.org/pypi/yamale/json
DEBUG: Downloading package archive from https://files.pythonhosted.org/packages/a1/52/0faa32aa15f241a9f950ded276c942db69bce8dda5f19241f6b960080dca/yamale-5.2.1.tar.gz into /tmp/tmpxfbxfbmn/yamale
DEBUG: Extracting archive /tmp/tmpxfbxfbmn/yamale.tar.gz to directory /tmp/tmpxfbxfbmn/yamale
DEBUG: Successfully extracted files to /tmp/tmpxfbxfbmn/yamale
DEBUG: Removing temporary archive file /tmp/tmpxfbxfbmn/yamale.tar.gz
DEBUG: Retrieving PyPI package metadata from https://pypi.org/pypi/yamale/json
DEBUG: Running metadata rules against package 'yamale'
DEBUG: Running rule potentially_compromised_email_domain against package 'yamale'
DEBUG: Running rule unclaimed_maintainer_email_domain against package 'yamale'
DEBUG: Running rule bundled_binary against package 'yamale'
DEBUG: Running bundled binary heuristic on package yamale version None
DEBUG: Running rule single_python_file against package 'yamale'
DEBUG: Running rule empty_information against package 'yamale'
DEBUG: Running PyPI empty description heuristic on package yamale version None
DEBUG: Running rule deceptive_author against package 'yamale'
DEBUG: Running rule repository_integrity_mismatch against package 'yamale'
DEBUG: Running repository integrity mismatch heuristic on PyPI package yamale version None
DEBUG: Using GitHub URL https://github.com/23andMe/Yamale
DEBUG: Running rule typosquatting against package 'yamale'
DEBUG: Running typosquatting heuristic on PyPI package yamale
DEBUG: Running rule release_zero against package 'yamale'
DEBUG: Running zero version heuristic on PyPI package yamale version None
DEBUG: Running source code rules against directory '/tmp/tmpxfbxfbmn/yamale'
DEBUG: Running source code rules against /tmp/tmpxfbxfbmn/yamale
DEBUG: Invoking semgrep with command line: semgrep --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/exfiltrate-sensitive-data.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/shady-links.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/obfuscation.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/exec-base64.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/clipboard-access.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/download-executable.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/steganography.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/dll-hijacking.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/cmd-overwrite.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/silent-process-execution.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/code-execution.yml --config /usr/local/lib/python3.10/site-packages/guarddog/analyzer/sourcecode/bidirectional-characters.yml --exclude='helm' --exclude='.idea' --exclude='venv' --exclude='test' --exclude='tests' --exclude='.env' --exclude='dist' --exclude='build' --exclude='semgrep' --exclude='migrations' --exclude='.github' --exclude='.semgrep_logs' --no-git-ignore --json --quiet --max-target-bytes=10000000 /tmp/tmpxfbxfbmn/yamale
Found 0 potentially malicious indicators scanning yamale
fjmnav-nudge commented 2 months ago

Hi @sobregosodd

you are totally right, sorry I forgot to paste the exact command:

docker run --rm ghcr.io/datadog/guarddog:v1.11.0 --log-level=DEBUG pypi scan yamale --rules obfuscation --rules exfiltrate-sensitive-data --rules download-executable --rules silent-process-execution --rules steganography --rules cmd-overwrite --rules release_zero --rules potentially_compromised_email_domain --rules single_python_file --exit-non-zero-on-finding

the output:

DEBUG: Considering that 'yamale' is a remote target
DEBUG: Retrieving PyPI package metadata from https://pypi.org/pypi/yamale/json
DEBUG: Downloading package archive from https://files.pythonhosted.org/packages/a1/52/0faa32aa15f241a9f950ded276c942db69bce8dda5f19241f6b960080dca/yamale-5.2.1.tar.gz into /tmp/tmp6n4xcgqm/yamale
DEBUG: Extracting archive /tmp/tmp6n4xcgqm/yamale.tar.gz to directory /tmp/tmp6n4xcgqm/yamale
DEBUG: Successfully extracted files to /tmp/tmp6n4xcgqm/yamale
DEBUG: Removing temporary archive file /tmp/tmp6n4xcgqm/yamale.tar.gz
DEBUG: Retrieving PyPI package metadata from https://pypi.org/pypi/yamale/json
DEBUG: Running metadata rules against package 'yamale'

Error 'unsupported operand type(s) for &: 'set' and 'tuple'' occurred while scanning remote package.%

as soon as I specify the rules it breaks

sobregosodd commented 2 months ago

Thank you, I was able to reproduce, it would be straight forward to fix

fjmnav-nudge commented 2 months ago

thank you for the quick fix @sobregosodd !! It is working great now!