Requirements.txt not able to be analyzed

h4ckerfr0st commented 3 weeks ago

Issue analyzing requirements.txt

Basically i have a project in which i am running o gitlab ci/cd in Docker and im encountering an error when it comes to analyze requirements.txt that owasp_dependency_check informs.

What have i done until now based on the error below whilst running this command:

Remove any analogous character : cat -A /app/requirements.txt
Identify encoding now at this moment is : requirements.txt: text/plain; charset=us-ascii, same as utf-8 baiscally
run different places and commands
Run a different requirements.txt from other sources

This is my requirements.txt :

boto3==1.24.32
sqlalchemy==1.3.24
psycopg2==2.8.6
ujson==1.35
flask==1.0.3
flask-cors==3.0.9
pytz==2023.3.post1
cryptography==41.0.5
jira==2.0.0
requests==2.25.1
elasticsearch==7.14.0
elasticsearch-dsl==7.4.0
XlsxWriter==3.2.0
aiohttp==3.10.5
async-timeout==4.0.3
attrs==20.3.0
botocore==1.27.32
certifi==2020.12.5
cffi==1.14.5
chardet==3.0.4
click==8.1.3
debtcollector==2.2.0
defusedxml==0.7.1
eight==1.0.1
expiringdict==1.1.4
fasteners==0.16
future==0.18.2
idna==2.9
importlib-metadata==4.3.0
IPy==1.01
iso8601==0.1.14
itsdangerous==1.1.0
Jinja2==2.11.3
jmespath==0.10.0
MarkupSafe==1.1.1
motor==2.4.0
multidict==4.7.6
netaddr==0.8.0
netifaces==0.10.9
oauthlib==3.1.0
oslo.concurrency==3.25.0
oslo.config==8.5.0
oslo.i18n==5.0.1
oslo.utils==4.8.0
packaging==20.9
pbr==5.5.1
pycparser==2.20
PyJWT==2.0.1
pymongo==3.13.0
pyparsing==2.4.7
python-dateutil==2.7.3
PyYAML==6.0.1
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
rfc3986==1.4.0
s3transfer==0.6.0
six==1.15.0
stevedore==3.3.0
tenacity==5.1.4
typing==3.5.3.0
typing-extensions==3.7.4.3
urllib3==1.26.9
uWSGI==2.0.22
watchtower==0.8.0
Werkzeug==1.0.1
wrapt==1.14.1
yarl==1.6.3
zipp==3.11.0
anyio==3.5.0
charset-normalizer==3.3.2
h11==0.14.0
httpcore==0.17.0
httpx==0.24.0
sniffio==1.2.0
retry==0.9.2
decorator==5.1.1
elastic-apm[flask]>=6.8.1,<7
blinker>=1.5,<1.6

Below is my gitlab CI/CD pipeline.

dependency_check: image: xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/appsec/python_analyzer:latest stage: security script:

tr -d '$' < requirements.txt > requirements_cleaned.txt
mv requirements_cleaned.txt requirements.txt
dependency-check --disableAssembly --enableExperimental -s . -o build --format XML --project "xxxxxxxx" --exclude ".git/" --exclude ".venv/" --exclude "/pycache/" --exclude ".tox/**" --nvdApiKey "xxxxxxxxxxxxx"

About ODC: https://jeremylong.github.io/DependencyCheck/general/internals.html False Positives: https://jeremylong.github.io/DependencyCheck/general/suppression.html ? Sponsor: https://github.com/sponsors/jeremylong [INFO] Analysis Started [INFO] Finished Archive Analyzer (0 seconds) [INFO] Finished File Name Analyzer (0 seconds) [INFO] Finished Python Distribution Analyzer (0 seconds) [INFO] Finished Python Package Analyzer (0 seconds) [WARN] An unexpected error occurred during analysis of '/app/requirements.txt' (pip Analyzer): java.nio.charset.MalformedInputException: Input length = 1 [ERROR] java.io.UncheckedIOException: java.nio.charset.MalformedInputException: Input length = 1 at java.base/java.nio.file.FileChannelLinesSpliterator.readLine(FileChannelLinesSpliterator.java:176) at java.base/java.nio.file.FileChannelLinesSpliterator.forEachRemaining(FileChannelLinesSpliterator.java:116) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.owasp.dependencycheck.analyzer.PipAnalyzer.getFileContents(PipAnalyzer.java:188) at org.owasp.dependencycheck.analyzer.PipAnalyzer.analyzeDependency(PipAnalyzer.java:140) at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131) at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88) at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.nio.charset.MalformedInputException: Input length = 1 at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274) at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:350) at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:179) at java.base/java.io.BufferedReader.fill(BufferedReader.java:161) at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326) at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392) at java.base/java.nio.file.FileChannelLinesSpliterator.readLine(FileChannelLinesSpliterator.java:174)

Thanks for the help in the mean time if possible, please.

aikebah commented 3 weeks ago

Based on your symptoms my gut feel would be that your docker container is configured with a multi-byte characterset as the platform default encoding. The requirements.txt is read using the 'default charset' as established by the Java Charset class.

https://github.com/jeremylong/DependencyCheck/blob/59147b0af151676b46a3116a270f7bd785afb838/core/src/main/java/org/owasp/dependencycheck/analyzer/PipAnalyzer.java#L186

rdimameza commented 2 weeks ago

I had the same issue and I could fix it adding the next two steps:

name: Set default charset to UTF-8 run: | export JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF-8"
name: Convert requirements.txt to UTF-8 run: | iconv -f -t UTF-8 requirements.txt -o requirements.txt

jeremylong / DependencyCheck

Requirements.txt not able to be analyzed #7127