search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.07k stars 258 forks source link

Update `Python` dependencies to update deprecated packages and resolve conflicts #1973

Open DonggeLiu opened 2 months ago

DonggeLiu commented 2 months ago

Deprecations: https://github.com/google/fuzzbench/actions/runs/8752698488/job/24020848759?pr=1956#step:8:391

Conflicts:

ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11; 1.6.2 Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0 Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10; 1.7.2 Requires-Python >=3.7,<3.11; 1.7.3 Requires-Python >=3.7,<3.11; 1.8.0 Requires-Python >=3.8,<3.11; 1.8.0rc1 Requires-Python >=3.8,<3.11; 1.8.0rc2 Requires-Python >=3.8,<3.11; 1.8.0rc3 Requires-Python >=3.8,<3.11; 1.8.0rc4 Requires-Python >=3.8,<3.11; 1.8.1 Requires-Python >=3.8,<3.11; 1.9.5 Requires-Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.7; 2020.10.8 Requires-Python <3.9,>=3.5; 2020.11.12 Requires-Python <3.9,>=3.6; 2020.11.23 Requires-Python <3.9,>=3.6; 2020.11.3 Requires-Python <3.9,>=3.6; 2020.12.16 Requires-Python <3.9,>=3.6; 2020.12.2 Requires-Python <3.9,>=3.6; 2020.12.23 Requires-Python <3.9,>=3.6; 2020.2.20 Requires-Python !=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<3.8,>=2.7; 2020.3.19 Requires-Python !=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<3.8,>=2.7; 2020.4.1 Requires-Python !=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<3.8,>=2.7; 2020.4.22 Requires-Python <3.8,>=3.5; 2020.5.13 Requires-Python <3.8,>=3.5; 2020.5.7 Requires-Python <3.8,>=3.5; 2020.6.1 Requires-Python <3.9,>=3.5; 2020.6.26 Requires-Python <3.9,>=3.5; 2020.7.14 Requires-Python <3.9,>=3.5; 2020.7.20 Requires-Python <3.9,>=3.5; 2020.7.24 Requires-Python <3.9,>=3.5; 2020.7.30 Requires-Python <3.9,>=3.5; 2020.8.10 Requires-Python <3.9,>=3.5; 2020.8.17 Requires-Python <3.9,>=3.5; 2020.8.28 Requires-Python <3.9,>=3.5; 2020.9.14 Requires-Python <3.9,>=3.5; 2020.9.16 Requires-Python <3.9,>=3.5; 2020.9.24 Requires-Python <3.9,>=3.5; 2020.9.29 Requires-Python <3.9,>=3.5; 2021.1.14 Requires-Python <3.9,>=3.6; 2021.1.21 Requires-Python <3.9,>=3.6; 2021.1.28 Requires-Python <3.9,>=3.6; 2021.1.8 Requires-Python <3.9,>=3.6; 2021.10.11 Requires-Python <3.10,>=3.6; 2021.10.18 Requires-Python <3.10,>=3.6; 2021.10.25 Requires-Python <3.10,>=3.6; 2021.10.4 Requires-Python <3.10,>=3.6; 2021.11.12 Requires-Python <3.10,>=3.6; 2021.11.18 Requires-Python <3.10,>=3.6; 2021.11.2 Requires-Python <3.10,>=3.6; 2021.11.24 Requires-Python <3.10,>=3.6; 2021.11.29 Requires-Python <3.10,>=3.6; 2021.12.15 Requires-Python <3.10,>=3.6; 2021.12.8 Requires-Python <3.10,>=3.6; 2021.2.19 Requires-Python <3.9,>=3.6; 2021.2.23 Requires-Python <3.10,>=3.6; 2021.2.9 Requires-Python <3.9,>=3.6; 2021.3.10 Requires-Python <3.10,>=3.6; 2021.3.22 Requires-Python <3.10,>=3.6; 2021.3.3 Requires-Python <3.10,>=3.6; 2021.4.1 Requires-Python <3.10,>=3.6; 2021.4.15 Requires-Python <3.10,>=3.6; 2021.4.26 Requires-Python <3.10,>=3.6; 2021.4.9 Requires-Python <3.10,>=3.6; 2021.5.11 Requires-Python <3.10,>=3.6; 2021.5.14 Requires-Python <3.10,>=3.6; 2021.5.19 Requires-Python <3.10,>=3.6; 2021.5.25 Requires-Python <3.10,>=3.6; 2021.5.4 Requires-Python <3.10,>=3.6; 2021.5.6 Requires-Python <3.10,>=3.6; 2021.6.17 Requires-Python <3.10,>=3.6; 2021.7.19 Requires-Python <3.10,>=3.6; 2021.7.27 Requires-Python <3.10,>=3.6; 2021.8.11 Requires-Python <3.10,>=3.6; 2021.8.24 Requires-Python <3.10,>=3.6; 2021.8.3 Requires-Python <3.10,>=3.6; 2021.9.27 Requires-Python <3.10,>=3.6; 2021.9.9 Requires-Python <3.10,>=3.6; 2022.1.13 Requires-Python <3.10,>=3.7; 2022.1.31 Requires-Python <3.10,>=3.7; 2022.1.5 Requires-Python <3.10,>=3.6; 2022.1.7 Requires-Python <3.10,>=3.7; 2022.10.13 Requires-Python <3.11,>=3.7; 2022.10.26 Requires-Python <3.11,>=3.7; 2022.11.10 Requires-Python <3.11,>=3.7; 2022.11.18 Requires-Python <3.11,>=3.7; 2022.11.29 Requires-Python <3.11,>=3.7; 2022.12.15 Requires-Python <3.11,>=3.7; 2022.12.9 Requires-Python <3.11,>=3.7; 2022.2.17 Requires-Python <3.10,>=3.7; 2022.2.23 Requires-Python <3.10,>=3.7; 2022.2.8 Requires-Python <3.10,>=3.7; 2022.3.21 Requires-Python <3.10,>=3.7; 2022.3.29 Requires-Python <3.10,>=3.7; 2022.3.8 Requires-Python <3.10,>=3.7; 2022.4.15 Requires-Python <3.11,>=3.7; 2022.4.22 Requires-Python <3.11,>=3.7; 2022.4.26 Requires-Python <3.11,>=3.7; 2022.4.6 Requires-Python <3.10,>=3.7; 2022.5.10 Requires-Python <3.11,>=3.7; 2022.5.19 Requires-Python <3.11,>=3.7; 2022.5.5 Requires-Python <3.11,>=3.7; 2022.6.14 Requires-Python <3.11,>=3.7; 2022.6.23 Requires-Python <3.11,>=3.7; 2022.6.30 Requires-Python <3.11,>=3.7; 2022.6.6 Requires-Python <3.11,>=3.7; 2022.7.18 Requires-Python <3.11,>=3.7; 2022.7.26 Requires-Python <3.11,>=3.7; 2022.8.17 Requires-Python <3.11,>=3.7; 2022.8.23 Requires-Python <3.11,>=3.7; 2022.8.3 Requires-Python <3.11,>=3.7; 2022.8.30 Requires-Python <3.11,>=3.7; 2022.9.19 Requires-Python <3.11,>=3.7; 2022.9.27 Requires-Python <3.11,>=3.7; 2022.9.8 Requires-Python <3.11,>=3.7; 2023.1.10 Requires-Python <3.11,>=3.7; 2023.1.17 Requires-Python <3.11,>=3.7; 2023.1.31 Requires-Python <3.11,>=3.7; 2023.2.14 Requires-Python <3.11,>=3.7; 2023.2.17 Requires-Python <3.11,>=3.7; 2023.2.9 Requires-Python <3.11,>=3.7; 2023.3.13 Requires-Python <3.11,>=3.7; 2023.3.2 Requires-Python <3.11,>=3.7; 2023.3.31 Requires-Python <3.11,>=3.7; 2023.3.8 Requires-Python <3.11,>=3.7; 2023.4.11 Requires-Python <3.11,>=3.7; 2023.4.18 Requires-Python <3.11,>=3.7; 2023.4.27 Requires-Python <3.11,>=3.7; 2023.5.24 Requires-Python <3.11,>=3.7; 2023.5.8 Requires-Python <3.11,>=3.7; 2023.6.16 Requires-Python <3.11,>=3.7; 2023.6.2 Requires-Python <3.11,>=3.7; 2023.7.12 Requires-Python <3.11,>=3.8; 2023.7.21 Requires-Python <3.11,>=3.8; 2023.7.28 Requires-Python <3.11,>=3.8; 2023.8.14 Requires-Python <3.11,>=3.8; 2023.8.22 Requires-Python <3.11,>=3.8; 2023.8.31 Requires-Python <3.11,>=3.8; 2023.9.11 Requires-Python <3.11,>=3.8
ERROR: Could not find a version that satisfies the requirement pytype==2022.10.13 (from versions: 2018.5.14, 2018.5.15, 2018.5.22, 2018.5.22.1, 2018.6.5, 2018.6.15, 2018.6.18, 2018.6.19, 2018.7.13, 2018.8.10, 2018.9.7, 2018.9.7.1, 2018.9.18, 2018.9.19, 2018.9.25, 2018.9.26, 2018.10.30, 2018.11.6, 2018.12.11, 2018.12.21, 2019.1.18, 2019.1.30, 2019.2.13, 2019.3.1, 2019.3.8, 2019.3.15, 2019.3.21, 2019.3.27, 2019.4.2, 2019.4.2.1, 2019.4.5, 2019.4.12, 2019.4.19, 2019.4.26, 2019.5.6, 2019.5.8, 2019.5.15, 2019.5.24, 2019.5.31, 2019.6.21, 2019.6.28, 2019.7.11, 2019.7.26, 2019.7.30, 2019.8.9, 2019.8.29, 2019.9.6, 2019.9.17, 2019.10.17, 2019.11.27, 2019.12.6, 2019.12.17, 2020.1.7, 2020.1.8, 2020.1.24, 2020.2.6, 2023.9.19, 2023.9.27, 2023.10.5, 2023.10.17, 2023.10.24, 2023.10.31, 2023.11.21, 2023.11.29, 2023.12.7, 2023.12.8, 2023.12.18, 2024.1.5, 2024.1.24, 2024.2.9, 2024.2.13, 2024.2.27, 2024.3.11, 2024.3.19, 2024.4.11)
ERROR: No matching distribution found for pytype==2022.10.13
DonggeLiu commented 2 months ago

Also, we did not document the exact version of matplotlib used in fuzzbench in requirements.txt.

Related: https://github.com/google/fuzzbench/pull/1982#issuecomment-2120433754

DonggeLiu commented 2 months ago

Adding a solution discovered by @DanBlackwell: https://github.com/google/fuzzbench/pull/1982#issuecomment-2120546061

DanBlackwell commented 2 months ago

Just to add a note here; trying to install matplotlib==2.2.5 may be tricky on newer Linux setups, as the pip wheel tries to use platform.linux_distribution which was removed in python3.8 (according to https://stackoverflow.com/a/64106589)

ChaitanyaYeole02 commented 1 day ago

Hey @DonggeLiu, is this been worked on or can I work on this? Can you please assign this issue to me?

DonggeLiu commented 1 day ago

@gustavogaldinoo: Are you working on this by any chance? If not, would now be a good time for @ChaitanyaYeole02 to help?

ChaitanyaYeole02 commented 1 day ago

@DonggeLiu, if there are any tasks, that you think can be taken by me. Please let me know. I am still a beginner with fuzzbench library.

DanBlackwell commented 1 day ago

Just a note that in the meantime if anyone wants to run generate_report.py locally, the following docker setup worked for me:

# load up a docker image that has python3.10; mount fuzzbench directory inside
docker run -v $(pwd):/tmp/ -it ubuntu:22.04

# now you're in docker, let's install the dependencies
apt update
apt install -y python3.10-dev python3.10-venv build-essential libpq-dev
cd /tmp/
apt install -y pip
pip install pyyaml google-cloud-storage google-cloud-error-reporting
make install-dependencies
source .venv/bin/activate
pip install pandas==1.5.2 clusterfuzz scikit_posthocs Orange3 sqlalchemy jinja2 psycopg2

# now we can run the generate_report.py script, note: --from-cached-data required!
PYTHONPATH=. python3 analysis/generate_report.py $EXPERIMENT_NAME -d $REPORT_DATA_DIR --from-cached-data --benchmarks bloaty_fuzz_target --fuzzers libafl aflplusplus