"No module named pkg_resources" on Python 3.12 due to missing "setuptools"

[jiangzhenjerry]

Describe the bug Running 0.45.24 on Python 3.12. It will not be able to work because "setuptools" is missing

Version ChangeDetection 0.45.24. Python 3.12.4

To Reproduce

Steps to reproduce the behavior:

1. Install Python 3.12.4
2. Install ChangeDetection 0.45.24
3. Run ChangeDetection

Expected behavior It should work fine

Actual behavior For every URL it will fail and show "No module named pkg_resources"

Solution pip install setuptools and it will work.

[dgtlmoon]

how did you install changedetection?

[dgtlmoon]

can you past the WHOLE output of the errors? i think theres some info missing here

[jiangzhenjerry]

I installed by pip install changedetection.io.

I want to clarify that the installation is totally fine - Nothing wrong happens during installation. The error only happens when it actually tries to detect changes for a URL (or any URL). Every URL ends up with "No module named pkg_resources" error, and that's the only error message from the console.

What I was trying to say is that setuptools should be included in requirements.txt. Once I installed setuptools everything works fine.

[dgtlmoon]

can you paste the logs?

[jiangzhenjerry]

2024-06-17 14:00:34.927 | INFO     | changedetectionio.store:sync_to_json:420 - Saving JSON..
(18328) wsgi starting up on http://0.0.0.0:5000
2024-06-17 14:00:34.952 | DEBUG    | changedetectionio.content_fetchers:<module>:41 - Falling back to selenium as fetcher
2024-06-17 14:00:34.955 | DEBUG    | changedetectionio.flask_app:ticker_thread_check_time_launch_checks:1672 - System env MINIMUM_SECONDS_RECHECK_TIME 3
2024-06-17 14:00:34.958 | DEBUG    | changedetectionio.flask_app:ticker_thread_check_time_launch_checks:1759 - > Queued watch UUID 5cdf23d3-a4dc-446a-bd10-7a7e87f8218f last checked at 0 queued at 1718658034.96 priority 1718658034 jitter 0.00s, 1718658034.96s since last checked
2024-06-17 14:00:34.959 | DEBUG    | changedetectionio.flask_app:ticker_thread_check_time_launch_checks:1759 - > Queued watch UUID 5ce8daa7-2781-4b52-ac87-07018205e884 last checked at 0 queued at 1718658034.96 priority 1718658034 jitter 0.00s, 1718658034.96s since last checked
2024-06-17 14:00:35.966 | INFO     | changedetectionio.update_worker:run:248 - Processing watch UUID 5cdf23d3-a4dc-446a-bd10-7a7e87f8218f Priority 1718658034 URL https://news.ycombinator.com/
2024-06-17 14:00:35.967 | INFO     | changedetectionio.update_worker:run:248 - Processing watch UUID 5ce8daa7-2781-4b52-ac87-07018205e884 Priority 1718658034 URL https://changedetection.io/CHANGELOG.txt
2024-06-17 14:00:35.970 | ERROR    | changedetectionio.update_worker:run:462 - Exception reached processing watch UUID: 5cdf23d3-a4dc-446a-bd10-7a7e87f8218f
2024-06-17 14:00:35.971 | ERROR    | changedetectionio.update_worker:run:463 - No module named 'pkg_resources'
2024-06-17 14:00:35.972 | DEBUG    | changedetectionio.update_worker:run:540 - Watch 5cdf23d3-a4dc-446a-bd10-7a7e87f8218f done in 0.00s
2024-06-17 14:00:35.973 | ERROR    | changedetectionio.update_worker:run:462 - Exception reached processing watch UUID: 5ce8daa7-2781-4b52-ac87-07018205e884
2024-06-17 14:00:35.973 | ERROR    | changedetectionio.update_worker:run:463 - No module named 'pkg_resources'
2024-06-17 14:00:35.974 | DEBUG    | changedetectionio.update_worker:run:540 - Watch 5ce8daa7-2781-4b52-ac87-07018205e884 done in 0.01s

[dgtlmoon]

and this is for regular plaintext fetch? no playwright etc?

[dgtlmoon]

Ahh i see in your logs (this is why i asked)

2024-06-17 14:00:34.952 | DEBUG    | changedetectionio.content_fetchers:<module>:41 - Falling back to selenium as fetcher

so you using selenium right?

[jiangzhenjerry]

My log above is generated from the default setting with default sample URLs. Just as if a user starts changedetection for the first time on their computer. This Falling back to selenium as fetcher has existed for a long time even in previous versions (I don't know why) but it's harmless and irrelevant to this bug.

[jiangzhenjerry]

Just so you know that setuptools is included in Python <3.11 but NOT included in Python 3.12. It is just a difference between 3.11 and 3.12 that users need to be aware of. Therefore, to run changedetection on 3.12 the users have to manually install setuptools. It is better if you include it in requirements.txt.

For more information you can Googel setuptools Python 3.12.

[dgtlmoon]

Mainly, I want to find out why exactly, not just blindly add a dependency to solve an error

The tests for the docker python 3.12 container pass without problem

For more information you can Googel setuptools Python 3.12.

Already did, nothing really jumped out

[dgtlmoon]

Ok in the 3.12 build test

https://github.com/dgtlmoon/changedetection.io/actions/runs/9546859589/job/26310613221#step:4:1090

setuptools is listed there, so i guess it must have come from the docker container

$ docker run python:3.12-slim-bullseye bash -c 'pip3 list'
Package    Version
---------- -------
pip        24.0
setuptools 70.0.0
wheel      0.43.0

[dgtlmoon]

it's good to know exactly why, instead of just adding a requirements.txt entry to solve something blindly.

I can reproduce this is in all versions 3.10/3.11 and 3.12

It's actually due to this line

https://github.com/dgtlmoon/changedetection.io/blob/a9ed113369746d0da23926033c0348c6edce3911/changedetectionio/content_fetchers/base.py#L67-L70

I would prefer to probably replace this call with something else

[dgtlmoon]

https://setuptools.pypa.io/en/latest/pkg_resources.html

Attention

Use of pkg_resources is deprecated in favor of importlib.resources, importlib.metadata and their backports (importlib_resources, importlib_metadata). Some useful APIs are also provided by packaging (e.g. requirements and version parsing). Users should refrain from new usage of pkg_resources and should work to port to importlib-based solutions.

[jiangzhenjerry]

I am glad you found the root cause! thanks!

[dgtlmoon]

thanks, it's best to understand the cause and fix things by removing, not by adding :-)