community.general.apache2_mod_proxy issue with new BeautifulSoup library

[jmsierra]

Summary

It seems that new beautifulsoup library version 4 changes the way the library is imported:

from bs4 import BeautifulSoup

causing errors even if pip3 library beautifulsoup4 is installed when calling module community.general.apache2_mod_proxy, which tries to import it as:

from BeautifulSoup import BeautifulSoup

Many thanks!

Issue Type

Bug Report

Component Name

community.general.apache2_mod_proxy

Ansible Version

$ ansible --version
ansible [core 2.11.1] 
  config file = /home/jmsierra/.ansible.cfg
  configured module search path = ['/home/jmsierra/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jmsierra/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/jmsierra/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/jmsierra/.local/bin/ansible
  python version = 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0]
  jinja version = 2.10.1
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general

# /home/jmsierra/.local/lib/python3.8/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 3.2.0  

# /home/jmsierra/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 4.4.0  

Configuration

$ ansible-config dump --only-changed
HOST_KEY_CHECKING(env: ANSIBLE_HOST_KEY_CHECKING) = False

OS / Environment

Ansible machine: Ubuntu 20 WSL Target machine: Ubuntu 20.04 Minimal

Steps to Reproduce

Installed beautiful soup 4 manually on dc1-frontend machine:

# pip3 install beautifulsoup
Collecting beautifulsoup
  Using cached BeautifulSoup-3.2.2.tar.gz (32 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-w9axyja9/beautifulsoup_88953322c3844e60bb79f27ef1c1efaa/setup.py", line 3
          "You're trying to run a very old release of Beautiful Soup under Python 3. This will not work."<>"Please use Beautiful Soup 4, available through the pip package 'beautifulsoup4'."
                                                                                                          ^
      SyntaxError: invalid syntax
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

# pip3 install beautifulsoup4

# pip3 list | grep soup
beautifulsoup4         4.10.0
soupsieve              2.3.1

Launch ansible module:

  pre_tasks:
    - name: Remove node from balancer
      community.general.apache2_mod_proxy:
        balancer_vhost: 'notification'
        member_host: 'ws://{{ ansible_host }}:15674/ws'
        state: drained
      delegate_to: dc1-frontend1

Expected Results

Expected for the target machine (ansible_host) to be drained from apache2 load balancer.

Actual Results

TASK [Remove node from balancer] ******************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'BeautifulSoup'
fatal: [dc1-sat1 -> dc1-frontend1]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (BeautifulSoup) on dc1-frontend1's Python /usr/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/modules/web_infrastructure/apache2_mod_proxy.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @oboukili click here for bot help

[felixfontein]

BeautifulSoup4 is a different library from BeautifulSoup. The module requires the latter, you seem to have installed the former.

It definitely does make sense to support both BS3 and BS4 though :)

[jmsierra]

Yes, I think BS4 is a new and disruptive version of BeautifulSoup. BS3 is no longer available in Ubuntu20 at least under python 3.8 and it asks you to install BS4 instead:

# pip3 install beautifulsoup
Collecting beautifulsoup
  Using cached BeautifulSoup-3.2.2.tar.gz (32 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-w9axyja9/beautifulsoup_88953322c3844e60bb79f27ef1c1efaa/setup.py", line 3
          "You're trying to run a very old release of Beautiful Soup under Python 3. This will not work."<>"Please use Beautiful Soup 4, available through the pip package 'beautifulsoup4'."
                                                                                                          ^
      SyntaxError: invalid syntax
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Thank you so much for your support!

[ansibullbot]

Files identified in the description:

• plugins/modules/apache2_mod_proxy.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[fwassink]

Please update the apache2_mod_proxy to import BeautifulSoup from bs4 Version 3 is deprecated. See: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#porting-code-to-bs4