Command `browse_website` fails on Python 3.10 due to Beautiful Soup 4

[peterbanda]

⚠️ Search for existing issues first ⚠️

• [X] I have searched the existing issues, and there is no existing issue for my problem

Which Operating System are you using?

Linux

GPT-3 or GPT-4?

GPT-4

Steps to reproduce 🕹

Python Version: 3.10.11 Beautiful Soup 4 Version: 4.6.0

python3.10 -m autogpt

Goal can be anything that requires Google search and web browsing.

Current behavior 😯

Site browsing/parsing fails with module 'collections' has no attribute 'Callable'

Expected behavior 🤔

I was able to fix the issue by following: https://stackoverflow.com/questions/69515086/error-attributeerror-collections-has-no-attribute-callable-using-beautifu

In particular, added the following imports to the file web_selenium.py before from bs4 import BeautifulSoup

import collections
import collections.abc

collections.Callable = collections.abc.Callable

I can provided PR for that if needed.

Your prompt 📝

N/A

Your Logs 📒

2023-04-19 10:22:02,603 INFO NEXT ACTION:  COMMAND = browse_website  ARGUMENTS = {'url': 'https://www.futuremarketinsights.com/reports/contract-lifecycle-management-market', 'question': 'What are the key findings of the report?'}
2023-04-19 10:22:31,913 INFO -=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= 
2023-04-19 10:22:39,458 INFO SYSTEM:  Command browse_website returned: Error: module 'collections' has no attribute 'Callable'

[peterbanda]

Hey folks,

I can now confirm the following. The issue appears on Python 3.10 (3.10.11) as well as 3.11 (3.11.3), and on both, the master branch and the last stable release - version 0.2.1.

In all four cases adding the lines I mentioned above fixes the issue 🎉

[peterbanda]

Created a PR for the issue: #2596

[Pwuts]

Is this still broken in branch 0.2.2-test2?

[peterbanda]

Hey @Pwuts,

I've just tested it on 0.2.2-stable (the latest release) and 0.2.2-test and the issue still occurs (tried Python3.10/3.11). Anyway, after applying the proposed fix, the issue disappeared in all the cases.

[Pwuts]

Strange. What's your setup (OS/platform)?

I'm asking because browsing works fine on most setups, except on Apple M1 and ARM-based devices because of missing drivers (#2600 #2219).

[peterbanda]

Hi,

I use Ubuntu/Linux, but actually I just found a better/more elegant solution.

Simply upgrade Beautiful Soup4 to the latest version:

pip install beautifulsoup4==4.12.2

My previous version was 4.6.0.

Tested on Auto-GPT master, 0.2.1, 0.2.2, and 0.2.2-test with Python 3.10/3.11. Worked like charm 🎉!

Will provide a new PR for that.

[peterbanda]

PR #2680 created... ready to roll 🎉

[Pwuts]

Nice, thanks!