An AttributeError is given when scraping for mastodon profile data.
How to reproduce
snscrape mastodon-profile @daphne@indieweb.social
Expected behaviour
Receiving profile data.
Screenshots and recordings
No response
Operating system
Ubuntu 20.04.
Python version: output of python3 --version
python 3.10.12
snscrape version: output of snscrape --version
snscrape 0.7.0.20230622
Scraper
mastodon-profile
How are you using snscrape?
CLI (snscrape ... as a command, e.g. in a terminal)
Backtrace
AttributeError: 'NoneType' object has no attribute 'find_all'
Log output
2023-09-24 19:51:37.739 INFO snscrape.base Retrieving https://indieweb.social/@daphne/with_replies
2023-09-24 19:51:38.048 INFO snscrape.base Retrieved https://indieweb.social/@daphne/with_replies: 200
2023-09-24 19:51:38.105 CRITICAL snscrape._cli Dumped stack and locals to /tmp/snscrape_locals_wvz4dp16
Traceback (most recent call last):
File "/home/alt/.local/bin/snscrape", line 8, in
sys.exit(main())
File "/home/alt/.local/lib/python3.10/site-packages/snscrape/_cli.py", line 323, in main
for i, item in enumerate(scraper.get_items(), start = 1):
File "/home/alt/.local/lib/python3.10/site-packages/snscrape/modules/mastodon.py", line 280, in get_items
yield from self._entries_toitems(soup.find('div', class = 'activity-stream').findall('div', class = 'entry'), r.url)
AttributeError: 'NoneType' object has no attribute 'find_all'
Dump of locals
No response
Additional context
API still works like a charm, so maybe the data structure was changed. But no idea how snscrape actually works regarding Mastodon. Anyhow, keeping the scrapibility of mastodon intact could be become essential.
JustAnotherArchivist
commented
1 year ago
Describe the bug
An AttributeError is given when scraping for mastodon profile data.
How to reproduce
snscrape mastodon-profile @daphne@indieweb.social
Expected behaviour
Receiving profile data.
Screenshots and recordings
No response
Operating system
Ubuntu 20.04.
Python version: output of
python3 --versionpython 3.10.12
snscrape version: output of
snscrape --versionsnscrape 0.7.0.20230622
Scraper
mastodon-profile
How are you using snscrape?
CLI (
snscrape ...as a command, e.g. in a terminal)Backtrace
AttributeError: 'NoneType' object has no attribute 'find_all'
Log output
2023-09-24 19:51:37.739 INFO snscrape.base Retrieving https://indieweb.social/@daphne/with_replies 2023-09-24 19:51:38.048 INFO snscrape.base Retrieved https://indieweb.social/@daphne/with_replies: 200 2023-09-24 19:51:38.105 CRITICAL snscrape._cli Dumped stack and locals to /tmp/snscrape_locals_wvz4dp16 Traceback (most recent call last): File "/home/alt/.local/bin/snscrape", line 8, in
sys.exit(main())
File "/home/alt/.local/lib/python3.10/site-packages/snscrape/_cli.py", line 323, in main
for i, item in enumerate(scraper.get_items(), start = 1):
File "/home/alt/.local/lib/python3.10/site-packages/snscrape/modules/mastodon.py", line 280, in get_items
yield from self._entries_toitems(soup.find('div', class = 'activity-stream').findall('div', class = 'entry'), r.url)
AttributeError: 'NoneType' object has no attribute 'find_all'
Dump of locals
No response
Additional context
API still works like a charm, so maybe the data structure was changed. But no idea how snscrape actually works regarding Mastodon. Anyhow, keeping the scrapibility of mastodon intact could be become essential.