search

kurtmckee / feedparser

Parse feeds in Python
https://feedparser.readthedocs.io
Other
1.99k stars 343 forks source link

[Bug] SSLCertVerificationError for URL with valid certificate. #310

Closed vishalnandagopal closed 1 year ago

vishalnandagopal commented 2 years ago

When I try to fetch https://lobste.rs/t/security.rss, it shows a SSLCertVerificationError even though the site is opening for me on Firefox, Chrome, the terminal (through curl) and I am also able to fetch the feed through the requests library (without disabling certificate verification).

Do you have any idea why an SSL error occurring on feedparser? Does it use a custom certificate library? The site has a valid certificate and I also ran tests from SSL Labs on it (It got an A+ grade).

>>> import feedparser                                                             
>>> feed = feedparser.parse("https://lobste.rs/t/security.rss")                   
>>> print(feed)
{'bozo': True, 'entries': [], 'feed': {}, 'headers': {}, 'bozo_exception': URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)'))}

For now, I am fetching the feed through requests.get(url) and then sending it to feedparser, but I am concerned about the security implications of sending untrusted xml/html to feedparser.

>>> import requests, feedparser
>>> html_text = requests.get("https://lobste.rs/t/security.rss").text
>>> feedparser.parse(html_text)
{'bozo': False, 'entries': [{'title': 'Curated list of KeePass-related projects', 'title_detail': {'type': 'text/plain', 'language': None, 'base': '', 'value': 'Curated list of KeePass-related projects'}, 'links': [{ .....
vishalnandagopal commented 2 years ago

Related to #84 #290 #168

n3s7or commented 2 years ago

I just tested this and in my environment it worked fine:

Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import feedparser
>>> url = 'https://lobste.rs/t/security.rss'
>>> feedparser.parse(url)
{'bozo': False, 'entries': [{'title': 'Making popular Ruby packages more secure', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Making popular Ruby packages more secure'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://blog.rubygems.org/2022/06/13/making-packages-more-secure.html'}], 'link': 'https://blog.rubygems.org/2022/06/13/making-packages-more-secure.html', 'id': 'https://lobste.rs/s/vduzkn', 'guidislink': False, 'authors': [{'name': 'telemachus', 'email': 'telemachus@users.lobste.rs'}], 'author': 'telemachus@users.lobste.rs (telemachus)', 'author_detail': {'name': 'telemachus', 'email': 'telemachus@users.lobste.rs'}, 'published': 'Mon, 13 Jun 2022 16:03:17 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=13, tm_hour=21, tm_min=3, tm_sec=17, tm_wday=0, tm_yday=164, tm_isdst=0), 'comments': 'https://lobste.rs/s/vduzkn/making_popular_ruby_packages_more_secure', 'summary': '<p><a href="https://lobste.rs/s/vduzkn/making_popular_ruby_packages_more_secure">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/vduzkn/making_popular_ruby_packages_more_secure">Comments</a></p>'}, 'tags': [{'term': 'ruby', 'scheme': None, 'label': None}, {'term': 'security', 'scheme': None, 'label': None}]}, {'title': 'Symbiote: A New, Nearly-Impossible-to-Detect Linux Threat', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Symbiote: A New, Nearly-Impossible-to-Detect Linux Threat'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://blogs.blackberry.com/en/2022/06/symbiote-a-new-nearly-impossible-to-detect-linux-threat'}], 'link': 'https://blogs.blackberry.com/en/2022/06/symbiote-a-new-nearly-impossible-to-detect-linux-threat', 'id': 'https://lobste.rs/s/ijottt', 'guidislink': False, 'authors': [{'name': 'Yogthos', 'email': 'Yogthos@users.lobste.rs'}], 'author': 'Yogthos@users.lobste.rs (Yogthos)', 'author_detail': {'name': 'Yogthos', 'email': 'Yogthos@users.lobste.rs'}, 'published': 'Fri, 10 Jun 2022 13:46:46 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=46, tm_sec=46, tm_wday=4, tm_yday=161, tm_isdst=0), 'comments': 'https://lobste.rs/s/ijottt/symbiote_new_nearly_impossible_detect', 'summary': '<p><a href="https://lobste.rs/s/ijottt/symbiote_new_nearly_impossible_detect">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/ijottt/symbiote_new_nearly_impossible_detect">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'linux', 'scheme': None, 'label': None}]}, {'title': 'If OpenSSL were a GUI', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'If OpenSSL were a GUI'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://smallstep.com/blog/if-openssl-were-a-gui/'}], 'link': 'https://smallstep.com/blog/if-openssl-were-a-gui/', 'id': 'https://lobste.rs/s/faymlu', 'guidislink': False, 'authors': [{'name': 'friendlysock', 'email': 'friendlysock@users.lobste.rs'}], 'author': 'friendlysock@users.lobste.rs (friendlysock)', 'author_detail': {'name': 'friendlysock', 'email': 'friendlysock@users.lobste.rs'}, 'published': 'Fri, 10 Jun 2022 13:30:22 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=30, tm_sec=22, tm_wday=4, tm_yday=161, tm_isdst=0), 'comments': 'https://lobste.rs/s/faymlu/if_openssl_were_gui', 'summary': '<p><a href="https://lobste.rs/s/faymlu/if_openssl_were_gui">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/faymlu/if_openssl_were_gui">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'satire', 'scheme': None, 'label': None}]}, {'title': 'PACMAN: a novel hardware attack that can bypass Pointer Authentication (PAC) on the Apple M1 CPU', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'PACMAN: a novel hardware attack that can bypass Pointer Authentication (PAC) on the Apple M1 CPU'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://pacmanattack.com/'}], 'link': 'https://pacmanattack.com/', 'id': 'https://lobste.rs/s/dxhkyn', 'guidislink': False, 'authors': [{'name': 'lattera', 'email': 'lattera@users.lobste.rs'}], 'author': 'lattera@users.lobste.rs (lattera)', 'author_detail': {'name': 'lattera', 'email': 'lattera@users.lobste.rs'}, 'published': 'Fri, 10 Jun 2022 12:10:39 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=10, tm_hour=17, tm_min=10, tm_sec=39, tm_wday=4, tm_yday=161, tm_isdst=0), 'comments': 'https://lobste.rs/s/dxhkyn/pacman_novel_hardware_attack_can_bypass', 'summary': '<p><a href="https://lobste.rs/s/dxhkyn/pacman_novel_hardware_attack_can_bypass">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/dxhkyn/pacman_novel_hardware_attack_can_bypass">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}]}, {'title': 'Private Access Tokens: eliminating CAPTCHAs on iPhones and Macs with open standards', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Private Access Tokens: eliminating CAPTCHAs on iPhones and Macs with open standards'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://blog.cloudflare.com/eliminating-captchas-on-iphones-and-macs-using-new-standard/'}], 'link': 'https://blog.cloudflare.com/eliminating-captchas-on-iphones-and-macs-using-new-standard/', 'id': 'https://lobste.rs/s/2xzc7k', 'guidislink': False, 'authors': [{'name': 'adamshaylor', 'email': 'adamshaylor@users.lobste.rs'}], 'author': 'adamshaylor@users.lobste.rs (adamshaylor)', 'author_detail': {'name': 'adamshaylor', 'email': 'adamshaylor@users.lobste.rs'}, 'published': 'Wed, 08 Jun 2022 22:25:35 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=9, tm_hour=3, tm_min=25, tm_sec=35, tm_wday=3, tm_yday=160, tm_isdst=0), 'comments': 'https://lobste.rs/s/2xzc7k/private_access_tokens_eliminating', 'summary': '<p><a href="https://lobste.rs/s/2xzc7k/private_access_tokens_eliminating">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/2xzc7k/private_access_tokens_eliminating">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'release', 'scheme': None, 'label': None}, {'term': 'web', 'scheme': None, 'label': None}]}, {'title': 'Encrypted Databases: From Theory to Systems', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Encrypted Databases: From Theory to Systems'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.cidrdb.org/cidr2021/papers/cidr2021_paper23.pdf'}], 'link': 'https://www.cidrdb.org/cidr2021/papers/cidr2021_paper23.pdf', 'id': 'https://lobste.rs/s/iunobp', 'guidislink': False, 'authors': [{'name': 'df', 'email': 'df@users.lobste.rs'}], 'author': 'df@users.lobste.rs (df)', 'author_detail': {'name': 'df', 'email': 'df@users.lobste.rs'}, 'published': 'Wed, 08 Jun 2022 12:08:21 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=8, tm_hour=17, tm_min=8, tm_sec=21, tm_wday=2, tm_yday=159, tm_isdst=0), 'comments': 'https://lobste.rs/s/iunobp/encrypted_databases_from_theory_systems', 'summary': '<p><a href="https://lobste.rs/s/iunobp/encrypted_databases_from_theory_systems">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/iunobp/encrypted_databases_from_theory_systems">Comments</a></p>'}, 'tags': [{'term': 'pdf', 'scheme': None, 'label': None}, {'term': 'security', 'scheme': None, 'label': None}, {'term': 'databases', 'scheme': None, 'label': None}, {'term': 'cryptography', 'scheme': None, 'label': None}]}, {'title': 'SFUZZ: High Performance Coverage-guided Greybox Fuzzer with Custom JIT Engine', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'SFUZZ: High Performance Coverage-guided Greybox Fuzzer with Custom JIT Engine'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://seal9055.com/blog/?p=sfuzz&d=fuzzing'}], 'link': 'https://seal9055.com/blog/?p=sfuzz&d=fuzzing', 'id': 'https://lobste.rs/s/anwmfb', 'guidislink': False, 'authors': [{'name': 'jakob', 'email': 'jakob@users.lobste.rs'}], 'author': 'jakob@users.lobste.rs (jakob)', 'author_detail': {'name': 'jakob', 'email': 'jakob@users.lobste.rs'}, 'published': 'Wed, 08 Jun 2022 06:27:23 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=8, tm_hour=11, tm_min=27, tm_sec=23, tm_wday=2, tm_yday=159, tm_isdst=0), 'comments': 'https://lobste.rs/s/anwmfb/sfuzz_high_performance_coverage_guided', 'summary': '<p><a href="https://lobste.rs/s/anwmfb/sfuzz_high_performance_coverage_guided">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/anwmfb/sfuzz_high_performance_coverage_guided">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'assembly', 'scheme': None, 'label': None}]}, {'title': 'dot: The Deepfake Offensive Toolkit', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'dot: The Deepfake Offensive Toolkit'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://github.com/sensity-ai/dot'}], 'link': 'https://github.com/sensity-ai/dot', 'id': 'https://lobste.rs/s/uuhty9', 'guidislink': False, 'authors': [{'name': 'feoh', 'email': 'feoh@users.lobste.rs'}], 'author': 'feoh@users.lobste.rs (feoh)', 'author_detail': {'name': 'feoh', 'email': 'feoh@users.lobste.rs'}, 'published': 'Tue, 07 Jun 2022 09:19:35 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=7, tm_hour=14, tm_min=19, tm_sec=35, tm_wday=1, tm_yday=158, tm_isdst=0), 'comments': 'https://lobste.rs/s/uuhty9/dot_deepfake_offensive_toolkit', 'summary': '<p><a href="https://lobste.rs/s/uuhty9/dot_deepfake_offensive_toolkit">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/uuhty9/dot_deepfake_offensive_toolkit">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'ai', 'scheme': None, 'label': None}]}, {'title': 'A layered approach to content blocking', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'A layered approach to content blocking'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://seirdy.one/posts/2022/06/04/layered-content-blocking/'}], 'link': 'https://seirdy.one/posts/2022/06/04/layered-content-blocking/', 'id': 'https://lobste.rs/s/zg8ssb', 'guidislink': False, 'authors': [{'name': 'Seirdy', 'email': 'Seirdy@users.lobste.rs'}], 'author': 'Seirdy@users.lobste.rs (Seirdy)', 'author_detail': {'name': 'Seirdy', 'email': 'Seirdy@users.lobste.rs'}, 'published': 'Mon, 06 Jun 2022 19:09:58 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=7, tm_hour=0, tm_min=9, tm_sec=58, tm_wday=1, tm_yday=158, tm_isdst=0), 'comments': 'https://lobste.rs/s/zg8ssb/layered_approach_content_blocking', 'summary': '<p><a href="https://lobste.rs/s/zg8ssb/layered_approach_content_blocking">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/zg8ssb/layered_approach_content_blocking">Comments</a></p>'}, 'tags': [{'term': 'browsers', 'scheme': None, 'label': None}, {'term': 'security', 'scheme': None, 'label': None}, {'term': 'a11y', 'scheme': None, 'label': None}]}, {'title': "Exploiting the Wii U's USB Descriptor parsing", 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': "Exploiting the Wii U's USB Descriptor parsing"}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://garyodernichts.blogspot.com/2022/06/exploiting-wii-us-usb-descriptor-parsing.html'}], 'link': 'https://garyodernichts.blogspot.com/2022/06/exploiting-wii-us-usb-descriptor-parsing.html', 'id': 'https://lobste.rs/s/e3q2mo', 'guidislink': False, 'authors': [{'name': 'crazyloglad', 'email': 'crazyloglad@users.lobste.rs'}], 'author': 'crazyloglad@users.lobste.rs (crazyloglad)', 'author_detail': {'name': 'crazyloglad', 'email': 'crazyloglad@users.lobste.rs'}, 'published': 'Mon, 06 Jun 2022 14:52:46 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=6, tm_hour=19, tm_min=52, tm_sec=46, tm_wday=0, tm_yday=157, tm_isdst=0), 'comments': 'https://lobste.rs/s/e3q2mo/exploiting_wii_u_s_usb_descriptor_parsing', 'summary': '<p><a href="https://lobste.rs/s/e3q2mo/exploiting_wii_u_s_usb_descriptor_parsing">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/e3q2mo/exploiting_wii_u_s_usb_descriptor_parsing">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}]}, {'title': 'Deep inside TOMOYO Linux (2009)', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Deep inside TOMOYO Linux (2009)'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://osdn.net/projects/tomoyo/docs/lca2009-kumaneko.pdf'}], 'link': 'https://osdn.net/projects/tomoyo/docs/lca2009-kumaneko.pdf', 'id': 'https://lobste.rs/s/g26kdf', 'guidislink': False, 'authors': [{'name': 'calvin', 'email': 'calvin@users.lobste.rs'}], 'author': 'calvin@users.lobste.rs (calvin)', 'author_detail': {'name': 'calvin', 'email': 'calvin@users.lobste.rs'}, 'published': 'Sun, 05 Jun 2022 11:12:23 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=5, tm_hour=16, tm_min=12, tm_sec=23, tm_wday=6, tm_yday=156, tm_isdst=0), 'comments': 'https://lobste.rs/s/g26kdf/deep_inside_tomoyo_linux_2009', 'summary': '<p><a href="https://lobste.rs/s/g26kdf/deep_inside_tomoyo_linux_2009">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/g26kdf/deep_inside_tomoyo_linux_2009">Comments</a></p>'}, 'tags': [{'term': 'pdf', 'scheme': None, 'label': None}, {'term': 'security', 'scheme': None, 'label': None}, {'term': 'linux', 'scheme': None, 'label': None}]}, {'title': 'An Apple a day keeps the exploiter away', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'An Apple a day keeps the exploiter away'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.sstic.org/media/SSTIC2022/SSTIC-actes/an_apple_a_day/SSTIC2022-Slides-an_apple_a_day-benoist-vanderbeken_perigaud.pdf'}], 'link': 'https://www.sstic.org/media/SSTIC2022/SSTIC-actes/an_apple_a_day/SSTIC2022-Slides-an_apple_a_day-benoist-vanderbeken_perigaud.pdf', 'id': 'https://lobste.rs/s/pc0ssd', 'guidislink': False, 'authors': [{'name': 'fro', 'email': 'fro@users.lobste.rs'}], 'author': 'fro@users.lobste.rs (fro)', 'author_detail': {'name': 'fro', 'email': 'fro@users.lobste.rs'}, 'published': 'Sun, 05 Jun 2022 11:11:43 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=5, tm_hour=16, tm_min=11, tm_sec=43, tm_wday=6, tm_yday=156, tm_isdst=0), 'comments': 'https://lobste.rs/s/pc0ssd/apple_day_keeps_exploiter_away', 'summary': '<p><a href="https://lobste.rs/s/pc0ssd/apple_day_keeps_exploiter_away">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/pc0ssd/apple_day_keeps_exploiter_away">Comments</a></p>'}, 'tags': [{'term': 'pdf', 'scheme': None, 'label': None}, {'term': 'security', 'scheme': None, 'label': None}, {'term': 'ios', 'scheme': None, 'label': None}, {'term': 'slides', 'scheme': None, 'label': None}]}, {'title': 'Boot drive encryption security on Windows', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Boot drive encryption security on Windows'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://threedots.ovh/blog/2022/06/boot-drive-encryption-security-on-windows/'}], 'link': 'https://threedots.ovh/blog/2022/06/boot-drive-encryption-security-on-windows/', 'id': 'https://lobste.rs/s/ikzqnw', 'guidislink': False, 'authors': [{'name': 'never_released', 'email': 'never_released@users.lobste.rs'}], 'author': 'never_released@users.lobste.rs (never_released)', 'author_detail': {'name': 'never_released', 'email': 'never_released@users.lobste.rs'}, 'published': 'Sat, 04 Jun 2022 09:31:40 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=4, tm_hour=14, tm_min=31, tm_sec=40, tm_wday=5, tm_yday=155, tm_isdst=0), 'comments': 'https://lobste.rs/s/ikzqnw/boot_drive_encryption_security_on', 'summary': '<p><a href="https://lobste.rs/s/ikzqnw/boot_drive_encryption_security_on">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/ikzqnw/boot_drive_encryption_security_on">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'windows', 'scheme': None, 'label': None}]}, {'title': 'Apple Silicon “Augury” DMP Vulnerability', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Apple Silicon “Augury” DMP Vulnerability'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://mjtsai.com/blog/2022/06/03/apple-silicon-augury-dmp-vulnerability/'}], 'link': 'https://mjtsai.com/blog/2022/06/03/apple-silicon-augury-dmp-vulnerability/', 'id': 'https://lobste.rs/s/rqpqje', 'guidislink': False, 'authors': [{'name': 'Gaelan', 'email': 'Gaelan@users.lobste.rs'}], 'author': 'Gaelan@users.lobste.rs (Gaelan)', 'author_detail': {'name': 'Gaelan', 'email': 'Gaelan@users.lobste.rs'}, 'published': 'Fri, 03 Jun 2022 11:43:43 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=3, tm_hour=16, tm_min=43, tm_sec=43, tm_wday=4, tm_yday=154, tm_isdst=0), 'comments': 'https://lobste.rs/s/rqpqje/apple_silicon_augury_dmp_vulnerability', 'summary': '<p><a href="https://lobste.rs/s/rqpqje/apple_silicon_augury_dmp_vulnerability">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/rqpqje/apple_silicon_augury_dmp_vulnerability">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'hardware', 'scheme': None, 'label': None}]}, {'title': 'Dangerous Gift', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Dangerous Gift'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.tbray.org/ongoing/When/202x/2022/06/02/Dangerous-Gift'}], 'link': 'https://www.tbray.org/ongoing/When/202x/2022/06/02/Dangerous-Gift', 'id': 'https://lobste.rs/s/zemzol', 'guidislink': False, 'authors': [{'name': 'napkindrawing', 'email': 'napkindrawing@users.lobste.rs'}], 'author': 'napkindrawing@users.lobste.rs (napkindrawing)', 'author_detail': {'name': 'napkindrawing', 'email': 'napkindrawing@users.lobste.rs'}, 'published': 'Fri, 03 Jun 2022 08:42:36 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=3, tm_hour=13, tm_min=42, tm_sec=36, tm_wday=4, tm_yday=154, tm_isdst=0), 'comments': 'https://lobste.rs/s/zemzol/dangerous_gift', 'summary': '<p><a href="https://lobste.rs/s/zemzol/dangerous_gift">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/zemzol/dangerous_gift">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'law', 'scheme': None, 'label': None}]}, {'title': 'Zero-Day Exploitation of Atlassian Confluence', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Zero-Day Exploitation of Atlassian Confluence'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.volexity.com/blog/2022/06/02/zero-day-exploitation-of-atlassian-confluence/'}], 'link': 'https://www.volexity.com/blog/2022/06/02/zero-day-exploitation-of-atlassian-confluence/', 'id': 'https://lobste.rs/s/fhsz9q', 'guidislink': False, 'authors': [{'name': 'dijit', 'email': 'dijit@users.lobste.rs'}], 'author': 'dijit@users.lobste.rs (dijit)', 'author_detail': {'name': 'dijit', 'email': 'dijit@users.lobste.rs'}, 'published': 'Thu, 02 Jun 2022 23:17:06 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=3, tm_hour=4, tm_min=17, tm_sec=6, tm_wday=4, tm_yday=154, tm_isdst=0), 'comments': 'https://lobste.rs/s/fhsz9q/zero_day_exploitation_atlassian', 'summary': '<p><a href="https://lobste.rs/s/fhsz9q/zero_day_exploitation_atlassian">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/fhsz9q/zero_day_exploitation_atlassian">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}]}, {'title': 'Use compiler flags for stack protection in GCC and Clang', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Use compiler flags for stack protection in GCC and Clang'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://developers.redhat.com/articles/2022/06/02/use-compiler-flags-stack-protection-gcc-and-clang'}], 'link': 'https://developers.redhat.com/articles/2022/06/02/use-compiler-flags-stack-protection-gcc-and-clang', 'id': 'https://lobste.rs/s/malqbu', 'guidislink': False, 'authors': [{'name': 'fcambus', 'email': 'fcambus@users.lobste.rs'}], 'author': 'fcambus@users.lobste.rs (fcambus)', 'author_detail': {'name': 'fcambus', 'email': 'fcambus@users.lobste.rs'}, 'published': 'Thu, 02 Jun 2022 16:35:24 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=6, tm_mday=2, tm_hour=21, tm_min=35, tm_sec=24, tm_wday=3, tm_yday=153, tm_isdst=0), 'comments': 'https://lobste.rs/s/malqbu/use_compiler_flags_for_stack_protection', 'summary': '<p><a href="https://lobste.rs/s/malqbu/use_compiler_flags_for_stack_protection">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/malqbu/use_compiler_flags_for_stack_protection">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'compilers', 'scheme': None, 'label': None}]}, {'title': 'Curated list of KeePass-related projects', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Curated list of KeePass-related projects'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://github.com/lgg/awesome-keepass'}], 'link': 'https://github.com/lgg/awesome-keepass', 'id': 'https://lobste.rs/s/vkcw9u', 'guidislink': False, 'authors': [{'name': 'atoponce', 'email': 'atoponce@users.lobste.rs'}], 'author': 'atoponce@users.lobste.rs (atoponce)', 'author_detail': {'name': 'atoponce', 'email': 'atoponce@users.lobste.rs'}, 'published': 'Tue, 31 May 2022 16:40:17 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=5, tm_mday=31, tm_hour=21, tm_min=40, tm_sec=17, tm_wday=1, tm_yday=151, tm_isdst=0), 'comments': 'https://lobste.rs/s/vkcw9u/curated_list_keepass_related_projects', 'summary': '<p><a href="https://lobste.rs/s/vkcw9u/curated_list_keepass_related_projects">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/vkcw9u/curated_list_keepass_related_projects">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}]}, {'title': 'Microsoft Office RCE - “Follina” MSDT Attack', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Microsoft Office RCE - “Follina” MSDT Attack'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.huntress.com/blog/microsoft-office-remote-code-execution-follina-msdt-bug'}], 'link': 'https://www.huntress.com/blog/microsoft-office-remote-code-execution-follina-msdt-bug', 'id': 'https://lobste.rs/s/twpfqm', 'guidislink': False, 'authors': [{'name': 'JulianWgs', 'email': 'JulianWgs@users.lobste.rs'}], 'author': 'JulianWgs@users.lobste.rs (JulianWgs)', 'author_detail': {'name': 'JulianWgs', 'email': 'JulianWgs@users.lobste.rs'}, 'published': 'Tue, 31 May 2022 13:17:37 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=5, tm_mday=31, tm_hour=18, tm_min=17, tm_sec=37, tm_wday=1, tm_yday=151, tm_isdst=0), 'comments': 'https://lobste.rs/s/twpfqm/microsoft_office_rce_follina_msdt_attack', 'summary': '<p><a href="https://lobste.rs/s/twpfqm/microsoft_office_rce_follina_msdt_attack">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/twpfqm/microsoft_office_rce_follina_msdt_attack">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}]}, {'title': 'Certificate Transparency data is used to compromise WordPress before installation', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Certificate Transparency data is used to compromise WordPress before installation'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.feistyduck.com/bulletproof-tls-newsletter/issue_89_certificate_transparency_data_is_used_to_compromise_wordpress_before_installation'}], 'link': 'https://www.feistyduck.com/bulletproof-tls-newsletter/issue_89_certificate_transparency_data_is_used_to_compromise_wordpress_before_installation', 'id': 'https://lobste.rs/s/hb5tdl', 'guidislink': False, 'authors': [{'name': 'hanno', 'email': 'hanno@users.lobste.rs'}], 'author': 'hanno@users.lobste.rs (hanno)', 'author_detail': {'name': 'hanno', 'email': 'hanno@users.lobste.rs'}, 'published': 'Tue, 31 May 2022 07:19:21 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=5, tm_mday=31, tm_hour=12, tm_min=19, tm_sec=21, tm_wday=1, tm_yday=151, tm_isdst=0), 'comments': 'https://lobste.rs/s/hb5tdl/certificate_transparency_data_is_used', 'summary': '<p><a href="https://lobste.rs/s/hb5tdl/certificate_transparency_data_is_used">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/hb5tdl/certificate_transparency_data_is_used">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}, {'term': 'web', 'scheme': None, 'label': None}, {'term': 'cryptography', 'scheme': None, 'label': None}]}, {'title': "Using Verizon's API to get a vanity phone number with 4 consecutive digits", 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': "Using Verizon's API to get a vanity phone number with 4 consecutive digits"}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://blog.jonlu.ca/posts/verizon-rare-numbers?ref=ls'}], 'link': 'https://blog.jonlu.ca/posts/verizon-rare-numbers?ref=ls', 'id': 'https://lobste.rs/s/ivakz5', 'guidislink': False, 'authors': [{'name': 'JonLuca', 'email': 'JonLuca@users.lobste.rs'}], 'author': 'JonLuca@users.lobste.rs (JonLuca)', 'author_detail': {'name': 'JonLuca', 'email': 'JonLuca@users.lobste.rs'}, 'published': 'Mon, 30 May 2022 09:26:24 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=5, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=24, tm_wday=0, tm_yday=150, tm_isdst=0), 'comments': 'https://lobste.rs/s/ivakz5/using_verizon_s_api_get_vanity_phone', 'summary': '<p><a href="https://lobste.rs/s/ivakz5/using_verizon_s_api_get_vanity_phone">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/ivakz5/using_verizon_s_api_get_vanity_phone">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}]}, {'title': 'Set Up HTTPS by Default in Your Browser', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Set Up HTTPS by Default in Your Browser'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.eff.org/https-everywhere/set-https-default-your-browser'}], 'link': 'https://www.eff.org/https-everywhere/set-https-default-your-browser', 'id': 'https://lobste.rs/s/splkqh', 'guidislink': False, 'authors': [{'name': 'jparise', 'email': 'jparise@users.lobste.rs'}], 'author': 'jparise@users.lobste.rs (jparise)', 'author_detail': {'name': 'jparise', 'email': 'jparise@users.lobste.rs'}, 'published': 'Sun, 29 May 2022 20:32:08 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=5, tm_mday=30, tm_hour=1, tm_min=32, tm_sec=8, tm_wday=0, tm_yday=150, tm_isdst=0), 'comments': 'https://lobste.rs/s/splkqh/set_up_https_by_default_your_browser', 'summary': '<p><a href="https://lobste.rs/s/splkqh/set_up_https_by_default_your_browser">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/splkqh/set_up_https_by_default_your_browser">Comments</a></p>'}, 'tags': [{'term': 'browsers', 'scheme': None, 'label': None}, {'term': 'security', 'scheme': None, 'label': None}, {'term': 'privacy', 'scheme': None, 'label': None}]}, {'title': 'ssenv: Unset all known api token environment variables', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'ssenv: Unset all known api token environment variables'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://github.com/abbbi/ssenv'}], 'link': 'https://github.com/abbbi/ssenv', 'id': 'https://lobste.rs/s/8gx4aw', 'guidislink': False, 'authors': [{'name': 'abi', 'email': 'abi@users.lobste.rs'}], 'author': 'abi@users.lobste.rs (abi)', 'author_detail': {'name': 'abi', 'email': 'abi@users.lobste.rs'}, 'published': 'Sat, 28 May 2022 17:34:35 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=5, tm_mday=28, tm_hour=22, tm_min=34, tm_sec=35, tm_wday=5, tm_yday=148, tm_isdst=0), 'comments': 'https://lobste.rs/s/8gx4aw/ssenv_unset_all_known_api_token', 'summary': '<p><a href="https://lobste.rs/s/8gx4aw/ssenv_unset_all_known_api_token">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/8gx4aw/ssenv_unset_all_known_api_token">Comments</a></p>'}, 'tags': [{'term': 'security', 'scheme': None, 'label': None}]}, {'title': 'PACMAN: Attacking ARM Pointer Authentication with Speculative Execution', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'PACMAN: Attacking ARM Pointer Authentication with Speculative Execution'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://people.csail.mit.edu/weontaek/pubs/PACMAN_ISCA22.pdf'}], 'link': 'https://people.csail.mit.edu/weontaek/pubs/PACMAN_ISCA22.pdf', 'id': 'https://lobste.rs/s/0l57b2', 'guidislink': False, 'authors': [{'name': 'lattera', 'email': 'lattera@users.lobste.rs'}], 'author': 'lattera@users.lobste.rs (lattera)', 'author_detail': {'name': 'lattera', 'email': 'lattera@users.lobste.rs'}, 'published': 'Sat, 28 May 2022 08:02:38 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=5, tm_mday=28, tm_hour=13, tm_min=2, tm_sec=38, tm_wday=5, tm_yday=148, tm_isdst=0), 'comments': 'https://lobste.rs/s/0l57b2/pacman_attacking_arm_pointer', 'summary': '<p><a href="https://lobste.rs/s/0l57b2/pacman_attacking_arm_pointer">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/0l57b2/pacman_attacking_arm_pointer">Comments</a></p>'}, 'tags': [{'term': 'pdf', 'scheme': None, 'label': None}, {'term': 'security', 'scheme': None, 'label': None}]}, {'title': 'Ghost in the Shellcode', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Ghost in the Shellcode'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.notcheckmark.com/2022/05/ghost-in-the-shellcode/'}], 'link': 'https://www.notcheckmark.com/2022/05/ghost-in-the-shellcode/', 'id': 'https://lobste.rs/s/ylrjgq', 'guidislink': False, 'authors': [{'name': 'withzombies', 'email': 'withzombies@users.lobste.rs'}], 'author': 'withzombies@users.lobste.rs (withzombies)', 'author_detail': {'name': 'withzombies', 'email': 'withzombies@users.lobste.rs'}, 'published': 'Fri, 27 May 2022 14:26:17 -0500', 'published_parsed': time.struct_time(tm_year=2022, tm_mon=5, tm_mday=27, tm_hour=19, tm_min=26, tm_sec=17, tm_wday=4, tm_yday=147, tm_isdst=0), 'comments': 'https://lobste.rs/s/ylrjgq/ghost_shellcode', 'summary': '<p><a href="https://lobste.rs/s/ylrjgq/ghost_shellcode">Comments</a></p>', 'summary_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': '<p><a href="https://lobste.rs/s/ylrjgq/ghost_shellcode">Comments</a></p>'}, 'tags': [{'term': 'event', 'scheme': None, 'label': None}, {'term': 'security', 'scheme': None, 'label': None}, {'term': 'historical', 'scheme': None, 'label': None}]}], 'feed': {'title': 'Lobsters: security - Netsec, appsec, and infosec', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'Lobsters: security - Netsec, appsec, and infosec'}, 'subtitle': 'security - Netsec, appsec, and infosec', 'subtitle_detail': {'type': 'text/html', 'language': None, 'base': 'https://lobste.rs/t/security.rss', 'value': 'security - Netsec, appsec, and infosec'}, 'links': [{'rel': 'alternate', 'type': 'text/html', 'href': 'https://lobste.rs/'}, {'href': 'https://lobste.rs/t/security.rss', 'rel': 'self', 'type': 'application/rss+xml'}], 'link': 'https://lobste.rs/'}, 'headers': {'server': 'nginx', 'date': 'Mon, 13 Jun 2022 22:47:56 GMT', 'content-type': 'application/rss+xml; charset=utf-8', 'transfer-encoding': 'chunked', 'connection': 'close', 'x-frame-options': 'SAMEORIGIN', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'x-download-options': 'noopen', 'x-permitted-cross-domain-policies': 'none', 'referrer-policy': 'strict-origin-when-cross-origin', 'etag': 'W/"4ecfeb39211c45f339a438a1268c0879"', 'cache-control': 'max-age=0, private, must-revalidate', 'content-security-policy-report-only': "default-src 'none'; connect-src 'self'; font-src 'self' https: data:; img-src * data:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; form-action 'self'; report-uri /csp-violation-report", 'x-request-id': 'ca1d78d0-7028-4a05-82fd-0b645aa50ecf', 'x-runtime': '0.068228', 'strict-transport-security': 'max-age=63072000; includeSubDomains'}, 'etag': 'W/"4ecfeb39211c45f339a438a1268c0879"', 'href': 'https://lobste.rs/t/security.rss', 'status': 200, 'encoding': 'utf-8', 'version': 'rss20', 'namespaces': {'': 'http://www.w3.org/2005/Atom'}}
iCiccio commented 2 years ago

I think that it is related also to #281, this is why it works for @n3s7or. I have the same issue that does not happen when I launch the script with an IDE or within the python interpreter, but it raises the error when I call the script from the terminal.

iCiccio commented 2 years ago

I think that it is related also to #281, this is why it works for @n3s7or. I have the same issue that does not happen when I launch the script with an IDE or within the python interpreter, but it raises the error when I call the script from the terminal.

I think that I have identified the problem. It depends on the amount of active/opened connections by your script. You can handle this issue in two possible ways. The first method (terminal):

$ ulimit -n 1048576
$ python your_script.py

The second method (in the script, I put it in the main file)

import resource
resource.setrlimit(resource.RLIMIT_OFILE, (1048576, 1048576))

where the number 1048576 is the integer that I choose.

I hope it will be useful

vishalnandagopal commented 2 years ago

It depends on the amount of active/opened connections by your script. You can handle this issue in two possible ways.

But in the code I used (in the 1st comment) shouldn't the number of open connections be one since it is fetching only one URL?

kurtmckee commented 1 year ago

feedparser is migrating off custom HTTP client code and will rely exclusively on the requests package.

The new code has not been released yet, but because requests will be solely responsible for handling HTTP requests, I'm closing this as no longer relevant.