SSL Error: Certificate verify failed, expired, _ssl.c:1123

nhohung commented 3 years ago

Hello,

I was trying to fetch some blog articles and ran into an error: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)

The problem seems to be reproducible, for example, with these two following articles: https://nguyenvantuan.info/nguyens-research-group/ https://nguyenvantuan.info/2021/09/28/giai-lasker-cho-2-nguoi-sang-che-vaccine-mrna/

They can be viewed normally on Pocket List, so I don't know what went wrong behind the certification check. I also looked for _ssl.c1123 but couldn't find anything helpful. Only this link suggests the a temporary error, but it doesn't apply to me.

Can you have a look of this? Thanks!

I'm using the latest version 2.7.3. Here is the error log:

Conversion options changed from defaults:
  output_profile: 'kindle_pw3'
  verbose: 2
Resolved conversion options
calibre version: 5.17.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x070F62C8>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_file_type': 'old',
 'mobi_ignore_margins': False,
 'mobi_keep_original_images': False,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindlePaperWhite3Output object at 0x070F65C8>,
 'page_breaks_before': None,
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: custom:1011
Using user agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
Fetching https://nguyenvantuan.info/nguyens-research-group/
Fetching https://nguyenvantuan.info/2021/09/28/giai-lasker-cho-2-nguoi-sang-che-vaccine-mrna/
Could not fetch link https://nguyenvantuan.info/nguyens-research-group/
Traceback (most recent call last):
  File "mechanize\_urllib2_fork.py", line 1229, in do_open
  File "http\client.py", line 1255, in request
  File "http\client.py", line 1301, in _send_request
  File "http\client.py", line 1250, in endheaders
  File "http\client.py", line 1010, in _send_output
  File "http\client.py", line 950, in send
  File "http\client.py", line 1424, in connect
  File "ssl.py", line 500, in wrap_socket
  File "ssl.py", line 1040, in _create
  File "ssl.py", line 1309, in do_handshake
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre\web\fetch\simple.py", line 529, in process_links
  File "calibre\web\fetch\simple.py", line 293, in fetch_url
  File "calibre\web\fetch\simple.py", line 274, in fetch_url
  File "mechanize\_mechanize.py", line 241, in open_novisit
  File "mechanize\_mechanize.py", line 287, in _mech_open
  File "mechanize\_opener.py", line 193, in open
  File "mechanize\_urllib2_fork.py", line 425, in _open
  File "mechanize\_urllib2_fork.py", line 414, in _call_chain
  File "calibre\utils\browser.py", line 29, in https_open
  File "mechanize\_urllib2_fork.py", line 1233, in do_open
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)>

Could not fetch link https://nguyenvantuan.info/2021/09/28/giai-lasker-cho-2-nguoi-sang-che-vaccine-mrna/
https://nguyenvantuan.info/nguyens-research-group/ saved to 
Traceback (most recent call last):
  File "mechanize\_urllib2_fork.py", line 1229, in do_open
  File "http\client.py", line 1255, in request
  File "http\client.py", line 1301, in _send_request
  File "http\client.py", line 1250, in endheaders
  File "http\client.py", line 1010, in _send_output
  File "http\client.py", line 950, in send
  File "http\client.py", line 1424, in connect
  File "ssl.py", line 500, in wrap_socket
  File "ssl.py", line 1040, in _create
  File "ssl.py", line 1309, in do_handshake
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre\web\fetch\simple.py", line 529, in process_links
  File "calibre\web\fetch\simple.py", line 293, in fetch_url
  File "calibre\web\fetch\simple.py", line 274, in fetch_url
  File "mechanize\_mechanize.py", line 241, in open_novisit
  File "mechanize\_mechanize.py", line 287, in _mech_open
  File "mechanize\_opener.py", line 193, in open
  File "mechanize\_urllib2_fork.py", line 425, in _open
  File "mechanize\_urllib2_fork.py", line 414, in _call_chain
  File "calibre\utils\browser.py", line 29, in https_open
  File "mechanize\_urllib2_fork.py", line 1233, in do_open
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)>

https://nguyenvantuan.info/2021/09/28/giai-lasker-cho-2-nguoi-sang-che-vaccine-mrna/ saved to 
Failed to download article: Nguyen’s Research Group from https://nguyenvantuan.info/nguyens-research-group/
Traceback (most recent call last):
  File "calibre\utils\threadpool.py", line 102, in run
  File "calibre\web\feeds\news.py", line 1178, in fetch_article
  File "calibre\web\feeds\news.py", line 1173, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in this log

Failed to download article: Giải Lasker cho 2 người sáng chế vaccine mRNA from https://nguyenvantuan.info/2021/09/28/giai-lasker-cho-2-nguoi-sang-che-vaccine-mrna/
Traceback (most recent call last):
  File "calibre\utils\threadpool.py", line 102, in run
  File "calibre\web\feeds\news.py", line 1178, in fetch_article
  File "calibre\web\feeds\news.py", line 1173, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in this log

Failed to generate default cover
Traceback (most recent call last):
  File "<string>", line 488, in default_cover
ValueError: Invalid format string

Failed to download the following articles:
Nguyen’s Research Group from testtn
https://nguyenvantuan.info/nguyens-research-group/
Traceback (most recent call last):
  File "calibre\utils\threadpool.py", line 102, in run
  File "calibre\web\feeds\news.py", line 1178, in fetch_article
  File "calibre\web\feeds\news.py", line 1173, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in this log

Giải Lasker cho 2 người sáng chế vaccine mRNA from testtn
https://nguyenvantuan.info/2021/09/28/giai-lasker-cho-2-nguoi-sang-che-vaccine-mrna/
Traceback (most recent call last):
  File "calibre\utils\threadpool.py", line 102, in run
  File "calibre\web\feeds\news.py", line 1178, in fetch_article
  File "calibre\web\feeds\news.py", line 1173, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in this log

Parsing all content...
Parsing feed_0/index.html ...
Forcing feed_0/index.html into XHTML namespace
Parsing index.html ...
Forcing index.html into XHTML namespace
Reading TOC from NCX...
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Found 3 items of level: div_1
Found 2 items of level: p_2
Found 1 items of level: div_2
Ignoring level p_2
div_1  left margin stats: Counter()
div_1  right margin stats: Counter()
div_2  left margin stats: Counter()
div_2  right margin stats: Counter()
Cleaning up manifest...
Trimming unused files from manifest...
Creating MOBI Output...
Serializing resources...
Using mastheadImage supplied in manifest...
Creating MOBI 6 output
Applying case-transforming CSS...
Parsing manglecase.css ...
Rasterizing SVG images...
Converting XHTML to Mobipocket markup...
Serializing markup content...
  Compressing markup content...
No TOC, MOBI index not generated
MOBI output written to C:\Users\hung\AppData\Local\Temp\calibre_0vwz4ggu\a21_nqzm_recipe_out.mobi

mmagnus commented 3 years ago

@nhohung for me it's ok

nhohung commented 3 years ago

@nhohung for me it's ok

Hmm, that's strange, but thanks for checking. I'm aware that the Root CA X3 security certificate has changed to ISRG Root X1 after September 30. Nevertheless, I updated my PC today and it still didn't solve the problem. :(

mmagnus commented 3 years ago

@alvaroreig does it work for you?

alvaroreig commented 3 years ago

Hi there,

I just tested it with the two articles @nhohung provided and it worked ok:

4% Article downloaded: Giải Lasker cho 2 người sáng chế vaccine mRNA
Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/web/fetch/simple.py", line 434, in process_images
    img = image_from_data(data)
  File "/usr/lib/calibre/calibre/utils/img.py", line 87, in image_from_data
    raise NotImage('Not a valid image (detected type: {})'.format(q))
calibre.utils.img.NotImage: Not a valid image (detected type: None)
5% Article downloaded: Heuristic Thinking, Choice Modelling and Compromise Effect
(...)
34% Article downloaded: Nguyen’s Research Group
qt5ct: using qt5ct plugin
34% Feeds downloaded to /tmp/calibre_4.99.4_tmp_s9npxova/3bgij96p_plumber/index.html

Both articles appear in the ebook.

Environment:

Recipe 2.7.3
/usr/bin/env python points to Python 2.7.18 (default, Mar  8 2021, 13:02:45)
calibre 4.99.4

Regards

nhohung commented 3 years ago

Thanks @alvaroreig for checking. Again this is odd, still don't know why it doesn't work for me. Apparently articles from other domains are fine, except this domain. My environment doesn't change (python 3.8.5 since August 2020, calibre 5.17 since May 2021). I'll try again on my Mac and let you know. Cheers.

nhohung commented 3 years ago

I've just tested it on my Mac and the articles were fetched normally. This case: python 2.7.16, calibre 3.48. This will need more trials to narrow down the root cause but I don't have much time during the week. Will catch up with you guys later. Thanks again.

mmagnus commented 2 years ago

Yeah, I checked again, for my Mac, both articles worked again:


(py37) [mx] Pocket-Plus-Calibre-Plugin$ git:(master) ✗ ./SendToKindle_Template.sh
rm: /Users/magnus//Documents/My Kindle Content/Pocket.mobi: No such file or directory
Conversion options changed from defaults:
  test: None
1% Converting input to HTML...
InputFormatPlugin: Recipe Input running
Using custom recipe
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36
1% Fetching feeds...
1% Got feeds from index page
1% Trying to download cover...
1% Generating masthead...
34% Masthead image downloaded
1% Starting download [10 threads]...
17% Article downloaded: Giải Lasker cho 2 người sáng chế vaccine mRNA
34% Article downloaded: Nguyen’s Research Group
34% Feeds downloaded to /private/var/folders/yc/ssr9692s5fzf7k165grnhpk80000gp/C/calibre_5.30.0_tmp_8xg3zv7r/l5f83vjt_plumber/index.html
34% Download finished
Parsing all content...
Forcing feed_0/index.html into XHTML namespace
Forcing feed_0/article_0/index.html into XHTML namespace
Forcing index.html into XHTML namespace
Forcing feed_0/article_1/index.html into XHTML namespace
Referenced file 'feed_1/index.html' not found
34% Running transforms on e-book...
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Cleaning up manifest...
Trimming unused files from manifest...
Creating MOBI Output...
67% Running MOBI Output plugin
Serializing resources...
Converting TOC for MOBI periodical indexing...
Creating MOBI 6 output
Generating in-line TOC...
Applying case-transforming CSS...
Rasterizing SVG images...
Converting XHTML to Mobipocket markup...
Serializing markup content...
  Compressing markup content...
Generating MOBI index for a periodical
MOBI output written to /Users/magnus/workspace/Pocket-Plus-Calibre-Plugin/Pocket.mobi
Output saved to   /Users/magnus/workspace/Pocket-Plus-Calibre-Plugin/Pocket.mobi

mmagnus commented 2 years ago

@nhohung I will close it for now, feel free to open it again if needed.

nhohung commented 2 years ago

@nhohung I will close it for now, feel free to open it again if needed. Yes, please. Thank you @mmagnus

mmagnus / Pocket-Plus-Calibre-Plugin

SSL Error: Certificate verify failed, expired, _ssl.c:1123 #45