Curl error 16 (CURLE_HTTP2)

Issue

I rather often encounter

Error downloading (curl) 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/noarch/python-rpm-generators-20210628.eccf3f2-1.1.noarch.rpm':
Error code: 'Curl error 16'
Error message: ''

on a wired stable internet connection on openSUSE TW when executing zypper refresh && zypper dup.

Possible solutions

This can be easily resolved by pressing "w" for "retry". Unfortunately, this needs manual observation of the downloads -- so not a good solution.

There could be two possible solutions at the first glance:

automatic retry
resolve the root course for these random error 16 errors.

About the error

CURLE_HTTP2 (16) A problem was detected in the HTTP2 framing layer. This is somewhat generic and can be one out of several problems, see the error buffer for details.

Source: https://curl.se/libcurl/c/libcurl-errors.html

Next time I will try to add the --verbose flag in order to hopefully get more information.

The only way to figure out more is by setting ZYPP_MEDIA_CURL_DEBUG=2 in the environment and inspecting the /var/log/zypper.log file after running zypper. If you have zypper-log installed you can use the command zypper log to list the last few runs and zypper log <PID> to only get the output of a selected run.

If you could provide that file after running into the same problem again we can look into it a bit more closely.

As requested, using the environment varaible ZYPP_MEDIA_CURL_DEBUG=2, here's the part of the log for the package gcc11-11.1.1+git400-1.4.x86_64.rpm which caused an issue this time:

2021-07-15 18:30:41 <1> host(6165) [zypp] PackageProvider.cc(providePackage):412 provide Package (7059)gcc11-11.1.1+git400-1.4.x86_64(https-download.opensuse.org-560f2b9f)
2021-07-15 18:30:41 <1> host(6165) [zypp++] DeltaCandidates.cc(deltaRpms):82 package: (7059)gcc11-11.1.1+git400-1.4.x86_64(https-download.opensuse.org-560f2b9f)
2021-07-15 18:30:41 <1> host(6165) [zypp] RepoProvideFile.cc(provideFile):241 [1]./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm{31,2 MiB|sha256-25ab6bd9a375d4085c953c6eebe612fd3376ae8aac7a3f9c5f382e35bc023441}
2021-07-15 18:30:41 <1> host(6165) [zypp:fetcher++] Fetcher.cc(addCachePath):316 Adding fetcher cache: '/var/cache/zypp/packages/https-download.opensuse.org-560f2b9f'.
2021-07-15 18:30:41 <1> host(6165) [zypp] RepoProvideFile.cc(provideFile):260 Added cache path /var/cache/zypp/packages/https-download.opensuse.org-560f2b9f
2021-07-15 18:30:41 <1> host(6165) [zypp] RepoProvideFile.cc(provideFile):294 Providing file of repo 'https-download.opensuse.org-560f2b9f' from https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/
2021-07-15 18:30:41 <1> host(6165) [zypp:fetcher] Fetcher.cc(downloadAndReadIndexList):690 No indexes to read.
2021-07-15 18:30:41 <1> host(6165) [zypp:fetcher] Fetcher.cc(locateInCache):347 start fetcher with 1 cache directories.
2021-07-15 18:30:41 <1> host(6165) [zypp:fetcher] Fetcher.cc(provideToDest):518 Not found in cache, retrieving...
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaSetAccess.cc(provide):255 Going to try to provide  file ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm from media number 1
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(1): desired (cached)
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(1): desired (cached)
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1335 dest: /var/tmp/AP_0xsG3fgI/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1336 temp: /var/tmp/AP_0xsG3fgI/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm.new.zypp.Vheb2M
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaCurl.cc(doGetFileCopyFile):1132 ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaCurl.cc(doGetFileCopyFile):1142 URL: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:30:41 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to location: https://download.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:30:56 <3> host(6165) [zypp] MediaCurl.cc(doGetFileCopyFile):1207 curl error: 16: , temp file size 0 bytes.
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaCurl.cc(evaluateCurlCode):823 THROW:    Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaCurl.cc(doGetFileCopyFile):1223 RETHROW:  Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaMultiCurl.cc(doGetFileCopy):1364 RETHROW:  Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <1> host(6165) [zypp] PathInfo.cc(unlink):668 unlink /var/tmp/AP_0xsG3fgI/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm.new.zypp.Vheb2M
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaCurl.cc(getFileCopy):652 RETHROW:  Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaSetAccess.cc(provide):268 CAUGHT:   Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(1): desired (cached)
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaHandler.cc(getDetectedDevices):1397 No devices for this medium
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaSetAccess.cc(provide):280 Media couldn't provide file ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm , releasing.
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaManager.cc(release):447 release(id=1)
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaHandler.cc(release):750 Request to release attached media https<https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/>, use count=1
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaHandler.cc(release):757 Releasing media https<https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/>
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaHandler.cc(disconnect):729 Disconnected: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ attached; localRoot "/var/tmp/AP_0xsG3fgI"
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:30:56 <1> host(6165) [zypp] PathInfo.cc(recursive_rmdir):435 recursive_rmdir /var/tmp/AP_0xsG3fgI
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaHandler.cc(removeAttachPoint):192 Deleted default attach point /var/tmp/AP_0xsG3fgI
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaHandler.cc(release):811 Released: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ not attached; localRoot ""
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaManager.cc(releaseAll):484 Releasing all attached media
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaManager.cc(releaseAll):502 Media id 1 not attached
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaManager.cc(releaseAll):512 Exit
2021-07-15 18:30:56 <1> host(6165) [zypper++] media.cc(requestMedia):203 medium problem, url: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/, error 3, label 'openSUSE:Tumbleweed', #1

and after retrying it worked

2021-07-15 18:36:40 <1> host(6165) [zypper] prompt.cc(get_prompt_reply):436 reply: w (1)
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaSetAccess.cc(provide):335 ProvideFile exception caught, callback answer: 1
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaSetAccess.cc(provide):368 Going to try again
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(close):288 Close to access handler using id 1 requested
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaAccess.cc(close):247 Close: https(https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ not attached; localRoot "") (OK)
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaAccess.cc(open):117 Trying scheme 'https'
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/, )
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaMultiCurl.cc(MediaMultiCurl):1165 MediaMultiCurl::MediaMultiCurl(https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/, )
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaAccess.cc(open):196 Opened: https(https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ not attached; localRoot "")
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(open):258 Opened new media access using id 2 to https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaSetAccess.cc(provide):255 Going to try to provide  file ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm from media number 1
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(attach):376 attach(id=2)
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaHandler.cc(createAttachPoint):390 Create attach point: attach root is not a writable directory: '/var/adm/mount'
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaHandler.cc(createAttachPoint):373 Created default attach point /var/tmp/AP_0x8i7wPp
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:40 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaHandler.cc(attach):690 Attached: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ attached; localRoot "/var/tmp/AP_0x8i7wPp"
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):96 checkDesired(2): desired (report by zypp::media::NoVerifier)
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(2): desired (cached)
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(2): desired (cached)
2021-07-15 18:36:40 <1> host(6165) [zypp] PathInfo.cc(assert_dir):354 mkdir /var/tmp/AP_0x8i7wPp/x86_64 00755
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1335 dest: /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1336 temp: /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm.new.zypp.07Wgg8
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaCurl.cc(doGetFileCopyFile):1132 ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaCurl.cc(doGetFileCopyFile):1142 URL: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:40 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to location: https://download.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1374 HTTP response: 200
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://mirror.de.leaseweb.net/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://mirror1.hs-esslingen.de/pub/Mirrors/ftp.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(https://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://ftp.uni-erlangen.de/pub/mirrors/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://widehat.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:41 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:44 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:44 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:46 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:46 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:48 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:48 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:49 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:49 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:51 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:51 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:53 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:53 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:55 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:55 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:56 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:56 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:57 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:57 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:58 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:58 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:59 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:59 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:00 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:00 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:01 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:01 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:01 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:01 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:02 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:02 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:03 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:03 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:04 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:04 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:05 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:05 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:06 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:06 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:07 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:07 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1150 overall result
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #0: state: 5 received: 7504857 url: http://mirror.de.leaseweb.net/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #1: state: 5 received: 6291456 url: http://mirror1.hs-esslingen.de/pub/Mirrors/ftp.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #2: state: 3 received: 6995528 url: https://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #3: state: 4 received: 5505024 url: http://ftp.uni-erlangen.de/pub/mirrors/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #4: state: 5 received: 6718425 url: http://widehat.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp] PathInfo.cc(rename):710 rename /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm.new.zypp.07Wgg8 -> /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1504 done: /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm{- 0644 0/0 size 32670681}
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(provideFile):1014 provideFile(./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm)
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(2): desired (cached)
2021-07-15 18:37:08 <1> host(6165) [zypp:fetcher] Fetcher.cc(validate):366 Checking job [/var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm] (2 checkers )
2021-07-15 18:37:08 <1> host(6165) [zypp] PathInfo.cc(hardlinkCopy):878 hardlinkCopy /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm -> /var/cache/zypp/packages/https-download.opensuse.org-560f2b9f/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaSetAccess.cc(releaseFile):88 Going to release file ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm from media number 1
2021-07-15 18:37:08 <1> host(6165) [zypp] PathInfo.cc(unlink):668 unlink /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [Progress++] ProgressData.cc(report):88 {#63|}END
2021-07-15 18:37:08 <1> host(6165) [zypp] RepoProvideFile.cc(provideFile):307 provideFile at /var/cache/zypp/packages/https-download.opensuse.org-560f2b9f/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [zypp] PackageProvider.cc(providePackage):513 provided Package (7059)gcc11-11.1.1+git400-1.4.x86_64(https-download.opensuse.org-560f2b9f) at /var/cache/zypp/packages/https-download.opensuse.org-560f2b9f/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm

You may be experiencing this issue alone, because almost nobody uses download.opensuse.org with https nowadays. (Or am I wrong?) I tend to think that 'error 16' it is not a problem with libzypp, because it never happened to me when using https://mirrorcache.opensuse.org in .repo files. (and I use it a lot, locally and in CI) Today I configured Apache on download.opensuse.org to redirect all https requests to mirrorcache.opensuse.org and saw 'error 16' once. The same happened to my colleague in another environment. (I haven't found any traces of the error in Apache logs - nothing is logged so it wasn't a problem with occasional broken mirror. The same with zypper.log on client - no useful info was there, -vvv to zypper also wasn't helpful).

BUT, I think libzypp should provide a way to automatically retry such error when using https.

Side note: that error may be related to haproxy+apache as fedora is struggling with similar issue as well: https://bugzilla.redhat.com/show_bug.cgi?id=1690971

I haven't experienced Curl error 16 for several weeks using zypper with https, while before I did see it regularly. I think it was fixed somehow.

Me neither. Shall we close it and reopen if it happens again?

Just happened again:

Abrufen: kpat-21.08.3-1.1.x86_64.rpm .................................................................................................................................................................................................................................[Fehler]
Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/kpat-21.08.3-1.1.x86_64.rpm':
Fehlercode: 'Curl error 16'
Fehlermeldung: ''

Just happened again:

@ericschdt - thx for reporting just on be on the safe side: did you see the error on recent Tumbleweed? In other words - is there a chance that libcurl or related libraries were several months old on that machine, so the experienced problem might be already fixed recently? Or is that machine updated regularly and it is unlikely that libcurl was that old?

@ericschdt - thx for reporting just on be on the safe side: did you see the error on recent Tumbleweed?

Yes! Currently openSUSE Tumbleweed 20211106

In other words - is there a chance that libcurl or related libraries were several months old on that machine, so the experienced problem might be already fixed recently?

I currently use libcurl4 7.79.1-1.3

Or is that machine updated regularly and it is unlikely that libcurl was that old?

It is regularly updated (at least weekly).

Running openSUSE Tumbleweed 20211209 with curl 7.80.0 (x86_64-suse-linux-gnu) libcurl/7.80.0 OpenSSL/1.1.1l-fips zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 libidn2/2.3.2 libpsl/0.21.1 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 OpenLDAP/2.4.59 Release-Date: 2021-11-10 and encountering the error for the first time ever.

I am also running openSUSE Tumbleweed 20211209 with curl 7.80.0 (x86_64-suse-linux-gnu) and encountering error for the first time. It occurs on zypper ref for repositories Virtualization, snappy, and openSUSE:Tumbleweed.

Also encountering this error on TW 20211210

@cglmrfreeman @dwymark @Nano-Ocelot it looks something triggered it today. Do you see the error only on .rpm or on other files as well? I see reddit complains about repomd.xml , which is kind of new problem as far as I understand https://www.reddit.com/r/openSUSE/comments/rgs6zs/keep_getting_curl_error_when_doing_zypper_dup/

@cglmrfreeman @dwymark @Nano-Ocelot it looks something triggered it today. Do you see the error only on .rpm or on other files as well? I see reddit complains about repomd.xml , which is kind of new problem as far as I understand https://www.reddit.com/r/openSUSE/comments/rgs6zs/keep_getting_curl_error_when_doing_zypper_dup/

I think it is the same error as that thread. I managed to finally get through and update to 20211213 so I'll have to confirm next time I update, but I was getting failures to zypper ref repos with errors for repomd.xml. But after 5 or 6 tries finally refreshed all the repos. Then when I used zypper dup, every 2-6 rpms would error 16 as well. I had to repeatedly retry to get through it all.

@cglmrfreeman @dwymark @Nano-Ocelot it looks something triggered it today. Do you see the error only on .rpm or on other files as well? I see reddit complains about repomd.xml , which is kind of new problem as far as I understand https://www.reddit.com/r/openSUSE/comments/rgs6zs/keep_getting_curl_error_when_doing_zypper_dup/

The error occurred for me for repomd.xml, among other files. I am including output from zypper ref below:

Repository 'Main Repository (NON-OSS)' is up to date.                                                                                                                                                                                                                         
Repository 'Main Repository (OSS)' is up to date.                                                                                                                                                                                                                             
Repository 'Main Update Repository' is up to date.                                                                                                                                                                                                                            
Problem retrieving files from 'Virtualization'.
Download (curl) error for 'https://download.opensuse.org/repositories/Virtualization/openSUSE_Tumbleweed/repodata/repomd.xml':
Error code: Curl error 16
Error message: 

Please see the above error message for a hint.
Skipping repository 'Virtualization' because of the above error.
Retrieving repository 'openSUSE:Tumbleweed' metadata ..................................................................................................................................................................................................................[error]
Repository 'openSUSE:Tumbleweed' is invalid.
[https-download.opensuse.org-c50dffcc|https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/'
 - Download (curl) error for 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/repodata/repomd.xml':
Error code: Curl error 16
Error message: 

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE:Tumbleweed' because of the above error.
Retrieving repository 'snappy' metadata ...............................................................................................................................................................................................................................[error]
Repository 'snappy' is invalid.
[snappy|https://download.opensuse.org/repositories/system:/snappy/openSUSE_Tumbleweed] Valid metadata not found at specified URL
History:
 - Download (curl) error for 'https://download.opensuse.org/repositories/system:/snappy/openSUSE_Tumbleweed/repodata/3d0df32a12706cc9c7378c8e7e230a12ec66fe8f941c3a6fa547859480798b82-primary.xml.gz':
Error code: Curl error 16
Error message: 

 - Can't provide ./repodata/3d0df32a12706cc9c7378c8e7e230a12ec66fe8f941c3a6fa547859480798b82-primary.xml.gz

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'snappy' because of the above error.
Retrieving repository 'vivaldi' metadata ...............................................................................................................................................................................................................................[done]
Building repository 'vivaldi' cache ....................................................................................................................................................................................................................................[done]
Some of the repositories have not been refreshed because of an error.

@Vogtinator @andrii-suse The openqa logs both show the same behavior (2021-12-13 18:58:48 and 19:44:19 for https://download.opensuse.org/repositories/benchmark:/ltp:/devel/openSUSE_Leap_15.3/x86_64/ltp-20211213.b48cd64ba-lp153.913.1.x86_64.rpm)

We got a metalink file for the package. The file contained no or too few mirrors to do a chunked download. We ask d.o.o. for the file again awaiting to be redirected. While this worked for a couple of files before (packages as well as repo metadata), it fails for ltp-20211213.

Maybe openqa can export ZYPP_MEDIA_CURL_DEBUG=2 in the environment for a while. It may help to see the http header and server responses in the zypper.log at the time the error occurs.

Maybe openqa can export ZYPP_MEDIA_CURL_DEBUG=2 in the environment for a while. It may help to see the http header and server responses in the zypper.log at the time the error occurs.

Can be done, for the install_ltp test specifically. Though I wonder whether it'll happen in install_ltp again at all... If it happens again there, I'll make a PR to add it.

For anyone thinking of running a script to automatically reload zipper on failures like this one from reddit, doing so didn't end up working for me as the curl failures just kept coming in hot and heavy. However what I did not realize when I ran this is that for each instance of zypper I ran, a new snapshot on my btrfs filesystem would be created, and my machine over time became very not happy with freezes and hangs. A sudo btrfs quota disable / and sudo btrfs quota enable / seemed to calm things down (at least in this boot instance) but I thought I would post a word of caution to mitigate anyone who might try this method after searching for a definitive way to update through the curl error 16s.

@cglmrfreeman Could you provide your /var/log/zypper.log? (per email to me is ok) I'd like to check whether the error exclusively occurs on connections to d.o.o. By npw I never saw it happen when connecting to a different server.

Regarding the snapper plugin taking snapshots I filed https://github.com/openSUSE/libzypp/issues/365.

Great! Not at home at the moment, but I will upload the log when I can!

Hello everyone, I think I have this problem too.

Here what I get when I want to refresh to cache of each repository:

❯ sudo zypper ref
[sudo] password for root: 
Sorry, try again.
[sudo] password for root: 
Repository 'database-x86_64' is up to date.                                                                                                                                                                                                                                   
Repository 'Main Repository (NON-OSS)' is up to date.                                                                                                                                                                                                                         
Repository 'Main Repository (OSS)' is up to date.                                                                                                                                                                                                                             
Repository 'Main Update Repository' is up to date.                                                                                                                                                                                                                            
Repository 'openSUSE:Tumbleweed' is up to date.                                                                                                                                                                                                                               
Repository 'home:MasterPatricko' is up to date.                                                                                                                                                                                                                               
Problem retrieving files from 'server:database'.
Download (curl) error for 'https://download.opensuse.org/repositories/server:/database/openSUSE_Tumbleweed/repodata/repomd.xml':
Error code: Curl error 16
Error message: 

Please see the above error message for a hint.
Skipping repository 'server:database' because of the above error.
Repository 'openSUSE-20211110-0' is up to date.                                                                                                                                                                                                                               
Repository 'packman' is up to date.                                                                                                                                                                                                                                           
Repository 'spotify-easyrpm' is up to date.                                                                                                                                                                                                                                   
Repository 'vscode' is up to date.                                                                                                                                                                                                                                            
Some of the repositories have not been refreshed because of an error.

I would like to do a zypper dup but I have some packages which cannot be dowloaded. Zypper give up when he cannot download the file 5 times.

If I set up this var ZYPP_MEDIA_CURL_DEBUG=2 I don't have any error anymore.

❯ ZYPP_MEDIA_CURL_DEBUG=2 sudo zypper ref
Repository 'database-x86_64' is up to date.                                                                                                                                                                                                                                   
Repository 'Main Repository (NON-OSS)' is up to date.                                                                                                                                                                                                                         
Repository 'Main Repository (OSS)' is up to date.                                                                                                                                                                                                                             
Repository 'Main Update Repository' is up to date.                                                                                                                                                                                                                            
Repository 'openSUSE:Tumbleweed' is up to date.                                                                                                                                                                                                                               
Repository 'home:MasterPatricko' is up to date.                                                                                                                                                                                                                               
Repository 'server:database' is up to date.                                                                                                                                                                                                                                   
Repository 'openSUSE-20211110-0' is up to date.                                                                                                                                                                                                                               
Repository 'packman' is up to date.                                                                                                                                                                                                                                           
Repository 'spotify-easyrpm' is up to date.                                                                                                                                                                                                                                   
Repository 'vscode' is up to date.                                                                                                                                                                                                                                            
All repositories have been refreshed.

But I'm still unable to do an upgrade

Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/libopenssl1_1-1.1.1l-4.1.x86_64.rpm':
Error code: Curl error 55
Error message: Connection died, tried 5 times before giving up

Abort, retry, ignore? [a/r/i/...? shows all options] (a):

Reply

andrii-suse: You may be experiencing this issue alone, because almost nobody uses download.opensuse.org with https nowadays. (Or am I wrong?)

I've been seeing people complaining about this issue quite often. Even though we're using htpp://d.o.o as our defaults on Tumbleweed and Leap (tested on 2 VM without edid repos), people, myself included, must be editing their repos to use https. In our wiki there isn't any https link in use too. Now YaST 1-Click seems to allway add repos using https.

Since around two months ago I've been running into this issue while either zypper ref, zypper dup, or zypper in. The last two happen because by default Zypper refreshes the repos automatically.

A couple of refreshes today, Dec 20

First `zypper ref` failure

I did a zypper ref today and ran into Curl error 16 and 52

zypper ref

07:06 PM geeko@multiverse (~)# zypper ref
07:06 PM geeko@multiverse (~)# ZYPP_MEDIA_CURL_DEBUG=2
07:06 PM geeko@multiverse (~)# echo $ZYPP_MEDIA_CURL_DEBUG
2
07:06 PM geeko@multiverse (~)# zypper ref
Retrieving repository 'openSUSE:Tumbleweed (OSS)' metadata ........................................................[error]
Repository 'openSUSE:Tumbleweed (OSS)' is invalid.
[01-oss|https://download.opensuse.org/tumbleweed/repo/oss/] Valid metadata not found at specified URL
History:
- Download (curl) error for 'https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml':
Error code: Curl error 16
Error message:
- Can't provide /repodata/repomd.xml
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE:Tumbleweed (OSS)' because of the above error.
Problem retrieving files from 'openSUSE:Tumbleweed (NON-OSS)'.
Download (curl) error for 'https://download.opensuse.org/tumbleweed/repo/non-oss/media.1/media':
Error code: Curl error 16
Error message:
Please see the above error message for a hint.
Skipping repository 'openSUSE:Tumbleweed (NON-OSS)' because of the above error.
Repository 'openSUSE:Tumbleweed:Update' is up to date.
Retrieving repository 'Packman (Essentials)' metadata ..............................................................[done]
Building repository 'Packman (Essentials)' cache ...................................................................[done]
Retrieving repository 'Packman (Multimedia)' metadata ..............................................................[done]
Building repository 'Packman (Multimedia)' cache ...................................................................[done]
Retrieving repository 'GNOME:Factory' metadata .....................................................................[done]
Problem retrieving files from 'GNOME:Apps'.
Download (curl) error for 'https://download.opensuse.org/repositories/GNOME:/Apps/openSUSE_Factory/repodata/repomd.xml':
Error code: Curl error 52
Error message: Empty reply from server
Please see the above error message for a hint.
Skipping repository 'GNOME:Apps' because of the above error.
Problem retrieving files from 'openSUSE:Tools'.
Download (curl) error for 'https://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_Tumbleweed/media.1/media':
Error code: Curl error 16
Error message:
Please see the above error message for a hint.
Skipping repository 'openSUSE:Tools' because of the above error.
Retrieving repository 'home:luc14n0' metadata ......................................................................[done]
Building repository 'home:luc14n0' cache ...........................................................................[done]
Some of the repositories have not been refreshed because of an error.

Second `zypper ref -vvv` failure

zypper ref -vvv

07:11 PM geeko@multiverse (~)# ZYPP_MEDIA_CURL_DEBUG=2 zypper -vvv ref
Verbosity: 3
Initializing Target
Specified repositories:
Checking whether to refresh metadata for openSUSE:Tumbleweed (OSS)
Retrieving: https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml ......................[done (9.9 KiB/s)]
Retrieving: https://download.opensuse.org/tumbleweed/repo/oss/media.1/media ........................................[done]
Retrieving: https://download.opensuse.org/tumbleweed/repo/oss/media.1/media ........................................[done]
Retrieving: https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.asc ....................[done (481 B/s)]
Retrieving: https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.key ..............................[done]
Retrieving: https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml .................................[error]
Retrieving repository 'openSUSE:Tumbleweed (OSS)' metadata ........................................................[error]
Repository 'openSUSE:Tumbleweed (OSS)' is invalid.
[01-oss|https://download.opensuse.org/tumbleweed/repo/oss/] Valid metadata not found at specified URL
History:
- Download (curl) error for 'https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml':
Error code: Curl error 16
Error message:
- Can't provide /repodata/repomd.xml
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE:Tumbleweed (OSS)' because of the above error.
Checking whether to refresh metadata for openSUSE:Tumbleweed (NON-OSS)
Retrieving: https://download.opensuse.org/tumbleweed/repo/non-oss/repodata/repomd.xml ..............................[done]
Retrieving: https://download.opensuse.org/tumbleweed/repo/non-oss/media.1/media ....................................[done]
Retrieving: https://download.opensuse.org/tumbleweed/repo/non-oss/media.1/media ....................................[done]
Retrieving: https://download.opensuse.org/tumbleweed/repo/non-oss/repodata/repomd.xml.asc ..........................[done]
Retrieving: https://download.opensuse.org/tumbleweed/repo/non-oss/repodata/repomd.xml.key .........................[error]
Retrieving: https://download.opensuse.org/tumbleweed/repo/non-oss/repodata/repomd.xml ..................[done (9.4 KiB/s)]
Repository: openSUSE:Tumbleweed (NON-OSS)
Key Fingerprint: 22C0 7BA5 3417 8CD0 2EFE 22AA B88B 2FD4 3DBD C284
Key Name: openSUSE Project Signing Key
Key Algorithm: RSA 2048
Key Created: Mon May 5 05:37:40 2014
Key Expires: Thu May 2 05:37:40 2024
Rpm Name: gpg-pubkey-3dbdc284-53674dd4
Retrieving: https://download.opensuse.org/tumbleweed/repo/non-oss/repodata/bb00f540fe800fe1594bc5fc4f0f411f3460a2782[done]
Retrieving: https://download.opensuse.org/tumbleweed/repo/non-oss/repodata/07b367cb48822fbc1a1f5474a1bb45b236e68b213[done]
Retrieving repository 'openSUSE:Tumbleweed (NON-OSS)' metadata .....................................................[done]
Building repository 'openSUSE:Tumbleweed (NON-OSS)' cache ..........................................................[done]
Checking whether to refresh metadata for openSUSE:Tumbleweed:Update
Retrieving: https://download.opensuse.org/update/tumbleweed/repodata/repomd.xml ...................................[error]
Problem retrieving files from 'openSUSE:Tumbleweed:Update'.
Download (curl) error for 'https://download.opensuse.org/update/tumbleweed/repodata/repomd.xml':
Error code: Curl error 16
Error message:
Please see the above error message for a hint.
Skipping repository 'openSUSE:Tumbleweed:Update' because of the above error.
Skipping disabled repository 'openSUSE:Tumbleweed (Source)'
Skipping disabled repository 'openSUSE:Tumbleweed (Debug)'
Checking whether to refresh metadata for Packman (Essentials)
Retrieving: https://ftp.gwdg.de/pub/linux/misc/packman/suse/openSUSE_Tumbleweed/Essentials/repodata/repo[done (1.3 KiB/s)]
Repository 'Packman (Essentials)' is up to date.
Checking whether to refresh metadata for Packman (Multimedia)
Retrieving: https://ftp.gwdg.de/pub/linux/misc/packman/suse/openSUSE_Tumbleweed/Multimedia/repodata/repomd.xml .....[done]
Repository 'Packman (Multimedia)' is up to date.
Skipping disabled repository 'GNOME:Next'
Checking whether to refresh metadata for GNOME:Factory
Retrieving: https://download.opensuse.org/repositories/GNOME:/Factory/openSUSE_Factory/repodata/repomd.x[done (1.7 KiB/s)]
Repository 'GNOME:Factory' is up to date.
Checking whether to refresh metadata for GNOME:Apps
Retrieving: https://download.opensuse.org/repositories/GNOME:/Apps/openSUSE_Factory/repodata/repomd.xml ...........[error]
Problem retrieving files from 'GNOME:Apps'.
Download (curl) error for 'https://download.opensuse.org/repositories/GNOME:/Apps/openSUSE_Factory/repodata/repomd.xml':
Error code: Curl error 16
Error message:
Please see the above error message for a hint.
Skipping repository 'GNOME:Apps' because of the above error.
Checking whether to refresh metadata for openSUSE:Tools
Retrieving: https://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_Tumbleweed/repodata/repomd.xml .....[done]
Retrieving: https://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_Tumbleweed/media.1/media ......[not found]
Retrieving: https://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_Tumbleweed/repodata/repomd.xml.asc .[done]
Retrieving: https://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_Tumbleweed/repodata/repomd.xml.key [error]
Retrieving: https://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_Tumbleweed/repodata/repomd.xml .....[done]
Repository: openSUSE:Tools
Key Fingerprint: FCAD AFC8 1273 B9E7 F184 F2B0 8266 59A9 013E 5B65
Key Name: openSUSE:Tools OBS Project
Key Algorithm: RSA 2048
Key Created: Mon Dec 7 07:26:44 2020
Key Expires: Wed Feb 15 07:26:44 2023
Rpm Name: gpg-pubkey-013e5b65-5fce0364
Retrieving: https://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_Tumbleweed/repodata/2b9985ec93a5e24[error]
Retrieving repository 'openSUSE:Tools' metadata ...................................................................[error]
Repository 'openSUSE:Tools' is invalid.
[11-oS-tools|https://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_Tumbleweed/] Valid metadata not found at specified URL
History:
- Download (curl) error for 'https://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_Tumbleweed/repodata/2b9985ec93a5e2414bcb7deabb3d2cc19937c58579ec6b3008a217275eeebfa6-primary.xml.gz':
Error code: Curl error 16
Error message:
- Can't provide ./repodata/2b9985ec93a5e2414bcb7deabb3d2cc19937c58579ec6b3008a217275eeebfa6-primary.xml.gz
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE:Tools' because of the above error.
Checking whether to refresh metadata for home:luc14n0
Retrieving: https://download.opensuse.org/repositories/home:/luc14n0/openSUSE_Tumbleweed/repodata/repomd.xml ......[error]
Problem retrieving files from 'home:luc14n0'.
Download (curl) error for 'https://download.opensuse.org/repositories/home:/luc14n0/openSUSE_Tumbleweed/repodata/repomd.xml':
Error code: Curl error 16
Error message:
Please see the above error message for a hint.
Skipping repository 'home:luc14n0' because of the above error.
Some of the repositories have not been refreshed because of an error.

Info

zypper -vvv ref operations in zypper.log

08:12 PM geeko@multiverse (~)# rpm -q libcurl4 curl librepo0 rpm zypper
libcurl4-7.80.0-1.1.x86_64
curl-7.80.0-1.1.x86_64
librepo0-1.14.2-1.2.x86_64
rpm-4.17.0-2.1.x86_64
zypper-1.14.50-1.1.x86_64

Edit: this comment is not relevant anymore as I was too optimisting thinking about root cause (and the issue triggered on Wednesday, before the changes I had in mind below). ~I've reverted some config changes on download.opensuse.org from last Thursday, (when the issue triggered with high frequency).~ ~My guess is that the error was related to that change and should gone now, but let me know if it you still see it (often). I guess the error itself is not related to zypper, but zypper might be friendlier handling around it (e.g. complain and automatically retry?).~ ~I suggest discussing here what zypper can do if server's reply misbehaves and discuss infrastructure issues e.g. in https://progress.opensuse.org/issues/104193~

I'm currently doing a TW dup (from TW 20211215, so pretty recent), and I've seen several error 16's, some 8+ times in the process of downloading 259 packages. Just (as in a minute ago) made it past the downloads, currently installing.

I only stumbled upon this issue after idly browsing for the error, so it's a bit late to grab more verbose logs, I'm afraid, but at least it's a datapoint.

Today I experienced the problem for first time in months - saw it twice. Then I replaced download.opensuse.org with mirrorcache.opensuse.org and zypper finished downloading 2000+ files without errors. As I mentioned earlier it looks some issue around Apache / haproxy : sudo sed -i 's,download.opensuse.org,mirrorcache.opensuse.org,g' /etc/zypp/repos.d/*.repo

Today I experienced the problem for first time in months - saw it twice. Then I replaced download.opensuse.org with mirrorcache.opensuse.org and zypper finished downloading 2000+ files without errors. As I mentioned earlier it looks some issue around Apache / haproxy : sudo sed -i 's,download.opensuse.org,mirrorcache.opensuse.org,g' /etc/zypp/repos.d/*.repo

I wonder how close are we to switching mirrorbrain with mirrorcache, because if we are about to really switch chem we won't have to worry about this problem anymore, I reckon. Some people get concerned about using plain http without TLS like we never used it before... And I don't wanna tell people to edit their repos to use mirrorcache. For now I'm telling them you either use http like we have been using for ages or keep using https and handle this Curl error by retrying until it's sorted out.

Now I've heard that Flathub started to run into this issue as well.

For now I'm telling them you either use http like we have been using for ages or keep using https and handle this Curl error by retrying until it's sorted out.

Unlike normal errors, that one is not auto-retried and completely breaks unattended system updates which especially annoying when there are >4000 packages in queue. Also, only switching from https to http on download.opensuse.org links just replaces error code 12 with error code 55.

wonder how close are we to switching mirrorbrain with mirrorcache

https is actually fully served by mirrorcache already, but the error happens not in mirrorcache as far as I understand. Replacing download.o.o with mirrorcache.o.o is just omitting Apache. https://progress.opensuse.org/issues/104193 claims that it may be related to overload, so I've temporarily redirected 33% of http *.rpm requests to mirrorcache , hoping that it will decrease overall load on Nuremberg servers

So zypper.log with ZYPP_MEDIA_CURL_DEBUG=2 : note that access_log on apache has libquadmath0-11.2.1+git1018-1.1.x86_64.rpm for my IP but doesn't have any reference of libpython2_7-1_0-2.7.18-14.1.x86_64.rpm for it - not sure the request wasn't even logged

2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp] PackageProvider.cc(providePackage):513 provided Package (24924)libquadmath0-11.2.1+git1018-1.1.x86_64(download.opensuse.org-oss) at /var/cache/zypp/packages/download.opensuse.org-oss/x86_64/libquadmath0-11.2.1+git1018-1.1.x86_64.rpm
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp] PackageProvider.cc(providePackage):412 provide Package (24528)libpython2_7-1_0-2.7.18-14.1.x86_64(download.opensuse.org-oss)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] DeltaCandidates.cc(deltaRpms):82 package: (24528)libpython2_7-1_0-2.7.18-14.1.x86_64(download.opensuse.org-oss)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media] RepoProvideFile.cc(provideFile):241 [1]./x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm{788.3 KiB|sha256-0a5d762982aa75e188d5671fd97257b9d5ece6e0382b4f7e517b938d680e794e|}
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher++] Fetcher.cc(addCachePath):331 Adding fetcher cache: '/var/cache/zypp/packages/download.opensuse.org-oss'.
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media] RepoProvideFile.cc(provideFile):260 Added cache path /var/cache/zypp/packages/download.opensuse.org-oss
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media] RepoProvideFile.cc(provideFile):294 Providing file of repo 'download.opensuse.org-oss' from https://download.opensuse.org/tumbleweed/repo/oss/
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher] Fetcher.cc(downloadAndReadIndexList):705 No indexes to read.
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher] Fetcher.cc(locateInCache):362 start fetcher with 1 cache directories.
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher] Fetcher.cc(provideToDest):533 Not found in cache, retrieving...
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher++] MediaSetAccess.cc(provide):278 Going to try to provide  file ./x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm from media number 1
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaManager.cc(checkDesired):154 checkDesired(1): desired (cached)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1335 dest: /var/tmp/AP_0xAJvxU8/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1336 temp: /var/tmp/AP_0xAJvxU8/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm.new.zypp.Ze9iDC
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1008 ./x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1018 URL: https://download.opensuse.org/tumbleweed/repo/oss/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Found bundle for host download.opensuse.org: 0x562a865dbb70 [can multiplex]
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Re-using existing connection! (#17) with host download.opensuse.org
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Connected to download.opensuse.org (2001:67c:2178:8::13) port 443 (#17)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Using Stream ID: d (easy handle 0x562a864acfe0)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > GET /tumbleweed/repo/oss/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm HTTP/2
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > Host: download.opensuse.org
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > user-agent: ZYpp 17.28.8 (curl 7.80.0)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > x-zypp-anonymousid: 87f76fb3-72cd-4a58-adeb-61c7063d5ba3
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > x-zypp-distributionflavor: appliance-custom
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > accept: */*, application/metalink+xml, application/metalink4+xml
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * stopped the pause stream!
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Closing connection 17
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * TLSv1.3 (IN), TLS alert, close notify (256):
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * TLSv1.3 (OUT), TLS alert, close notify (256):
2021-12-21 18:19:29 <3> localhost.localdomain(607) [zypp::media] MediaCurl.cc(doGetFileCopyFile):1083 curl error: 16: , temp file size 0 bytes.
2021-12-21 18:19:29 <5> localhost.localdomain(607) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaCurl.cc(evaluateCurlCode):699 THROW:    Download (curl) error for 'https://download.opensuse.org/tumbleweed/repo/oss/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm'

http log:

2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp] PackageProvider.cc(providePackage):513 provided Package (43066)python38-SQLAlchemy-1.4.27-1.1.x86_64(download.opensuse.org-oss) at /var/cache/zypp/packages/download.opensuse.org-oss/x86_64/python38-SQLAlchemy-1.4.27-1.1.x86_64.rpm
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp] PackageProvider.cc(providePackage):412 provide Package (31066)mpg123-pulse-1.29.3-1.1.x86_64(download.opensuse.org-oss)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] DeltaCandidates.cc(deltaRpms):82 package: (31066)mpg123-pulse-1.29.3-1.1.x86_64(download.opensuse.org-oss)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media] RepoProvideFile.cc(provideFile):241 [1]./x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm{32.1 KiB|sha256-efbb09146f3437b2a77266750f7c2fd288f3978d1a299b5918cb6fc7c7f9106f|}
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher++] Fetcher.cc(addCachePath):331 Adding fetcher cache: '/var/cache/zypp/packages/download.opensuse.org-oss'.
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media] RepoProvideFile.cc(provideFile):260 Added cache path /var/cache/zypp/packages/download.opensuse.org-oss
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media] RepoProvideFile.cc(provideFile):294 Providing file of repo 'download.opensuse.org-oss' from http://download.opensuse.org/tumbleweed/repo/oss/
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher] Fetcher.cc(downloadAndReadIndexList):705 No indexes to read.
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher] Fetcher.cc(locateInCache):362 start fetcher with 1 cache directories.
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher] Fetcher.cc(provideToDest):533 Not found in cache, retrieving...
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher++] MediaSetAccess.cc(provide):278 Going to try to provide  file ./x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm from media number 1
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaManager.cc(checkDesired):154 checkDesired(13): desired (cached)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1335 dest: /var/tmp/AP_0xfxSKSC/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1336 temp: /var/tmp/AP_0xfxSKSC/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm.new.zypp.711Biw
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1008 ./x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1018 URL: http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Found bundle for host download.opensuse.org: 0x561430b3dfc0 [serially]
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Can not multiplex, even if we wanted to!
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Re-using existing connection! (#13) with host download.opensuse.org
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Connected to download.opensuse.org (195.135.221.134) port 80 (#13)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > GET /tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm HTTP/1.1
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > Host: download.opensuse.org
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > User-Agent: ZYpp 17.28.8 (curl 7.80.0)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > X-ZYpp-AnonymousId: 87f76fb3-72cd-4a58-adeb-61c7063d5ba3
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > X-ZYpp-DistributionFlavor: appliance-custom
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > Accept: */*, application/metalink+xml, application/metalink4+xml
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Closing connection 13
2021-12-21 19:01:12 <3> localhost.localdomain(21118) [zypp::media] MediaCurl.cc(doGetFileCopyFile):1083 curl error: 55: Connection died, tried 5 times before giving up, temp file size 0 bytes.
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaCurl.cc(evaluateCurlCode):699 THROW:    Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaCurl.cc(doGetFileCopyFile):1099 RETHROW:  Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaMultiCurl.cc(doGetFileCopy):1364 RETHROW:  Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [DEFINE_LOGGROUP] PathInfo.cc(unlink):706 unlink /var/tmp/AP_0xfxSKSC/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm.new.zypp.711Biw
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaCurl.cc(getFileCopy):528 RETHROW:  Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaSetAccess.cc(provide):288 CAUGHT:   Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaHandler.cc(getDetectedDevices):1384 No devices for this medium

So, as stated in the progress issue, the bottom of all this could be that rsync is not catching up with the list of files to be mirrored and mirrorbrain gets "confused" about the file-to-be-mirrored-yet, curl try to fetch the file, but doesn't find it on any mirrors and tries to fetch the file direct on the "central" servers, and maybe because the servers in Nuremberg are overloaded, curl errors out?

I have the feeling I'm mistaken about how the whole cycle works. If someone can shed some light on this, please do.

both mirrorbrain and mirrorcache do redirect to mirrors properly from what I see. It looks cached connections to download.o.o die from time to time and zypper overreacts giving up completely. It should just try to re-establish new connection, but it doesn't (the way I read the log). Or maybe it just should wait some time before retrying, but the log just says that it gives up immediately. I wonder if anything changed recently around that code.

both mirrorbrain and mirrorcache do redirect to mirrors properly from what I see. It looks cached connections to download.o.o die from time to time and zypper overreacts giving up completely. It should just try to re-establish new connection, but it doesn't (the way I read the log). Or maybe it just should wait some time before retrying, but the log just says that it gives up immediately. I wonder if anything changed recently around that code.

Now I see! Indeed, the log shows it doesn't really retry.

https://github.com/curl/curl/issues/7400 looks similar, but that's apparently fixed in 7.79.0 (TW has 7.80.0).

It could still be an issue in curl. Was this ever observed outside of Tumbleweed?

curl/curl#7400 looks similar, but that's apparently fixed in 7.79.0 (TW has 7.80.0).

It could still be an issue in curl. Was this ever observed outside of Tumbleweed?

The issue has also been reportetd for Leap 15.3: https://forums.opensuse.org/showthread.php/564178-update-curl-error-16?p=3093019#post3093019

curl/curl#7400 looks similar, but that's apparently fixed in 7.79.0 (TW has 7.80.0).

It could still be an issue in curl. Was this ever observed outside of Tumbleweed?

+1 here. The issue was closed with:

.. I'm going to go ahead and merge the revert and we'll keep our eyes open and see if we run into any problems going forward.

After reading the issue we can see the whole thing wasn't something trivial.

Yeah the worst thing is that it is not clear who to blame: libzypp vs libcurl vs infrastructure behind download.o.o and it is not clear why the issue occurrences did skyrocket on 15-Dec . It looks infrastructure is partially to blame because connection is closed, but it is not clear why libcurl(?) doesn't put any effort in retry (like make a short pause between attempts).

Yeah the worst thing is that it is not clear who to blame: libzypp vs libcurl vs infrastructure behind download.o.o and it is not clear why the issue occurrences did skyrocket on 15-Dec . It looks infrastructure is partially to blame because connection is closed, but it is not clear why libcurl(?) doesn't put any effort in retry (like make a short pause between attempts).

Well. Maybe we can take libzypp out of this (maybe?) since even the curl command runs into the same issue. From the forum posted earlier, after the guy ran into Error code: Curl error 16 during zypper dup he tried:

linux-4skc:~ # curl -vvvIL 'https://download.opensuse.org/repositories/openSUSE:/Factory/snapshot/media.1/media' 
*   Trying 2001:67c:2178:8::13:443... 
*   Trying 195.135.221.134:443... 
* Connected to download.opensuse.org (195.135.221.134) port 443 (#0) 
* ALPN, offering h2 
* ALPN, offering http/1.1 
* TLSv1.3 (OUT), TLS handshake, Client hello (1): 
* TLSv1.3 (IN), TLS handshake, Server hello (2): 
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): 
* TLSv1.3 (IN), TLS handshake, Certificate (11): 
* TLSv1.3 (IN), TLS handshake, CERT verify (15): 
* TLSv1.3 (IN), TLS handshake, Finished (20): 
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): 
* TLSv1.3 (OUT), TLS handshake, Finished (20): 
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 
* ALPN, server accepted to use h2 
* Server certificate: 
*  subject: CN=opensuse.org 
*  start date: Oct 29 00:00:05 2021 GMT 
*  expire date: Jan 27 00:00:04 2022 GMT 
*  subjectAltName: host "download.opensuse.org" matched cert's "*.opensuse.org" 
*  issuer: C=US; O=Let's Encrypt; CN=R3 
*  SSL certificate verify ok. 
* Using HTTP2, server supports multiplexing 
* Connection state changed (HTTP/2 confirmed) 
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 
* Using Stream ID: 1 (easy handle 0x55c06be4ce00) 
> HEAD /repositories/openSUSE:/Factory/snapshot/media.1/media HTTP/2 
> Host: download.opensuse.org 
> user-agent: curl/7.80.0 
> accept: */* 
>  
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 
* old SSL session ID is stale, removing 
* Empty reply from server 
* Closing connection 0 
* TLSv1.3 (OUT), TLS alert, close notify (256): 
curl: (52) Empty reply from server 
linux-4skc:~ #

I just can't understand what's going on here, the connection using HTTP/2 is made by curl but in the end it's being refused, or something else goes wrong here?! (Anyone?)

* Connection state changed (HTTP/2 confirmed) 
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 
* Using Stream ID: 1 (easy handle 0x55c06be4ce00) 
> HEAD /repositories/openSUSE:/Factory/snapshot/media.1/media HTTP/2 
> Host: download.opensuse.org 
> user-agent: curl/7.80.0 
> accept: */* 
>  
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 
* old SSL session ID is stale, removing 
* Empty reply from server 
* Closing connection 0

We know that the infrastructure ('mirrorbrain') is a bit clueless about many (I don't know to what extent here) files that are not being mirrored in a timely manner, sometimes mirrors lag behind days, and it thinks that the files are already being mirrored. And the list of "ghost" files grows the newer the file is, according to the progress.o.o issue ticket.

So could it be that the issues with the infra is exposing the issues with curl? Does my reasoning makes sense? The more I go back to that Curl upstream issue and continue to (re)read the more I smell trouble.

One of the crossing issues that was mentioned there, was the curl/curl#6862 where someone ran into a problem when curl tried to establish a connection over HTTP/2, their server replied with HTTP_1_1_REQUIRED and then requiring a new connection. The problem was that curl was retrying over HTTP/2, again, instead of HTTP/1.1. The problem was apparently fixed.

If you read the whole thing it'll make more sense to you than it made to me, I'm sure.

Would it be possible to roll my own curl? I tried building from git but it lacked many of the protocols and features compared to the openSUSE:Factory vendor provides.

curl command runs into the same issue

That is actually a valuable observation, (I've tried curl and it always succeeded).

infrastructure ('mirrorbrain') is a bit clueless

My strong understanding is that issue happens before mirrorbrain/mirrorcache and it is not even present in Apache's logs. It might be some ha-proxy bug or other transport issue, but I doubt it is a Redirector to blame. Especially that Apache is configured to return files below 2K (like /media.1/media ) without involving a redirector.

But it looks you are on the right track, still not sure how to fix it yet.

Would it be possible to roll my own curl? I tried building from git but it lacked many of the protocols and features compared to the openSUSE:Factory vendor provides.

I'm afraid my friend that it would be too much trouble for you to get a previous version of an "OBS" curl working in a recent version of Tumbleweed.

My strong understanding is that issue happens before mirrorbrain/mirrorcache and it is not even present in Apache's logs. It might be some ha-proxy bug or other transport issue, but I doubt it is a Redirector to blame. Especially that Apache is configured to return files below 2K (like /media.1/media ) without involving a redirector.

Interesting, so even with the redirector out of the game the issue keeps popping up.

From the upstream Curl issue, Alex Crichton said that in his bisection of commits experimentation, after pinpointing the culprit, and making Cargo download its own dependencies 100x, the download would the fail spuriously, but before the failure it would succeed at least 100 times.

So, in order to gather more info, we could setup a testbed using Curl to loop though less-than-2k files downloading there headers, and abusing of verbosity/debugging facilities. I just need to know if there's a safe way of doing this, because I can't say if I let a script like this running for a while it could trigger some kind of response from the servers like "Hey, this dude here is doing some funny business and I don't like it, so let's block his requests.".

If anyone has some input on this or even a better idea please share with us. Maybe I should hit our mailing lists too.

I'm afraid my friend that it would be too much trouble for you to get a previous version of an "OBS" curl working in a recent version of Tumbleweed.

Well theoretically if I wanted to do that (even though finding old RPMs is difficult) I would probably find an older ISO of Tumblweed and extract it from that instead of building it from scratch. I was thinking more along the lines of building 7.81 from git, but after looking at OBS, there is some interesting bandaids and patches applied. I am not familiar enough with the OBS system to clone it and have the OBS system test/build the RPM.

I am assuming I could clone the git, apply all the patches and diffs manually and make sure all these options are used -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g as per the output of my rpm -q --queryformat="%{NAME}: %{OPTFLAGS}\n" curl

I am just not 100% sure this actually solves anything.

EDIT: For now I've changed repos to mirrorcache-us, however openSUSE:Factory, openSUSE:Factory:NonFree and openSUSE:Tools:openSUSE_Factory do not have repos under that subdomain.

So, I made some experiments with a Tumbleweed VM that needed a dup (it was sitting a couple of snapshots behind).

First thing I did, or second I can't remember, was to edit /etc/zypp/zypp.conf and set download.max_silent_tries = 50 and rebooted the VM, just to be sure. On the first zypper ref I tried it runs into the curl error 16 and says it tried 5 times. So it's confirmed, the part of the code that handles the automatic retrying is not working like it should. And zypper dup ends up with same error 16.

Next thing I did was to install DNF, on my standard system, with:

# zypper in dnf libdnf-repo-config-zypp

It already came with some handy zypper aliases then I issued dnf ref and dnf dup. Several package downloads were plagued:

[MIRROR] libthai-data-0.1.29-52.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opens
use.org/repositories/GNOME:/Factory/openSUSE_Factory/x86_64/libthai-data-0.1.29-52.1.x86_64.rpm []                        
(33/249): libjavascriptcoregtk-4_1-0-2.34.3-340.2.x86_64.rpm                              1.5 MB/s | 6.5 MB     00:04     
(34/249): libthai-data-0.1.29-52.1.x86_64.rpm                                             152 kB/s | 191 kB     00:01     
...
[MIRROR] blender-3.0.0-2.2.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org
/tumbleweed/repo/oss/x86_64/blender-3.0.0-2.2.x86_64.rpm []
(66/249): apparmor-utils-3.0.3-7.1.noarch.rpm                                             145 kB/s | 145 kB     00:00    
(67/249): blender-cycles-devel-3.0.0-2.2.noarch.rpm                                       512 kB/s | 559 kB     00:01    
(68/249): blender-demo-3.0.0-2.2.noarch.rpm                                               428 kB/s | 1.2 MB     00:02    
(69/249): MozillaThunderbird-91.4.1-1.1.x86_64.rpm                                        3.3 MB/s |  58 MB     00:17    
(70/249): chrony-4.2-1.1.x86_64.rpm                                                       178 kB/s | 296 kB     00:01    
(71/249): chrony-pool-openSUSE-4.2-1.1.noarch.rpm                                          17 kB/s |  33 kB     00:01    
(72/249): dirmngr-2.3.4-1.1.x86_64.rpm                                                    370 kB/s | 343 kB     00:00    
(73/249): efibootmgr-17-2.1.x86_64.rpm                                                     18 kB/s |  45 kB     00:02    
(74/249): blender-3.0.0-2.2.x86_64.rpm                                                    2.3 MB/s |  42 MB     00:17    
...
[MIRROR] gimp-plugins-python-2.10.30-1.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://downloa
d.opensuse.org/tumbleweed/repo/oss/x86_64/gimp-plugins-python-2.10.30-1.1.x86_64.rpm []
(77/249): gimp-2.10.30-1.1.x86_64.rpm                                                     4.8 MB/s |  13 MB     00:02    
(78/249): gimp-plugins-python-2.10.30-1.1.x86_64.rpm                                      173 kB/s | 305 kB     00:01    
...
[MIRROR] gpg2-2.3.4-1.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org/tu
mbleweed/repo/oss/x86_64/gpg2-2.3.4-1.1.x86_64.rpm []
(79/249): gpg2-2.3.4-1.1.x86_64.rpm                                                       702 kB/s | 2.3 MB     00:03    
...
[MIRROR] libyui-ncurses15-4.2.24-1.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org/tumbleweed/repo/oss/x86_64/libyui-ncurses15-4.2.24-1.1.x86_64.rpm []
[FAILED] libyui-ncurses15-4.2.24-1.1.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success 
(177-178/249): libvirt-libs-7.10.0-2.2.x8 87% [===================================-     ] 3.5 MB/s | 608 MB     00:25 ETA
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  libyui-ncurses15-4.2.24-1.1.x86_64: Cannot download, all mirrors were already tried without success
...
[MIRROR] qemu-seabios-1.15.0_0_g2dd4b9b-37.1.noarch.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org/tumbleweed/repo/oss/noarch/qemu-seabios-1.15.0_0_g2dd4b9b-37.1.noarch.rpm []
(223/249): qemu-ovmf-x86_64-202111-1.1.noarch.rpm                                         3.9 MB/s |  16 MB    00:04    
(224/249): qemu-seabios-1.15.0_0_g2dd4b9b-37.1.noarch.rpm                                 136 kB/s | 414 kB    00:03    
...
[MIRROR] vim-8.2.3863-1.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org/
tumbleweed/repo/oss/x86_64/vim-8.2.3863-1.1.x86_64.rpm []
(238/249): typelib-1_0-SpiceClientGtk-3_0-0.39-3.1.x86_64.rpm                              13 kB/s |  33 kB     00:02    
(239/249): vim-8.2.3863-1.1.x86_64.rpm

As you can see, only one package past the 10 retries threshold DNF has by default:

[FAILED] libyui-ncurses15-4.2.24-1.1.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success 
...
Error: Error downloading packages:
  libyui-ncurses15-4.2.24-1.1.x86_64: Cannot download, all mirrors were already tried without success

And this caused the whole transaction to fail. But the next dnf dup managed to download the remaining packages without hitting the threshold again.

Back at the Tumbleweed VM, I installed DNF and duped the system. As the VM was using original repo files with plain HTTP it thrown an "Error 502" at some point the retry threshold was past.

After that, I set retries=25 on /etc/dnf/dnf.conf file and dup again, after rolling back the VM, DNF hastily downloaded the almost 1K packages.

My last testsuite was to use HTTPS with retries=25 (DNF manual, dnf.conf section), the overall download time it took was quite longer then HTTP, I'm judging because of the TLS stuff, I don't know. Nevertheless I could see that at several packages DNF would take a while to download it (some took almost 15 minutes, but not be of download speed) I'm sure because of the retries it does and that the default timeout for package downloads are 30 seconds (DNF manual section). Of course DNF is smart enough to download the next package in the queue. With at least 25 retries as threshold people can leave the system unattended, I reckon, since a tried 2 HTTPS dup and both went successful.

I hope all of this helps someone to get that little spark one needs to light up his lamp.

I hope all of this helps someone to get that little spark one needs to light up his lamp.

Thank you ! Haven't had visible errors after setting up dnf. But, even more pleasing, it is noticeably faster in both operation and downloads than zypper. Probably because multi-download and multi-installation are actually working. But I had to tweak /etc/dnf/dnf.conf to be slightly more usable:

[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=False
skip_if_unavailable=True
retries=15
timeout=5
max_parallel_downloads=4
metadata_timer_sync=60
metadata_expire=600
minrate=100k
fastestmirror=True
allow_vendor_change=True
debuglevel=3
errorlevel=4
#exclude_from_weak_autodetect=False
install_weak_deps=False
log_compress=True

The only gripe I have for now is it not showing current version in the pre-update table to compare it with target. Change in vendor repos also not as obvious as with zypper.

Thank you ! Haven't had visible errors after setting up dnf. ...

I'm glad it helped you :+1:.

Just letting you guys know that I'm building older versions of Curl to pinpoint what release brought this issue down on us. I'm currently running 7.74.0 in a VM with ~1K packages to upgrade, the first dup went without any hiccups, now I'm gonna let it dup maybe 5 times to be sure, use both HTTP and HTTPS, if I don't see any curl errors I'll go to the next release.

Thank you ! Haven't had visible errors after setting up dnf. ...

I'm glad it helped you +1.

Just letting you guys know that I'm building older versions of Curl to pinpoint what release brought this issue down on us. I'm currently running 7.74.0 in a VM with ~1K packages to upgrade, the first dup went without any hiccups, now I'm gonna let it dup maybe 5 times to be sure, use both HTTP and HTTPS, if I don't see any curl errors I'll go to the next release.

It could be a coincidence, but since 66 IPs that caused a lot of traffic at download.opensuse.org have been blocked according to https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/message/YLPRFJOIN6HKTTGFY5P4N3PXSY3AG73K/, the "Curl error 16" message is gone and today's update to TW20220101 went like a charm.

It could be a coincidence, but since 66 IPs that caused a lot of traffic at download.opensuse.org have been blocked according to https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/message/YLPRFJOIN6HKTTGFY5P4N3PXSY3AG73K/, the "Curl error 16" message is gone and today's update to TW20220101 went like a charm.

Well, I'm still getting the error here using HTTPS. Maybe the HTTPS requests switch to mirrorcache two weeks ago have some influence here? I'm judging you're in Europe, am I wrong?

It could be a coincidence, but since 66 IPs that caused a lot of traffic at download.opensuse.org have been blocked according to https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/message/YLPRFJOIN6HKTTGFY5P4N3PXSY3AG73K/, the "Curl error 16" message is gone and today's update to TW20220101 went like a charm.

Well, I'm still getting the error here using HTTPS. Maybe the HTTPS requests switch to mirrorcache two weeks ago have some influence here? I'm judging you're in Europe, am I wrong?

Using https either with download.o.o or mirrorcache-eu.o.o doesn't make any difference, as it should I guess.

openSUSE / zypper