Open ericschdt opened 3 years ago
The only way to figure out more is by setting ZYPP_MEDIA_CURL_DEBUG=2
in the environment and inspecting the /var/log/zypper.log
file after running zypper. If you have zypper-log installed you can use the command zypper log
to list the last few runs and zypper log <PID>
to only get the output of a selected run.
If you could provide that file after running into the same problem again we can look into it a bit more closely.
As requested, using the environment varaible ZYPP_MEDIA_CURL_DEBUG=2
, here's the part of the log for the package gcc11-11.1.1+git400-1.4.x86_64.rpm
which caused an issue this time:
2021-07-15 18:30:41 <1> host(6165) [zypp] PackageProvider.cc(providePackage):412 provide Package (7059)gcc11-11.1.1+git400-1.4.x86_64(https-download.opensuse.org-560f2b9f)
2021-07-15 18:30:41 <1> host(6165) [zypp++] DeltaCandidates.cc(deltaRpms):82 package: (7059)gcc11-11.1.1+git400-1.4.x86_64(https-download.opensuse.org-560f2b9f)
2021-07-15 18:30:41 <1> host(6165) [zypp] RepoProvideFile.cc(provideFile):241 [1]./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm{31,2 MiB|sha256-25ab6bd9a375d4085c953c6eebe612fd3376ae8aac7a3f9c5f382e35bc023441}
2021-07-15 18:30:41 <1> host(6165) [zypp:fetcher++] Fetcher.cc(addCachePath):316 Adding fetcher cache: '/var/cache/zypp/packages/https-download.opensuse.org-560f2b9f'.
2021-07-15 18:30:41 <1> host(6165) [zypp] RepoProvideFile.cc(provideFile):260 Added cache path /var/cache/zypp/packages/https-download.opensuse.org-560f2b9f
2021-07-15 18:30:41 <1> host(6165) [zypp] RepoProvideFile.cc(provideFile):294 Providing file of repo 'https-download.opensuse.org-560f2b9f' from https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/
2021-07-15 18:30:41 <1> host(6165) [zypp:fetcher] Fetcher.cc(downloadAndReadIndexList):690 No indexes to read.
2021-07-15 18:30:41 <1> host(6165) [zypp:fetcher] Fetcher.cc(locateInCache):347 start fetcher with 1 cache directories.
2021-07-15 18:30:41 <1> host(6165) [zypp:fetcher] Fetcher.cc(provideToDest):518 Not found in cache, retrieving...
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaSetAccess.cc(provide):255 Going to try to provide file ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm from media number 1
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(1): desired (cached)
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(1): desired (cached)
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1335 dest: /var/tmp/AP_0xsG3fgI/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1336 temp: /var/tmp/AP_0xsG3fgI/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm.new.zypp.Vheb2M
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaCurl.cc(doGetFileCopyFile):1132 ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:30:41 <1> host(6165) [zypp++] MediaCurl.cc(doGetFileCopyFile):1142 URL: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:30:41 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to location: https://download.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:30:56 <3> host(6165) [zypp] MediaCurl.cc(doGetFileCopyFile):1207 curl error: 16: , temp file size 0 bytes.
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaCurl.cc(evaluateCurlCode):823 THROW: Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaCurl.cc(doGetFileCopyFile):1223 RETHROW: Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaMultiCurl.cc(doGetFileCopy):1364 RETHROW: Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <1> host(6165) [zypp] PathInfo.cc(unlink):668 unlink /var/tmp/AP_0xsG3fgI/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm.new.zypp.Vheb2M
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaCurl.cc(getFileCopy):652 RETHROW: Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 MediaSetAccess.cc(provide):268 CAUGHT: Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm':
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlercode: 'Curl error 16'
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166 Fehlermeldung: ''
2021-07-15 18:30:56 <5> host(6165) [zypp] Exception.cc(log):166
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(1): desired (cached)
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaHandler.cc(getDetectedDevices):1397 No devices for this medium
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaSetAccess.cc(provide):280 Media couldn't provide file ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm , releasing.
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaManager.cc(release):447 release(id=1)
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaHandler.cc(release):750 Request to release attached media https<https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/>, use count=1
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaHandler.cc(release):757 Releasing media https<https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/>
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaHandler.cc(disconnect):729 Disconnected: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ attached; localRoot "/var/tmp/AP_0xsG3fgI"
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:30:56 <1> host(6165) [zypp] PathInfo.cc(recursive_rmdir):435 recursive_rmdir /var/tmp/AP_0xsG3fgI
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaHandler.cc(removeAttachPoint):192 Deleted default attach point /var/tmp/AP_0xsG3fgI
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaHandler.cc(release):811 Released: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ not attached; localRoot ""
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaManager.cc(releaseAll):484 Releasing all attached media
2021-07-15 18:30:56 <1> host(6165) [zypp++] MediaManager.cc(releaseAll):502 Media id 1 not attached
2021-07-15 18:30:56 <1> host(6165) [zypp] MediaManager.cc(releaseAll):512 Exit
2021-07-15 18:30:56 <1> host(6165) [zypper++] media.cc(requestMedia):203 medium problem, url: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/, error 3, label 'openSUSE:Tumbleweed', #1
and after retrying it worked
2021-07-15 18:36:40 <1> host(6165) [zypper] prompt.cc(get_prompt_reply):436 reply: w (1)
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaSetAccess.cc(provide):335 ProvideFile exception caught, callback answer: 1
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaSetAccess.cc(provide):368 Going to try again
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(close):288 Close to access handler using id 1 requested
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaAccess.cc(close):247 Close: https(https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ not attached; localRoot "") (OK)
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaAccess.cc(open):117 Trying scheme 'https'
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/, )
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaMultiCurl.cc(MediaMultiCurl):1165 MediaMultiCurl::MediaMultiCurl(https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/, )
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaAccess.cc(open):196 Opened: https(https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ not attached; localRoot "")
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(open):258 Opened new media access using id 2 to https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaSetAccess.cc(provide):255 Going to try to provide file ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm from media number 1
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(attach):376 attach(id=2)
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaHandler.cc(createAttachPoint):390 Create attach point: attach root is not a writable directory: '/var/adm/mount'
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaHandler.cc(createAttachPoint):373 Created default attach point /var/tmp/AP_0x8i7wPp
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:40 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:40 <1> host(6165) [zypp] MediaHandler.cc(attach):690 Attached: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ attached; localRoot "/var/tmp/AP_0x8i7wPp"
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):96 checkDesired(2): desired (report by zypp::media::NoVerifier)
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(2): desired (cached)
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(2): desired (cached)
2021-07-15 18:36:40 <1> host(6165) [zypp] PathInfo.cc(assert_dir):354 mkdir /var/tmp/AP_0x8i7wPp/x86_64 00755
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1335 dest: /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1336 temp: /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm.new.zypp.07Wgg8
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaCurl.cc(doGetFileCopyFile):1132 ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:40 <1> host(6165) [zypp++] MediaCurl.cc(doGetFileCopyFile):1142 URL: https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:40 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to location: https://download.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1374 HTTP response: 200
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://mirror.de.leaseweb.net/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://mirror1.hs-esslingen.de/pub/Mirrors/ftp.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(https://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://ftp.uni-erlangen.de/pub/mirrors/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://widehat.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm, )
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):475 Proxy: not explicitly set
2021-07-15 18:36:41 <1> host(6165) [zypp++] MediaCurl.cc(setupEasy):476 Proxy: libcurl may look into the environment
2021-07-15 18:36:41 <1> host(6165) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2021-07-15 18:36:41 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:41 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:44 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:44 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:46 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:46 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:48 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:48 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:49 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:49 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:51 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:51 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:53 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:53 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:55 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:55 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:56 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:56 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:57 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:57 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:58 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:58 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:59 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:36:59 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:00 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:00 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:01 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:01 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:01 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:01 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:02 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:02 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:03 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:03 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:04 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:04 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:05 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:05 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:06 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:06 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:07 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:07 <1> host(6165) [zypp++] CurlHelper.cc(log_redirects_curl):78 redirecting to Location: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1150 overall result
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #0: state: 5 received: 7504857 url: http://mirror.de.leaseweb.net/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #1: state: 5 received: 6291456 url: http://mirror1.hs-esslingen.de/pub/Mirrors/ftp.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #2: state: 3 received: 6995528 url: https://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #3: state: 4 received: 5505024 url: http://ftp.uni-erlangen.de/pub/mirrors/opensuse/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <2> host(6165) [zypp] MediaMultiCurl.cc(run):1154 #4: state: 5 received: 6718425 url: http://widehat.opensuse.org/tumbleweed/repo/oss/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(release):743 Request to release media - not attached; eject ''
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(removeAttachPoint):180 MediaHandler - checking if to remove attach point
2021-07-15 18:37:08 <1> host(6165) [zypp] PathInfo.cc(rename):710 rename /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm.new.zypp.07Wgg8 -> /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1504 done: /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm{- 0644 0/0 size 32670681}
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaHandler.cc(provideFile):1014 provideFile(./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm)
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaManager.cc(checkDesired):98 checkDesired(2): desired (cached)
2021-07-15 18:37:08 <1> host(6165) [zypp:fetcher] Fetcher.cc(validate):366 Checking job [/var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm] (2 checkers )
2021-07-15 18:37:08 <1> host(6165) [zypp] PathInfo.cc(hardlinkCopy):878 hardlinkCopy /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm -> /var/cache/zypp/packages/https-download.opensuse.org-560f2b9f/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [zypp++] MediaSetAccess.cc(releaseFile):88 Going to release file ./x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm from media number 1
2021-07-15 18:37:08 <1> host(6165) [zypp] PathInfo.cc(unlink):668 unlink /var/tmp/AP_0x8i7wPp/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [Progress++] ProgressData.cc(report):88 {#63|}END
2021-07-15 18:37:08 <1> host(6165) [zypp] RepoProvideFile.cc(provideFile):307 provideFile at /var/cache/zypp/packages/https-download.opensuse.org-560f2b9f/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
2021-07-15 18:37:08 <1> host(6165) [zypp] PackageProvider.cc(providePackage):513 provided Package (7059)gcc11-11.1.1+git400-1.4.x86_64(https-download.opensuse.org-560f2b9f) at /var/cache/zypp/packages/https-download.opensuse.org-560f2b9f/x86_64/gcc11-11.1.1+git400-1.4.x86_64.rpm
You may be experiencing this issue alone, because almost nobody uses download.opensuse.org with https nowadays. (Or am I wrong?) I tend to think that 'error 16' it is not a problem with libzypp, because it never happened to me when using https://mirrorcache.opensuse.org in .repo files. (and I use it a lot, locally and in CI) Today I configured Apache on download.opensuse.org to redirect all https requests to mirrorcache.opensuse.org and saw 'error 16' once. The same happened to my colleague in another environment. (I haven't found any traces of the error in Apache logs - nothing is logged so it wasn't a problem with occasional broken mirror. The same with zypper.log on client - no useful info was there, -vvv to zypper also wasn't helpful).
BUT, I think libzypp should provide a way to automatically retry such error when using https.
Side note: that error may be related to haproxy+apache as fedora is struggling with similar issue as well: https://bugzilla.redhat.com/show_bug.cgi?id=1690971
I haven't experienced Curl error 16
for several weeks using zypper with https, while before I did see it regularly. I think it was fixed somehow.
Me neither. Shall we close it and reopen if it happens again?
Just happened again:
Abrufen: kpat-21.08.3-1.1.x86_64.rpm .................................................................................................................................................................................................................................[Fehler]
Fehler beim Download (curl) für 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/x86_64/kpat-21.08.3-1.1.x86_64.rpm':
Fehlercode: 'Curl error 16'
Fehlermeldung: ''
Just happened again:
@ericschdt - thx for reporting just on be on the safe side: did you see the error on recent Tumbleweed? In other words - is there a chance that libcurl or related libraries were several months old on that machine, so the experienced problem might be already fixed recently? Or is that machine updated regularly and it is unlikely that libcurl was that old?
@ericschdt - thx for reporting just on be on the safe side: did you see the error on recent Tumbleweed?
Yes! Currently openSUSE Tumbleweed 20211106
In other words - is there a chance that libcurl or related libraries were several months old on that machine, so the experienced problem might be already fixed recently?
I currently use libcurl4 7.79.1-1.3
Or is that machine updated regularly and it is unlikely that libcurl was that old?
It is regularly updated (at least weekly).
Running openSUSE Tumbleweed 20211209
with curl 7.80.0 (x86_64-suse-linux-gnu) libcurl/7.80.0 OpenSSL/1.1.1l-fips zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 libidn2/2.3.2 libpsl/0.21.1 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 OpenLDAP/2.4.59 Release-Date: 2021-11-10
and encountering the error for the first time ever.
I am also running openSUSE Tumbleweed 20211209
with curl 7.80.0 (x86_64-suse-linux-gnu)
and encountering error for the first time. It occurs on zypper ref
for repositories Virtualization
, snappy
, and openSUSE:Tumbleweed
.
Also encountering this error on TW 20211210
@cglmrfreeman @dwymark @Nano-Ocelot it looks something triggered it today. Do you see the error only on .rpm or on other files as well? I see reddit complains about repomd.xml , which is kind of new problem as far as I understand https://www.reddit.com/r/openSUSE/comments/rgs6zs/keep_getting_curl_error_when_doing_zypper_dup/
@cglmrfreeman @dwymark @Nano-Ocelot it looks something triggered it today. Do you see the error only on .rpm or on other files as well? I see reddit complains about repomd.xml , which is kind of new problem as far as I understand https://www.reddit.com/r/openSUSE/comments/rgs6zs/keep_getting_curl_error_when_doing_zypper_dup/
I think it is the same error as that thread. I managed to finally get through and update to 20211213 so I'll have to confirm next time I update, but I was getting failures to zypper ref
repos with errors for repomd.xml. But after 5 or 6 tries finally refreshed all the repos. Then when I used zypper dup
, every 2-6 rpms would error 16 as well. I had to repeatedly retry to get through it all.
@cglmrfreeman @dwymark @Nano-Ocelot it looks something triggered it today. Do you see the error only on .rpm or on other files as well? I see reddit complains about repomd.xml , which is kind of new problem as far as I understand https://www.reddit.com/r/openSUSE/comments/rgs6zs/keep_getting_curl_error_when_doing_zypper_dup/
The error occurred for me for repomd.xml, among other files. I am including output from zypper ref
below:
Repository 'Main Repository (NON-OSS)' is up to date.
Repository 'Main Repository (OSS)' is up to date.
Repository 'Main Update Repository' is up to date.
Problem retrieving files from 'Virtualization'.
Download (curl) error for 'https://download.opensuse.org/repositories/Virtualization/openSUSE_Tumbleweed/repodata/repomd.xml':
Error code: Curl error 16
Error message:
Please see the above error message for a hint.
Skipping repository 'Virtualization' because of the above error.
Retrieving repository 'openSUSE:Tumbleweed' metadata ..................................................................................................................................................................................................................[error]
Repository 'openSUSE:Tumbleweed' is invalid.
[https-download.opensuse.org-c50dffcc|https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/] Valid metadata not found at specified URL
History:
- [|] Error trying to read from 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/'
- Download (curl) error for 'https://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/repodata/repomd.xml':
Error code: Curl error 16
Error message:
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE:Tumbleweed' because of the above error.
Retrieving repository 'snappy' metadata ...............................................................................................................................................................................................................................[error]
Repository 'snappy' is invalid.
[snappy|https://download.opensuse.org/repositories/system:/snappy/openSUSE_Tumbleweed] Valid metadata not found at specified URL
History:
- Download (curl) error for 'https://download.opensuse.org/repositories/system:/snappy/openSUSE_Tumbleweed/repodata/3d0df32a12706cc9c7378c8e7e230a12ec66fe8f941c3a6fa547859480798b82-primary.xml.gz':
Error code: Curl error 16
Error message:
- Can't provide ./repodata/3d0df32a12706cc9c7378c8e7e230a12ec66fe8f941c3a6fa547859480798b82-primary.xml.gz
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'snappy' because of the above error.
Retrieving repository 'vivaldi' metadata ...............................................................................................................................................................................................................................[done]
Building repository 'vivaldi' cache ....................................................................................................................................................................................................................................[done]
Some of the repositories have not been refreshed because of an error.
This error also occured twice in openQA in the same hour, with the same URL:
https://openqa.opensuse.org/tests/2083639#step/install_ltp/28 https://openqa.opensuse.org/tests/2083699#step/install_ltp/28
@Vogtinator @andrii-suse The openqa logs both show the same behavior (2021-12-13 18:58:48 and 19:44:19 for https://download.opensuse.org/repositories/benchmark:/ltp:/devel/openSUSE_Leap_15.3/x86_64/ltp-20211213.b48cd64ba-lp153.913.1.x86_64.rpm)
We got a metalink file for the package. The file contained no or too few mirrors to do a chunked download. We ask d.o.o. for the file again awaiting to be redirected. While this worked for a couple of files before (packages as well as repo metadata), it fails for ltp-20211213.
Maybe openqa can export ZYPP_MEDIA_CURL_DEBUG=2
in the environment for a while. It may help to see the http header and server responses in the zypper.log at the time the error occurs.
Maybe openqa can export ZYPP_MEDIA_CURL_DEBUG=2 in the environment for a while. It may help to see the http header and server responses in the zypper.log at the time the error occurs.
Can be done, for the install_ltp
test specifically. Though I wonder whether it'll happen in install_ltp
again at all... If it happens again there, I'll make a PR to add it.
For anyone thinking of running a script to automatically reload zipper on failures like this one from reddit, doing so didn't end up working for me as the curl failures just kept coming in hot and heavy. However what I did not realize when I ran this is that for each instance of zypper I ran, a new snapshot on my btrfs filesystem would be created, and my machine over time became very not happy with freezes and hangs. A sudo btrfs quota disable /
and sudo btrfs quota enable /
seemed to calm things down (at least in this boot instance) but I thought I would post a word of caution to mitigate anyone who might try this method after searching for a definitive way to update through the curl error 16s.
@cglmrfreeman Could you provide your /var/log/zypper.log? (per email to me is ok) I'd like to check whether the error exclusively occurs on connections to d.o.o. By npw I never saw it happen when connecting to a different server.
Regarding the snapper plugin taking snapshots I filed https://github.com/openSUSE/libzypp/issues/365.
Great! Not at home at the moment, but I will upload the log when I can!
Hello everyone, I think I have this problem too.
Here what I get when I want to refresh to cache of each repository:
❯ sudo zypper ref
[sudo] password for root:
Sorry, try again.
[sudo] password for root:
Repository 'database-x86_64' is up to date.
Repository 'Main Repository (NON-OSS)' is up to date.
Repository 'Main Repository (OSS)' is up to date.
Repository 'Main Update Repository' is up to date.
Repository 'openSUSE:Tumbleweed' is up to date.
Repository 'home:MasterPatricko' is up to date.
Problem retrieving files from 'server:database'.
Download (curl) error for 'https://download.opensuse.org/repositories/server:/database/openSUSE_Tumbleweed/repodata/repomd.xml':
Error code: Curl error 16
Error message:
Please see the above error message for a hint.
Skipping repository 'server:database' because of the above error.
Repository 'openSUSE-20211110-0' is up to date.
Repository 'packman' is up to date.
Repository 'spotify-easyrpm' is up to date.
Repository 'vscode' is up to date.
Some of the repositories have not been refreshed because of an error.
I would like to do a zypper dup
but I have some packages which cannot be dowloaded. Zypper give up when he cannot download the file 5 times.
If I set up this var ZYPP_MEDIA_CURL_DEBUG=2
I don't have any error anymore.
❯ ZYPP_MEDIA_CURL_DEBUG=2 sudo zypper ref
Repository 'database-x86_64' is up to date.
Repository 'Main Repository (NON-OSS)' is up to date.
Repository 'Main Repository (OSS)' is up to date.
Repository 'Main Update Repository' is up to date.
Repository 'openSUSE:Tumbleweed' is up to date.
Repository 'home:MasterPatricko' is up to date.
Repository 'server:database' is up to date.
Repository 'openSUSE-20211110-0' is up to date.
Repository 'packman' is up to date.
Repository 'spotify-easyrpm' is up to date.
Repository 'vscode' is up to date.
All repositories have been refreshed.
But I'm still unable to do an upgrade
Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/libopenssl1_1-1.1.1l-4.1.x86_64.rpm':
Error code: Curl error 55
Error message: Connection died, tried 5 times before giving up
Abort, retry, ignore? [a/r/i/...? shows all options] (a):
andrii-suse: You may be experiencing this issue alone, because almost nobody uses download.opensuse.org with https nowadays. (Or am I wrong?)
I've been seeing people complaining about this issue quite often. Even though we're using htpp://d.o.o
as our defaults on Tumbleweed and Leap (tested on 2 VM without edid repos), people, myself included, must be editing their repos to use https
. In our wiki there isn't any https
link in use too. Now YaST 1-Click
seems to allway add repos using https
.
Since around two months ago I've been running into this issue while either zypper ref
, zypper dup
, or zypper in
. The last two happen because by default Zypper refreshes the repos automatically.
zypper ref
failureI did a zypper ref
today and ran into Curl error 16
and 52
zypper ref
zypper ref -vvv
failurezypper ref -vvv
zypper -vvv ref operations in zypper.log
08:12 PM geeko@multiverse (~)# rpm -q libcurl4 curl librepo0 rpm zypper
libcurl4-7.80.0-1.1.x86_64
curl-7.80.0-1.1.x86_64
librepo0-1.14.2-1.2.x86_64
rpm-4.17.0-2.1.x86_64
zypper-1.14.50-1.1.x86_64
Edit: this comment is not relevant anymore as I was too optimisting thinking about root cause (and the issue triggered on Wednesday, before the changes I had in mind below). ~I've reverted some config changes on download.opensuse.org from last Thursday, (when the issue triggered with high frequency).~ ~My guess is that the error was related to that change and should gone now, but let me know if it you still see it (often). I guess the error itself is not related to zypper, but zypper might be friendlier handling around it (e.g. complain and automatically retry?).~ ~I suggest discussing here what zypper can do if server's reply misbehaves and discuss infrastructure issues e.g. in https://progress.opensuse.org/issues/104193~
I'm currently doing a TW dup (from TW 20211215, so pretty recent), and I've seen several error 16's, some 8+ times in the process of downloading 259 packages. Just (as in a minute ago) made it past the downloads, currently installing.
I only stumbled upon this issue after idly browsing for the error, so it's a bit late to grab more verbose logs, I'm afraid, but at least it's a datapoint.
Today I experienced the problem for first time in months - saw it twice. Then I replaced download.opensuse.org with mirrorcache.opensuse.org and zypper finished downloading 2000+ files without errors. As I mentioned earlier it looks some issue around Apache / haproxy : sudo sed -i 's,download.opensuse.org,mirrorcache.opensuse.org,g' /etc/zypp/repos.d/*.repo
Today I experienced the problem for first time in months - saw it twice. Then I replaced download.opensuse.org with mirrorcache.opensuse.org and zypper finished downloading 2000+ files without errors. As I mentioned earlier it looks some issue around Apache / haproxy :
sudo sed -i 's,download.opensuse.org,mirrorcache.opensuse.org,g' /etc/zypp/repos.d/*.repo
I wonder how close are we to switching mirrorbrain
with mirrorcache
, because if we are about to really switch chem we won't have to worry about this problem anymore, I reckon. Some people get concerned about using plain http
without TLS like we never used it before... And I don't wanna tell people to edit their repos to use mirrorcache. For now I'm telling them you either use http
like we have been using for ages or keep using https
and handle this Curl
error by retrying until it's sorted out.
Now I've heard that Flathub started to run into this issue as well.
For now I'm telling them you either use
http
like we have been using for ages or keep usinghttps
and handle thisCurl
error by retrying until it's sorted out.
Unlike normal errors, that one is not auto-retried and completely breaks unattended system updates which especially annoying when there are >4000 packages in queue. Also, only switching from https
to http
on download.opensuse.org
links just replaces error code 12 with error code 55.
wonder how close are we to switching
mirrorbrain
withmirrorcache
https is actually fully served by mirrorcache already, but the error happens not in mirrorcache as far as I understand. Replacing download.o.o with mirrorcache.o.o is just omitting Apache. https://progress.opensuse.org/issues/104193 claims that it may be related to overload, so I've temporarily redirected 33% of http *.rpm requests to mirrorcache , hoping that it will decrease overall load on Nuremberg servers
So zypper.log with ZYPP_MEDIA_CURL_DEBUG=2 : note that access_log on apache has libquadmath0-11.2.1+git1018-1.1.x86_64.rpm for my IP but doesn't have any reference of libpython2_7-1_0-2.7.18-14.1.x86_64.rpm for it - not sure the request wasn't even logged
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp] PackageProvider.cc(providePackage):513 provided Package (24924)libquadmath0-11.2.1+git1018-1.1.x86_64(download.opensuse.org-oss) at /var/cache/zypp/packages/download.opensuse.org-oss/x86_64/libquadmath0-11.2.1+git1018-1.1.x86_64.rpm
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp] PackageProvider.cc(providePackage):412 provide Package (24528)libpython2_7-1_0-2.7.18-14.1.x86_64(download.opensuse.org-oss)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] DeltaCandidates.cc(deltaRpms):82 package: (24528)libpython2_7-1_0-2.7.18-14.1.x86_64(download.opensuse.org-oss)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media] RepoProvideFile.cc(provideFile):241 [1]./x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm{788.3 KiB|sha256-0a5d762982aa75e188d5671fd97257b9d5ece6e0382b4f7e517b938d680e794e|}
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher++] Fetcher.cc(addCachePath):331 Adding fetcher cache: '/var/cache/zypp/packages/download.opensuse.org-oss'.
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media] RepoProvideFile.cc(provideFile):260 Added cache path /var/cache/zypp/packages/download.opensuse.org-oss
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media] RepoProvideFile.cc(provideFile):294 Providing file of repo 'download.opensuse.org-oss' from https://download.opensuse.org/tumbleweed/repo/oss/
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher] Fetcher.cc(downloadAndReadIndexList):705 No indexes to read.
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher] Fetcher.cc(locateInCache):362 start fetcher with 1 cache directories.
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher] Fetcher.cc(provideToDest):533 Not found in cache, retrieving...
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::fetcher++] MediaSetAccess.cc(provide):278 Going to try to provide file ./x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm from media number 1
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaManager.cc(checkDesired):154 checkDesired(1): desired (cached)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1335 dest: /var/tmp/AP_0xAJvxU8/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1336 temp: /var/tmp/AP_0xAJvxU8/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm.new.zypp.Ze9iDC
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1008 ./x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1018 URL: https://download.opensuse.org/tumbleweed/repo/oss/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Found bundle for host download.opensuse.org: 0x562a865dbb70 [can multiplex]
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Re-using existing connection! (#17) with host download.opensuse.org
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Connected to download.opensuse.org (2001:67c:2178:8::13) port 443 (#17)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Using Stream ID: d (easy handle 0x562a864acfe0)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > GET /tumbleweed/repo/oss/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm HTTP/2
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > Host: download.opensuse.org
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > user-agent: ZYpp 17.28.8 (curl 7.80.0)
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > x-zypp-anonymousid: 87f76fb3-72cd-4a58-adeb-61c7063d5ba3
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > x-zypp-distributionflavor: appliance-custom
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 > accept: */*, application/metalink+xml, application/metalink4+xml
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * stopped the pause stream!
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * Closing connection 17
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * TLSv1.3 (IN), TLS alert, close notify (256):
2021-12-21 18:19:29 <1> localhost.localdomain(607) [zypp++] CurlHelper.cc(log_curl):84 * TLSv1.3 (OUT), TLS alert, close notify (256):
2021-12-21 18:19:29 <3> localhost.localdomain(607) [zypp::media] MediaCurl.cc(doGetFileCopyFile):1083 curl error: 16: , temp file size 0 bytes.
2021-12-21 18:19:29 <5> localhost.localdomain(607) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaCurl.cc(evaluateCurlCode):699 THROW: Download (curl) error for 'https://download.opensuse.org/tumbleweed/repo/oss/x86_64/libpython2_7-1_0-2.7.18-14.1.x86_64.rpm'
http log:
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp] PackageProvider.cc(providePackage):513 provided Package (43066)python38-SQLAlchemy-1.4.27-1.1.x86_64(download.opensuse.org-oss) at /var/cache/zypp/packages/download.opensuse.org-oss/x86_64/python38-SQLAlchemy-1.4.27-1.1.x86_64.rpm
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp] PackageProvider.cc(providePackage):412 provide Package (31066)mpg123-pulse-1.29.3-1.1.x86_64(download.opensuse.org-oss)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] DeltaCandidates.cc(deltaRpms):82 package: (31066)mpg123-pulse-1.29.3-1.1.x86_64(download.opensuse.org-oss)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media] RepoProvideFile.cc(provideFile):241 [1]./x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm{32.1 KiB|sha256-efbb09146f3437b2a77266750f7c2fd288f3978d1a299b5918cb6fc7c7f9106f|}
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher++] Fetcher.cc(addCachePath):331 Adding fetcher cache: '/var/cache/zypp/packages/download.opensuse.org-oss'.
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media] RepoProvideFile.cc(provideFile):260 Added cache path /var/cache/zypp/packages/download.opensuse.org-oss
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media] RepoProvideFile.cc(provideFile):294 Providing file of repo 'download.opensuse.org-oss' from http://download.opensuse.org/tumbleweed/repo/oss/
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher] Fetcher.cc(downloadAndReadIndexList):705 No indexes to read.
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher] Fetcher.cc(locateInCache):362 start fetcher with 1 cache directories.
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher] Fetcher.cc(provideToDest):533 Not found in cache, retrieving...
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::fetcher++] MediaSetAccess.cc(provide):278 Going to try to provide file ./x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm from media number 1
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaManager.cc(checkDesired):154 checkDesired(13): desired (cached)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1335 dest: /var/tmp/AP_0xfxSKSC/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1336 temp: /var/tmp/AP_0xfxSKSC/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm.new.zypp.711Biw
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1008 ./x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1018 URL: http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Found bundle for host download.opensuse.org: 0x561430b3dfc0 [serially]
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Can not multiplex, even if we wanted to!
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Re-using existing connection! (#13) with host download.opensuse.org
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Connected to download.opensuse.org (195.135.221.134) port 80 (#13)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > GET /tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm HTTP/1.1
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > Host: download.opensuse.org
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > User-Agent: ZYpp 17.28.8 (curl 7.80.0)
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > X-ZYpp-AnonymousId: 87f76fb3-72cd-4a58-adeb-61c7063d5ba3
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > X-ZYpp-DistributionFlavor: appliance-custom
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 > Accept: */*, application/metalink+xml, application/metalink4+xml
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp++] CurlHelper.cc(log_curl):84 * Closing connection 13
2021-12-21 19:01:12 <3> localhost.localdomain(21118) [zypp::media] MediaCurl.cc(doGetFileCopyFile):1083 curl error: 55: Connection died, tried 5 times before giving up, temp file size 0 bytes.
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaCurl.cc(evaluateCurlCode):699 THROW: Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaCurl.cc(doGetFileCopyFile):1099 RETHROW: Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaMultiCurl.cc(doGetFileCopy):1364 RETHROW: Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [DEFINE_LOGGROUP] PathInfo.cc(unlink):706 unlink /var/tmp/AP_0xfxSKSC/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm.new.zypp.711Biw
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaCurl.cc(getFileCopy):528 RETHROW: Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 MediaSetAccess.cc(provide):288 CAUGHT: Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/mpg123-pulse-1.29.3-1.1.x86_64.rpm':
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error code: Curl error 55
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166 Error message: Connection died, tried 5 times before giving up
2021-12-21 19:01:12 <5> localhost.localdomain(21118) [DEFINE_LOGGROUP] Exception.cc(log):166
2021-12-21 19:01:12 <1> localhost.localdomain(21118) [zypp::media++] MediaHandler.cc(getDetectedDevices):1384 No devices for this medium
So, as stated in the progress issue, the bottom of all this could be that rsync
is not catching up with the list of files to be mirrored and mirrorbrain
gets "confused" about the file-to-be-mirrored-yet, curl
try to fetch the file, but doesn't find it on any mirrors and tries to fetch the file direct on the "central" servers, and maybe because the servers in Nuremberg are overloaded, curl
errors out?
I have the feeling I'm mistaken about how the whole cycle works. If someone can shed some light on this, please do.
both mirrorbrain and mirrorcache do redirect to mirrors properly from what I see. It looks cached connections to download.o.o die from time to time and zypper overreacts giving up completely. It should just try to re-establish new connection, but it doesn't (the way I read the log). Or maybe it just should wait some time before retrying, but the log just says that it gives up immediately. I wonder if anything changed recently around that code.
both mirrorbrain and mirrorcache do redirect to mirrors properly from what I see. It looks cached connections to download.o.o die from time to time and zypper overreacts giving up completely. It should just try to re-establish new connection, but it doesn't (the way I read the log). Or maybe it just should wait some time before retrying, but the log just says that it gives up immediately. I wonder if anything changed recently around that code.
Now I see! Indeed, the log shows it doesn't really retry.
https://github.com/curl/curl/issues/7400 looks similar, but that's apparently fixed in 7.79.0 (TW has 7.80.0).
It could still be an issue in curl. Was this ever observed outside of Tumbleweed?
curl/curl#7400 looks similar, but that's apparently fixed in 7.79.0 (TW has 7.80.0).
It could still be an issue in curl. Was this ever observed outside of Tumbleweed?
The issue has also been reportetd for Leap 15.3: https://forums.opensuse.org/showthread.php/564178-update-curl-error-16?p=3093019#post3093019
curl/curl#7400 looks similar, but that's apparently fixed in 7.79.0 (TW has 7.80.0).
It could still be an issue in curl. Was this ever observed outside of Tumbleweed?
+1 here. The issue was closed with:
.. I'm going to go ahead and merge the revert and we'll keep our eyes open and see if we run into any problems going forward.
After reading the issue we can see the whole thing wasn't something trivial.
Yeah the worst thing is that it is not clear who to blame: libzypp vs libcurl vs infrastructure behind download.o.o and it is not clear why the issue occurrences did skyrocket on 15-Dec . It looks infrastructure is partially to blame because connection is closed, but it is not clear why libcurl(?) doesn't put any effort in retry (like make a short pause between attempts).
Yeah the worst thing is that it is not clear who to blame: libzypp vs libcurl vs infrastructure behind download.o.o and it is not clear why the issue occurrences did skyrocket on 15-Dec . It looks infrastructure is partially to blame because connection is closed, but it is not clear why libcurl(?) doesn't put any effort in retry (like make a short pause between attempts).
Well. Maybe we can take libzypp out of this (maybe?) since even the curl
command runs into the same issue. From the forum posted earlier, after the guy ran into Error code: Curl error 16
during zypper dup
he tried:
linux-4skc:~ # curl -vvvIL 'https://download.opensuse.org/repositories/openSUSE:/Factory/snapshot/media.1/media'
* Trying 2001:67c:2178:8::13:443...
* Trying 195.135.221.134:443...
* Connected to download.opensuse.org (195.135.221.134) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=opensuse.org
* start date: Oct 29 00:00:05 2021 GMT
* expire date: Jan 27 00:00:04 2022 GMT
* subjectAltName: host "download.opensuse.org" matched cert's "*.opensuse.org"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55c06be4ce00)
> HEAD /repositories/openSUSE:/Factory/snapshot/media.1/media HTTP/2
> Host: download.opensuse.org
> user-agent: curl/7.80.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Empty reply from server
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server
linux-4skc:~ #
I just can't understand what's going on here, the connection using HTTP/2 is made by curl
but in the end it's being refused, or something else goes wrong here?! (Anyone?)
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55c06be4ce00)
> HEAD /repositories/openSUSE:/Factory/snapshot/media.1/media HTTP/2
> Host: download.opensuse.org
> user-agent: curl/7.80.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Empty reply from server
* Closing connection 0
We know that the infrastructure ('mirrorbrain') is a bit clueless about many (I don't know to what extent here) files that are not being mirrored in a timely manner, sometimes mirrors lag behind days, and it thinks that the files are already being mirrored. And the list of "ghost" files grows the newer the file is, according to the progress.o.o issue ticket.
So could it be that the issues with the infra is exposing the issues with curl
? Does my reasoning makes sense? The more I go back to that Curl upstream issue and continue to (re)read the more I smell trouble.
One of the crossing issues that was mentioned there, was the curl/curl#6862 where someone ran into a problem when curl
tried to establish a connection over HTTP/2, their server replied with HTTP_1_1_REQUIRED
and then requiring a new connection. The problem was that curl
was retrying over HTTP/2, again, instead of HTTP/1.1. The problem was apparently fixed.
If you read the whole thing it'll make more sense to you than it made to me, I'm sure.
Would it be possible to roll my own curl? I tried building from git but it lacked many of the protocols and features compared to the openSUSE:Factory
vendor provides.
curl
command runs into the same issue
That is actually a valuable observation, (I've tried curl and it always succeeded).
infrastructure ('mirrorbrain') is a bit clueless
My strong understanding is that issue happens before mirrorbrain/mirrorcache and it is not even present in Apache's logs. It might be some ha-proxy bug or other transport issue, but I doubt it is a Redirector to blame. Especially that Apache is configured to return files below 2K (like /media.1/media ) without involving a redirector.
But it looks you are on the right track, still not sure how to fix it yet.
Would it be possible to roll my own curl? I tried building from git but it lacked many of the protocols and features compared to the
openSUSE:Factory
vendor provides.
I'm afraid my friend that it would be too much trouble for you to get a previous version of an "OBS" curl
working in a recent version of Tumbleweed.
My strong understanding is that issue happens before mirrorbrain/mirrorcache and it is not even present in Apache's logs. It might be some ha-proxy bug or other transport issue, but I doubt it is a Redirector to blame. Especially that Apache is configured to return files below 2K (like /media.1/media ) without involving a redirector.
Interesting, so even with the redirector out of the game the issue keeps popping up.
From the upstream Curl
issue, Alex Crichton said that in his bisection of commits experimentation, after pinpointing the culprit, and making Cargo
download its own dependencies 100x, the download would the fail spuriously, but before the failure it would succeed at least 100 times.
So, in order to gather more info, we could setup a testbed using Curl
to loop though less-than-2k files downloading there headers, and abusing of verbosity/debugging facilities. I just need to know if there's a safe way of doing this, because I can't say if I let a script like this running for a while it could trigger some kind of response from the servers like "Hey, this dude here is doing some funny business and I don't like it, so let's block his requests.".
If anyone has some input on this or even a better idea please share with us. Maybe I should hit our mailing lists too.
I'm afraid my friend that it would be too much trouble for you to get a previous version of an "OBS" curl working in a recent version of Tumbleweed.
Well theoretically if I wanted to do that (even though finding old RPMs is difficult) I would probably find an older ISO of Tumblweed and extract it from that instead of building it from scratch. I was thinking more along the lines of building 7.81 from git, but after looking at OBS, there is some interesting bandaids and patches applied. I am not familiar enough with the OBS system to clone it and have the OBS system test/build the RPM.
I am assuming I could clone the git, apply all the patches and diffs manually and make sure all these options are used -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g
as per the output of my rpm -q --queryformat="%{NAME}: %{OPTFLAGS}\n" curl
I am just not 100% sure this actually solves anything.
EDIT: For now I've changed repos to mirrorcache-us
, however openSUSE:Factory
, openSUSE:Factory:NonFree
and openSUSE:Tools:openSUSE_Factory
do not have repos under that subdomain.
So, I made some experiments with a Tumbleweed VM that needed a dup
(it was sitting a couple of snapshots behind).
First thing I did, or second I can't remember, was to edit /etc/zypp/zypp.conf
and set download.max_silent_tries = 50
and rebooted the VM, just to be sure. On the first zypper ref
I tried it runs into the curl error 16 and says it tried 5 times. So it's confirmed, the part of the code that handles the automatic retrying is not working like it should. And zypper dup
ends up with same error 16.
Next thing I did was to install DNF, on my standard system, with:
# zypper in dnf libdnf-repo-config-zypp
It already came with some handy zypper aliases then I issued dnf ref
and dnf dup
. Several package downloads were plagued:
[MIRROR] libthai-data-0.1.29-52.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opens
use.org/repositories/GNOME:/Factory/openSUSE_Factory/x86_64/libthai-data-0.1.29-52.1.x86_64.rpm []
(33/249): libjavascriptcoregtk-4_1-0-2.34.3-340.2.x86_64.rpm 1.5 MB/s | 6.5 MB 00:04
(34/249): libthai-data-0.1.29-52.1.x86_64.rpm 152 kB/s | 191 kB 00:01
...
[MIRROR] blender-3.0.0-2.2.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org
/tumbleweed/repo/oss/x86_64/blender-3.0.0-2.2.x86_64.rpm []
(66/249): apparmor-utils-3.0.3-7.1.noarch.rpm 145 kB/s | 145 kB 00:00
(67/249): blender-cycles-devel-3.0.0-2.2.noarch.rpm 512 kB/s | 559 kB 00:01
(68/249): blender-demo-3.0.0-2.2.noarch.rpm 428 kB/s | 1.2 MB 00:02
(69/249): MozillaThunderbird-91.4.1-1.1.x86_64.rpm 3.3 MB/s | 58 MB 00:17
(70/249): chrony-4.2-1.1.x86_64.rpm 178 kB/s | 296 kB 00:01
(71/249): chrony-pool-openSUSE-4.2-1.1.noarch.rpm 17 kB/s | 33 kB 00:01
(72/249): dirmngr-2.3.4-1.1.x86_64.rpm 370 kB/s | 343 kB 00:00
(73/249): efibootmgr-17-2.1.x86_64.rpm 18 kB/s | 45 kB 00:02
(74/249): blender-3.0.0-2.2.x86_64.rpm 2.3 MB/s | 42 MB 00:17
...
[MIRROR] gimp-plugins-python-2.10.30-1.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://downloa
d.opensuse.org/tumbleweed/repo/oss/x86_64/gimp-plugins-python-2.10.30-1.1.x86_64.rpm []
(77/249): gimp-2.10.30-1.1.x86_64.rpm 4.8 MB/s | 13 MB 00:02
(78/249): gimp-plugins-python-2.10.30-1.1.x86_64.rpm 173 kB/s | 305 kB 00:01
...
[MIRROR] gpg2-2.3.4-1.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org/tu
mbleweed/repo/oss/x86_64/gpg2-2.3.4-1.1.x86_64.rpm []
(79/249): gpg2-2.3.4-1.1.x86_64.rpm 702 kB/s | 2.3 MB 00:03
...
[MIRROR] libyui-ncurses15-4.2.24-1.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org/tumbleweed/repo/oss/x86_64/libyui-ncurses15-4.2.24-1.1.x86_64.rpm []
[FAILED] libyui-ncurses15-4.2.24-1.1.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
(177-178/249): libvirt-libs-7.10.0-2.2.x8 87% [===================================- ] 3.5 MB/s | 608 MB 00:25 ETA
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
libyui-ncurses15-4.2.24-1.1.x86_64: Cannot download, all mirrors were already tried without success
...
[MIRROR] qemu-seabios-1.15.0_0_g2dd4b9b-37.1.noarch.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org/tumbleweed/repo/oss/noarch/qemu-seabios-1.15.0_0_g2dd4b9b-37.1.noarch.rpm []
(223/249): qemu-ovmf-x86_64-202111-1.1.noarch.rpm 3.9 MB/s | 16 MB 00:04
(224/249): qemu-seabios-1.15.0_0_g2dd4b9b-37.1.noarch.rpm 136 kB/s | 414 kB 00:03
...
[MIRROR] vim-8.2.3863-1.1.x86_64.rpm: Curl error (16): Error in the HTTP2 framing layer for https://download.opensuse.org/
tumbleweed/repo/oss/x86_64/vim-8.2.3863-1.1.x86_64.rpm []
(238/249): typelib-1_0-SpiceClientGtk-3_0-0.39-3.1.x86_64.rpm 13 kB/s | 33 kB 00:02
(239/249): vim-8.2.3863-1.1.x86_64.rpm
As you can see, only one package past the 10 retries threshold DNF has by default:
[FAILED] libyui-ncurses15-4.2.24-1.1.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
...
Error: Error downloading packages:
libyui-ncurses15-4.2.24-1.1.x86_64: Cannot download, all mirrors were already tried without success
And this caused the whole transaction to fail. But the next dnf dup
managed to download the remaining packages without hitting the threshold again.
Back at the Tumbleweed VM, I installed DNF and dup
ed the system. As the VM was using original repo files with plain HTTP it thrown an "Error 502" at some point the retry threshold was past.
After that, I set retries=25
on /etc/dnf/dnf.conf
file and dup
again, after rolling back the VM, DNF hastily downloaded the almost 1K packages.
My last testsuite was to use HTTPS with retries=25
(DNF manual, dnf.conf section), the overall download time it took was quite longer then HTTP, I'm judging because of the TLS stuff, I don't know. Nevertheless I could see that at several packages DNF would take a while to download it (some took almost 15 minutes, but not be of download speed) I'm sure because of the retries it does and that the default timeout for package downloads are 30 seconds (DNF manual section). Of course DNF is smart enough to download the next package in the queue. With at least 25 retries as threshold people can leave the system unattended, I reckon, since a tried 2 HTTPS dup
and both went successful.
I hope all of this helps someone to get that little spark one needs to light up his lamp.
I hope all of this helps someone to get that little spark one needs to light up his lamp.
Thank you ! Haven't had visible errors after setting up dnf
. But, even more pleasing, it is noticeably faster in both operation and downloads than zypper
. Probably because multi-download and multi-installation are actually working. But I had to tweak /etc/dnf/dnf.conf
to be slightly more usable:
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=False
skip_if_unavailable=True
retries=15
timeout=5
max_parallel_downloads=4
metadata_timer_sync=60
metadata_expire=600
minrate=100k
fastestmirror=True
allow_vendor_change=True
debuglevel=3
errorlevel=4
#exclude_from_weak_autodetect=False
install_weak_deps=False
log_compress=True
The only gripe I have for now is it not showing current version in the pre-update table to compare it with target. Change in vendor repos also not as obvious as with zypper.
Thank you ! Haven't had visible errors after setting up dnf. ...
I'm glad it helped you :+1:.
Just letting you guys know that I'm building older versions of Curl to pinpoint what release brought this issue down on us. I'm currently running 7.74.0 in a VM with ~1K packages to upgrade, the first dup
went without any hiccups, now I'm gonna let it dup
maybe 5 times to be sure, use both HTTP and HTTPS, if I don't see any curl errors I'll go to the next release.
Thank you ! Haven't had visible errors after setting up dnf. ...
I'm glad it helped you +1.
Just letting you guys know that I'm building older versions of Curl to pinpoint what release brought this issue down on us. I'm currently running 7.74.0 in a VM with ~1K packages to upgrade, the first
dup
went without any hiccups, now I'm gonna let itdup
maybe 5 times to be sure, use both HTTP and HTTPS, if I don't see any curl errors I'll go to the next release.
It could be a coincidence, but since 66 IPs that caused a lot of traffic at download.opensuse.org have been blocked according to https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/message/YLPRFJOIN6HKTTGFY5P4N3PXSY3AG73K/, the "Curl error 16" message is gone and today's update to TW20220101 went like a charm.
Sorry, the correct link reads: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/message/3RDUQFNL33ZVMRGQ6MLTI2WLGKWS3SY5/
It could be a coincidence, but since 66 IPs that caused a lot of traffic at download.opensuse.org have been blocked according to https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/message/YLPRFJOIN6HKTTGFY5P4N3PXSY3AG73K/, the "Curl error 16" message is gone and today's update to TW20220101 went like a charm.
Well, I'm still getting the error here using HTTPS. Maybe the HTTPS requests switch to mirrorcache two weeks ago have some influence here? I'm judging you're in Europe, am I wrong?
It could be a coincidence, but since 66 IPs that caused a lot of traffic at download.opensuse.org have been blocked according to https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/message/YLPRFJOIN6HKTTGFY5P4N3PXSY3AG73K/, the "Curl error 16" message is gone and today's update to TW20220101 went like a charm.
Well, I'm still getting the error here using HTTPS. Maybe the HTTPS requests switch to mirrorcache two weeks ago have some influence here? I'm judging you're in Europe, am I wrong?
Using https either with download.o.o or mirrorcache-eu.o.o doesn't make any difference, as it should I guess.
Issue
I rather often encounter
on a wired stable internet connection on openSUSE TW when executing
zypper refresh && zypper dup
.Possible solutions
This can be easily resolved by pressing "w" for "retry". Unfortunately, this needs manual observation of the downloads -- so not a good solution.
There could be two possible solutions at the first glance:
error 16
errors.About the error
Source: https://curl.se/libcurl/c/libcurl-errors.html
Next time I will try to add the
--verbose
flag in order to hopefully get more information.