search

openSUSE / zypper

World's most powerful command line package manager
http://en.opensuse.org/Portal:Zypper
Other
405 stars 110 forks source link

Fixing data processing because of reported error “CURLE_HTTP2_STREAM (92)” #457

Open elfring opened 2 years ago

elfring commented 2 years ago

Some software users (including me) became affected in special ways by known software evolution. :eyes: Thus I am looking for solutions also according to the following information which was presented during a distribution upgrade.

Download (curl) error for 'https://download.opensuse.org/debug/tumbleweed/repo/oss/x86_64/xrefresh-debuginfo-1.0.7-2.2.x86_64.rpm':
Error code: Curl error 92
Error message: HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
mlandres commented 2 years ago

Those are issues in the communication between libcurl and the server. Similar to the other ones mentioned in your links (16) Error in the HTTP2 framing layer (52) Empty reply from server

Whether it is related to the issues d.o.o had in the last time or due to a bad mirror advertised is hard to tell. Could you provide a zypper.log showing the issue (EMail to me is fine)?

elfring commented 2 years ago
mlandres commented 2 years ago

@elfring the error itself is probably nothing zypp itself can influence. But in the log I hope to see where exactly the error occurs. Whether it is in the direct communication with d.o.o or in communication with one of the mirror sites we were redirected to.

Basically I'd like to see the information that is missing in zyppers error report on the screen. The URL names the entry point at d.o.o, but due to redirects this is not necessarily the server we were talking to. The report needs to clearly state the server we were connected to. This is something we will enhance and which may help in the future to figure out where the error is located. Whether it's always the same broken or overloaded mirror, or completely random, which would hint to an issue on the local system - somewhere between libcurl and the kernel.

Currently this information is buried in the zypper.log. Until the error reporting is enhanced, I can just try to extract the info from logs provided in error reports. A grep -i 'curl' zypper.log is roughly the information I'm interested in. It tells how the (failed) requests were handled and I can compare the involved servers with those from other reports.

elfring commented 2 years ago

Basically I'd like to see the information that is missing in zyppers error report on the screen.

Would you like to help with improving error/exception handling for the involved software libraries? :thinking:

This is something we will enhance and which may help in the future to figure out where the error is located.

Can advanced system tests help any more within the openQA infrastructure? :thinking:

mlandres commented 2 years ago

Basically I'd like to see the information that is missing in zyppers error report on the screen.

Would you like to help with improving error/exception handling for the involved software libraries? thinking

I'm maintainer of zypper/libzypp.

This is something we will enhance and which may help in the future to figure out where the error is located.

Can advanced system tests help any more within the openQA infrastructure? thinking

No, IMO not in this case. Those kind of errors are often triggered by overloded servers or package loss on the network. That's hard to test on QA.

elfring commented 2 years ago

Would you like to help with improving error/exception handling for the involved software libraries? :thinking:

I'm maintainer of zypper/libzypp.

:thought_balloon: Would you support any further evolution also according to the software “libcurl”?

That's hard to test on QA.

:crystal_ball: Will software analysis approaches evolve accordingly?

elfring commented 2 years ago

A grep -i 'curl' zypper.log is roughly the information I'm interested in.

I can extract some data by a command variant like the following.

Sonne:~ # xzegrep '^2022-09-02 .+[Cc]url' /var/log/zypper.log-20220903.xz > /home/altes_Heim2/elfring/Projekte/zypper/zypper_log-curl-excerpt-20220902b.txt

:thought_balloon: But I imagine that an other search pattern will become more helpful than to fiddle with a corresponding file of the size “30.9 MiB”.

luc14n0 commented 1 year ago

I also have ran into various incarnations of Curl error (16, 43, 52, 92) since Oct 2021 (if memory serves). I'd be glad to share some log data if needed.

P.S.: I started to use mirrorcache-us.o.o (I'm far away from EU) instead for quite a while now so most of the logs wont have requests going through download.o.o though.

elfring commented 1 year ago

:crystal_ball: Would you get into the development mood to reconsider error handling approaches any more for affected (software) components?

mlandres commented 1 year ago

@luc14n0 Did switching to mirrorcache-us.o.o solve the curl error issues for you? If yes, we should to focus on relieving d.o.o. If not, a log showing a few failed downloads is welcome. Sending the output of grep -i 'curl' /var/log/zypper.log after some download errors happened would be fine (per mail to ma(at)suse.com). Thanks.

luc14n0 commented 1 year ago

@luc14n0 Did switching to mirrorcache-us.o.o solve the curl error issues for you? If yes, we should to focus on relieving d.o.o.

So sorry for the late reply, the notification got buried in my inbox.

Short answer is no, BUT it reduced it dramatically. Comparing it to when I was using download.o.o it's almost like it's gone.

For a longer answer: https://github.com/openSUSE/zypper/issues/399#issuecomment-1261828475

If not, a log showing a few failed downloads is welcome. Sending the output of grep -i 'curl' /var/log/zypper.log after some download errors happened would be fine (per mail to ma at s.c). Thanks.

Sure, I can unbury some of those from my rotation-logs and send you some samples.

mlandres commented 1 year ago

The logs we got so far (also thanks to Luciano) hint to downloadcontent*.opensuse.org. By now we see the issue (92) only if we are redirected to these mirrors. We try to investigate this on the servers side, but no result so far.

The next libzypp-17.31.7 will auto retry on Curl error 92.

Also available in libzypp-17.31.7 will be an alternative media backend. It can be enabled by setting an environment variable:

ZYPP_MEDIANETWORK=1 zypper ....

or in /etc/zypp/zypp.conf:

[main] techpreview.ZYPP_MEDIANETWORK=1

Maybe you like to give it a try

elfring commented 1 year ago

:thought_balloon: Do you track how “good” the current distribution upgrade is running (also according to another attempt for updating 10105 software packages for example)?

mlandres commented 1 year ago

@elfring If you are asking whether the d.o.o/mirrocache team is aware that d.o.o. is overloaded due to the tw update, they are. They receive the reports about the increasing number of HTTP/2 framing errors from inside and outside SUSE. And they are trying to fight it.

elfring commented 1 year ago

And they are trying to fight it.

:crystal_ball: Will this trigger further collateral evolution?

mlandres commented 1 year ago

Please update to libzypp-17.31.13 as soon as it is available.

Both together results in Curl error 92: HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1).