Hang with update

[sailingd]

Affected Version

Occurs with yay and yay-bin v3.440 and yay-git v3.440.r25.g4088d5d

Issue

Seeing what I think is same bug as #200 and hanging at Searching AUR for updates (Querying AUR... in git version).. started after upgrade from 3.373-1 to 3.440-1

Steps to reproduce

yay -Su causes hang at Searching AUR for updates... No CPU usage. All other operations besides update (-Su) seem fine.

Compiled with debug and I get the following stack trace after the hang when I SIGKILL if this is useful.

PC=0x45c319 m=0 sigcode=128

goroutine 0 [idle]: runtime.epollwait(0x4, 0x7ffe980abf20, 0xffffffff00000080, 0xfffffffc, 0xffffffff0077ea4a, 0xfffffffc, 0x0, 0x0, 0x0, 0x0, ...) /usr/lib/go/src/runtime/sys_linux_amd64.s:664 +0x19 runtime.netpoll(0xc420026f01, 0xc420026f01) /usr/lib/go/src/runtime/netpoll_epoll.go:71 +0x12c runtime.findrunnable(0xc420024a00, 0x0) /usr/lib/go/src/runtime/proc.go:2392 +0x65b runtime.schedule() /usr/lib/go/src/runtime/proc.go:2536 +0x13b runtime.park_m(0xc4203b8600) /usr/lib/go/src/runtime/proc.go:2599 +0xb6 runtime.mcall(0x65e650) /usr/lib/go/src/runtime/asm_amd64.s:351 +0x5b

goroutine 1 [semacquire, 2 minutes]: sync.runtime_Semacquire(0xc4202ec76c) /usr/lib/go/src/runtime/sema.go:56 +0x39 sync.(*WaitGroup).Wait(0xc4202ec760) /usr/lib/go/src/sync/waitgroup.go:129 +0x72 main.aurInfo(0xc420301980, 0x193, 0x193, 0x193, 0xc4202f0fb8, 0x205a8d0, 0x4, 0xc4202ec738) /home/sailingd/src/yay/src/yay-3.440/query.go:358 +0x36b main.depTreeRecursive(0xc420146620, 0xc4202eeb20, 0x205ac30, 0x205a8d0, 0xc4202fa000, 0x205a8d0, 0x72e4e0) /home/sailingd/src/yay/src/yay-3.440/dependencies.go:334 +0x197 main.getDepTree(0xc420300000, 0x195, 0x198, 0x24, 0x0, 0x0) /home/sailingd/src/yay/src/yay-3.440/dependencies.go:276 +0x284 main.install(0xc420088db0, 0xc420055cd0, 0x2) /home/sailingd/src/yay/src/yay-3.440/install.go:51 +0x550 main.handleSync(0xc42009a430, 0x1) /home/sailingd/src/yay/src/yay-3.440/cmd.go:480 +0x3a3 main.handleCmd(0x0, 0x0) /home/sailingd/src/yay/src/yay-3.440/cmd.go:312 +0xaec main.main() /home/sailingd/src/yay/src/yay-3.440/cmd.go:223 +0x25d

rax 0xfffffffffffffffc rbx 0xffffffff rcx 0x45c319 rdx 0x80 rdi 0x4 rsi 0x7ffe980abf20 rbp 0x7ffe980ac520 rsp 0x7ffe980abee0 r8 0x3 r9 0xa83d20 r10 0xffffffff r11 0x246 r12 0xffffffffffffffff r13 0xf6 r14 0xf5 r15 0x200 rip 0x45c319 rflags 0x246 cs 0x33 fs 0x0 gs 0x0

####################################

yay -Qqm output:

abs acd_cli acroread akonadi-socialutils alock-svn anaconda android-sdk-platform-tools archlinux-artwork asoundconf avant-window-navigator avogadro bcg729 bindiff binfmt-qemu-static blogilo bsg-diaspora-sa cartoonizer-gimp-plugin castnow-git celestia-addon-international_space_station celestia-addon-sun celestia-addon-voyagers chromium-widevine ckmame clamz clrmamepro clutter-gst2 codecs64 cower cutecom dhewm3 discord disper dockbarx dockmanager dolphinpart4 doom3 doom3-data downgrader dropbox duff dwm dxpc dynagen dynamips e_dbus ec2-ami-tools ec2-api-tools eclipse-pydev eigen2 elementary-icons elementary-xfce-icons evhz-git exe-thumbnailer f3 fdupves-git firmware-mod-kit flightgear flightgear-data flowcanvas fluidr3 focuswriter frame freeradius-client gdatafs gdk-pixbuf geis ghostwriter gi gigolo gimp-extras gimp-plugin-refocusit gimp-plugin-resynthesizer gimp-script-smart-remove gkrellm-gkfreq glib gnofract4d gnome-breeze-git gnome-desktop2 gnome-menus2 gnome-nds-thumbnailer gnome-shell-extension-pixel-saver-git gnome-vfs gnome-xcf-thumbnailer gns3-converter gns3-gui gns3-server gnuplot-py gnustep-base gnustep-make goffice0.8 goocanvas1 google-earth google-talkplugin gpgmepp grail grantlee-qt4 graylog grip gst-python2-git gstreamer0.10 gstreamer0.10-bad gstreamer0.10-bad-plugins gstreamer0.10-base gstreamer0.10-base-plugins gstreamer0.10-ffmpeg gstreamer0.10-good gstreamer0.10-good-plugins gstreamer0.10-ugly gstreamer0.10-ugly-plugins gtk gtk-engine-aurora gtk-engine-unico gtk-theme-config gtkglarea1 gtkhtml4 gtkpacman hardening-wrapper heif hsqldb-java ht htdig icaclient icewm-themes ida-free ido ike-scan ioquake3 ioquake3-xcsv ipad_charge isl jitsi jperf js17 katepart4 kcm-qt-graphicssystem kdeaccessibility-jovie kdeaccessibility-kaccessible kdebase-kdepasswd kdebase-lib kdegraphics-ksaneplugin kdemultimedia-kscd kdemultimedia-mplayerthumbs kdenetwork-kppp kdesdk-strigi-analyzers kdewebdev-kfilereplace kdewebdev-kommander kdgantt2 kfilemetadata4 kirigami konsolepart4 lcdtest ld-lsb lemonbar lha lib32-js17 lib32-json-c lib32-libmng lib32-libtxc_dxtn lib32-libxaw lib32-libxp lib32-libxpm lib32-openmotif lib32-pangox-compat lib32-sdl_net lib32-sdl_sound libbonobo libbonoboui libc++ libc++abi libc++experimental libdesktop-agnostic libee libestr libgames-support libgee06 libgnome libgnome-data libgnome-desktop-sharp libgnome-media-profiles libgnomecanvasmm libgnomeui libical2 libkactivities4 libkcddb4 libkcompactdisc4 libkdcraw4 libkdegames4 libkexiv2_4 libkface4 libkipi4 libkolab libksane4 libkscreen4 liblogging libmowgli libmusicbrainz3 liboil libpar2 libqzeitgeist libsocialweb libtracker-sparql libtxc_dxtn libvlcpp-git libwebcam libxfcegui4 libzeitgeist magicrescue makemkv mash mbrowse menulibre mime-editor minecraft mingw32-binutils mingw32-gcc mingw32-pthreads mingw32-runtime mingw32-w32api miraclecast-git mkbold-mkitalic mkchromecast moonlight mprime mrtg mugshot multipath-tools musicbrainz namebench ncompress netexpect netkit-bsd-finger nexuiz normalize nouveau-fw numix-circle-icon-theme-git numix-icon-theme-git nvidia-docker nx-common obex-data-server openarena openarena-data opennx openobex openslp orbit2 otf-stix owncloud p7zip-extra pacaur pacmixer perl-config-general perl-crypt-rc4 perl-digest-perl-md5 perl-extutils-makemaker-aur perl-geoip perl-ole-storage-lite perl-spreadsheet-parseexcel perl-unicode-map phoronix-test-suite php-mcrypt php-pear pim-storage-service-manager plank-config plank-theme-numix plex-media-server-plexpass pmount pocketsphinx powder-toy powershell powerstat-git prelink prettyping procexp proggyfonts psensor pulseaudio_ctl purple-hangouts-hg purple-skypeweb-git pycharm-edu pygoocanvas pysnmp python-aiohttp-cors-gns3 python-aiohttp-gns3 python-epub python-fusepy python-marisa-trie python-multidict-gns3 python-path python-pychromecast python-pyqtgraph python-typing python-ws4py python-yarl-gns3 python-zipstream python2-beautifulsoup3 python2-epub python2-gnomedesktop python2-gnomevfs python2-graphy python2-path python2-pychromecast python2-pydeep python2-pylzma python2-pyqtgraph python2-rarfile python2-rsvg python2-speechd python2-tinycss python2-tweepy python2-uniconvertor python2-zipstream python3-kitchen pyxml qemu-user-static qt4-gstreamer qt5-quick1 qtwebkit rar rbenv rss-glx rsvg2-sharp rtv ruby-build samsung_magician-consumer-ssd scalpel-git sfarklib sfarkxtc shapely shaperprobe shashlik-bin shimmer-wallpapers signal simgear slowmovideo-git spacefm sphinxbase ssdeep st stag-graph-git strigi sublime-text teamviewer termcap termsyn termsyn-font tnef toilet torcs torcs-data ttf-atarismall ttf-mac-fonts ttf-microsoft-consolas ttf-monaco ttf-ms-fonts ttf-oxygen ttf-roboto-mono ttf-vista-fonts udis86-git udisks ufraw-thumbnailer unagi unetbootin unifi unity-like-plank-theme untex urbanterror urbanterror-data usbmount ut2004 vapoursynth-plugin-f3kdb videothumbnail virtualbox-ext-oracle virtuoso-base vobcopy vulkan-man-pages wiifuse windowmaker windowmaker-extra wnck-sharp wordgrinder wxgtk2.8 wxlauncher xboxdrv xchat xfce-theme-albatross xfce-theme-blackbird xfce-theme-bluebird xfce-theme-cobibird xfce-theme-greybird-git xfce-theme-manager xfce4-dockbarx-plugin xfce4-embed-plugin xfce4-indicator-plugin xfce4-mixer xfce4-netspeed-plugin xfce4-quicklauncher-plugin xflux xorg-server-utils xorg-utils xrdp xsendkey xtraceroute yay-git zukitwo-themes

[Morganamilo]

Would you be able to bisect this for me? I can't reproduce it on my end so I am unable. Maybe finding the commit will help shed some insight on this.

[Morganamilo]

Also I really appreciate that stack trace. Looks like I messed up some of the concurrency in aurInfo() I'll look into it soon.

[Morganamilo]

So I think I fixed it, Pushed the change to master. Feel free to test it @sailingd @despairblue

[sailingd]

Awesome. It completes now, but I get an error from the aur server -- looks like maybe I'm querying aur for too many packages?

:: Querying AUR... Get https://aur.archlinux.org/rpc.php?arg%5B%5D=teamviewer&... (snipped long list of packages in URL request)...

http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=ENHANCE_YOUR_CALM, debug=""

Maybe there's a hard limit on the request size that I'm hitting. I'll try getting rid of some junk and try it again.

[Morganamilo]

Yeah seems like it. The problem was the wait group would not finish if any of the go-routines errored. So I was expecting this to reveal the real error. Try to lower the requestsplitn value in the yay config.

[sailingd]

Lowered even all the way down to 10 and it still fails with the same http2 error. I can copy and paste that same URL into a browser or with curl and it works fine.

[Morganamilo]

I think you might be getting rate limited for making too many requests too quickly. I'll make a branch where aurInfo() is no longer concurrent.

[Morganamilo]

Can you build and run https://github.com/Morganamilo/yay/tree/debug?

[Morganamilo]

I've found a way to test it myself, basically just yay -S <a ton of packages> the list of packages you have installed works quite nicely. I should be able to figure this out form here then. Thanks for the help and cooperation.

[Morganamilo]

Okay so I've found the issue and it comes down to me being the biggest idiot alive.

somehow I managed to write the min() function as:

func min(a, b int) int {
    if a < b {
        return a
    }
    return a
}

I think it's obvious enough whats wrong there.

Oh well it's fixed now. I'm leaving this here as my comment of shame and closing the issue.

[Jguer]

Also slipped through the code review, don't feel bad. Glad to see this solved. Maybe we should hurry and close this cycle for release as it has some very important bugfixes.

[Morganamilo]

Sure, you can make a release after merging #220 I'll leave the other stuff for the next release.