search

flora-pm / flora-server

A package index for the Haskell ecosystem
https://flora.pm/about
Other
125 stars 38 forks source link

Hang on import-index #460

Closed RaoulHC closed 11 months ago

RaoulHC commented 11 months ago

Environment

This has also been confirmed by @tchoutri on the development branch.

Expected Behaviour

For everything to run to completion and successfully import an index.

Actual Behaviour

It sets up the package and requirements tables but then just hangs indefinitely. Running it again seems to get it to finish successfully.

Instructions for reproduction

Using bsdtar allow you to create a subset of the hackage index to test this, for example just with the servant package here.

$ make db-reset
$ cabal update
$ bsdtar -c -f test-index.tar --include='servant/**.cabal' @/home/$USER/.cabal/packages/hackage.haskell.org/01-index.tar
$ gzip test-index.tar
$ cabal run flora-cli -- import-index test-index.tar.gz --repository hackage
RaoulHC commented 11 months ago

Looks like having flora-server running is necessary for this to hang, with no instant running it works correctly, but given that's a common scenario this still definitely needs to be fixed.

RaoulHC commented 11 months ago

This is another too many connections issue, if you get a libpq error in the flora-server instance this causes a poolboy job to never return and hang, though occasionally you get a poolboy job returning the libpq error, with import-index actually returning in this case.

I have no idea why a connection error in the flora-server would cause the a poolboy job to hang, as I would have thought by virtue of the poolboy job not getting a connection error it should successfully persist all the necessary rows, but maybe someone more knowledgable about psql connections can shed some light. I also wonder if it might be an issue with poolboy where an exception is getting raised but not getting caught.

In any case this shouldn't be an issue where a pgbouncer instance is running, nor is it an issue if the number of connections is readjusted so that a server running + flora-cli don't go above the db's max connections.