Getting `reading response header from upstream` error when trying to create new secrets with 1.7.2 on S3

jerrinss5 commented 5 months ago

Steps to reproduce

Install v1.7.2 - I tested it using docker container https://hub.docker.com/r/privatebin/nginx-fpm-alpine/ and tied to an S3 bucket
Create a new secret and it runs in to an error
Same doesn't happen with v.1.7.1

What happens

2024/05/06 16:01:58 [error] 18#18: *7961 FastCGI sent in stderr: "PrivateBin\Controller->_create()
PrivateBin/PrivateBin#8 /var/www/index.php(18): PrivateBin\Controller->__construct()
PrivateBin/PrivateBin#9 {main}
  thrown in /srv/vendor/aws/aws-sdk-php/src/Credentials/CredentialProvider.php on line 72" while reading response header from upstream, client: <CUT>, server: , request: "POST / HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm.sock:", host: "<cut>"

What should happen

Secret should be created without error

Additional information

NA

Basic information

Server address: BA

Server OS: Docker container https://hub.docker.com/r/privatebin/nginx-fpm-alpine/

Webserver: Amazon Linux 2

Browser: NA

PrivateBin version: 1.7.2

I can reproduce this issue on https://privatebin.net: No

elrido commented 5 months ago

Is this the full error message? It looks incomplete, there is no actual exception output, just a location. Also, line 72 in aws-sdk-php/src/Credentials/CredentialProvider.php is a comment block?

My current assumption is that we picked up something due to an upgrade of that library or such in the image, as we haven't changed anything on our side. Unfortunately we have no way of testing it, so have to rely on feedback of users and collected logs.

jerrinss5 commented 5 months ago

That's the only error I saw. There was some deprecation notice for PHP but wasn't sure if it was related or could lead to an error.

024/05/06 15:59:31 [error] 19#19: *7938 FastCGI sent in stderr: "PHP message: PHP Deprecated:  Return type of Aws\HandlerList::count() should either be compatible with Countable::count(): int, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HandlerList.php on line 284; PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 41; PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetGet($offset) should either be compatible with ArrayAccess::offsetGet(mixed $offset): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 30; PHP message: PHP Deprecated: Return type of Aws\Api\AbstractModel::offsetSet($offset, $value) should either be compatible with ArrayAccess::offsetSet(mixed $offset, mixed $value): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 36; PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetUnset($offset) should either be compatible with ArrayAccess::offsetUnset(mixed $offset): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 46; PHP message: PHP Deprecated:  Return type of Aws\Endpoint\Partition::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HasDataTrait.php on line 41; PHP message: PHP Deprecated

elrido commented 5 months ago

If that is the only error you saw, I have nothing to work out what the problem is. The error is coming from within the AWS SDK library. We have not updated that nor any of the code using it. The PHP 8.3 version is also still the same in that image. I have no access to AWS, so I have no way of reproducing this. I recommend you raise an issue with the upstream, maybe they can tell you what is going on? If they give you any hints, let us know.

elrido commented 5 months ago

The only other thing we have to go on are the build logs for the container image over in that projects github actions. I see this being reported regarding the AWS SDK installation:

PrivateBin/PrivateBin#8 3.414 Cannot use aws/aws-sdk-php's latest version 3.305.9 as it requires ext-simplexml * which is missing from your platform.
PrivateBin/PrivateBin#8 3.421 Using version ^3.33 for aws/aws-sdk-php
PrivateBin/PrivateBin#8 3.443 Cannot use google/cloud-storage's latest version v1.41.4 as it requires php ^8.0 which is not satisfied by your platform.
PrivateBin/PrivateBin#8 3.444 Using version ^1.30 for google/cloud-storage

That sudden backlevel of the AWS library may explain the issue, though I'm really not familiar with that library. This is a community provided backend.

Aside: The GPC related message is more than odd, since this runs on PHP 8.2 or 8.3 which should satisfy ^8.0.

The simplexml library did get installed:

PrivateBin/PrivateBin#8 2.282 (97/102) Installing php83-simplexml (8.3.6-r0)

I remember noticing that composer still is on php 8.2 on alpine 3.19, though. Maybe we now need to explicitly add php82-simplexml, so composer detects it at install time, then remove it again? Or ensure composer runs on the same php version as the rest?

PS: I see that the edge image builds use composer on php 8.3 - can any of you reporting this issue try and tell us if switching to the privatebin/s3:edge or privatebin/unit-s3:edge image solves your issue? That would corroborate that this 8.2/8.3 mix is the source of the issue.

QuentinBtd commented 5 months ago

PS: I see that the edge image builds use composer on php 8.3 - can any of you reporting this issue try and tell us if switching to the privatebin/s3:edge or privatebin/unit-s3:edge image solves your issue? That would corroborate that this 8.2/8.3 mix is the source of the issue.

I have the same problem with 1.7.2 images (nginx-fpm-alpine or s3). I tried with edge image... It works. 😃

elrido commented 5 months ago

Thank you for testing. So since this is an infra problem and not one with the PrivateBin software, I've transferred the issue over to the container image project. I currently see two solution paths:

short term, I could simply add the missing module to the composer related packages in the 8.2 variant, so we can trick composer into still installing the later library - need to be careful not to break the edge build, though, where this is not necessary
mid term, as in the next week(s), Alpine 3.20 will get released, at the level you currently get in edge, which avoids causing the issue and means the extra library can be removed again

longer term we need to take more care when switching php releases, and need to pay attention when composer is on a different release than the image targets. 8.4 (currently in development) will be our next opportunity to screw this up or deal with it more gracefully.

Maybe a multi-stage build could side-step the issue - download the release and composer artifacts in a build stage with git, wget, gnupg validation and composer, then transfer that application into the runtime container. Would also avoid the extra cleanup steps we currently have to do to remove composer & Co. from the image.

jerrinss5 commented 5 months ago

Thank you @QuentinBtd for testing and @elrido for the analysis and fix suggestions

elrido commented 4 months ago

The fixed images are now available under tag 1.7.2-alpine3.19.1-s3-composer and 1.7.2, stable and latest got updated to point to it as well.

jerrinss5 commented 4 months ago

Thanks @elrido - I can confirm that it works fine on the latest image.

PrivateBin / docker-nginx-fpm-alpine