Head command hangs - Githubissues

Hristosko commented 1 month ago

Checkboxes for prior research

[X] I've gone through Developer Guide and API reference
[X] I've checked AWS Forums and StackOverflow.
[X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

Executing the sequence head, put, head for the same object that initially doesn't exist lead to a hang in the second head call

Regression Issue

[ ] Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/package-name@3.651.1

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

node v18.18.0

Reproduction Steps

import { HeadObjectCommand, PutObjectCommand, S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { expect, test } from 'vitest';

describe('AWS hang', () => {
    const s3Config: S3ClientConfig = {
        region: 'us-east-1',
        credentials: {
            accessKeyId: '***',
            secretAccessKey: '***',
        },
        tls: true,
        forcePathStyle: true,
        endpoint: 'https://localhost:11801',
    };

    const s3Client = new S3Client(s3Config);
    const bucketName = 'default-bucket-name';
    const key = 'some_key';

    test('repro', async () => {
        await expect(s3Client.send(new HeadObjectCommand({ Bucket: bucketName, Key: key }))).rejects.toThrow();
        await expect(
            s3Client.send(new PutObjectCommand({ Bucket: bucketName, Key: key, Body: new Uint8Array() })),
        ).rejects.toThrow();
        await expect(s3Client.send(new HeadObjectCommand({ Bucket: bucketName, Key: key }))).rejects.toThrow();
    });
});

Observed Behavior

The second head request hangs and we never get the expected error. This is reproducible with MinIO as backed server, It only happens with HTTPS, the test passes if we use HTTP. forcePathStyle doesn't make any difference. However the test passes if we use an actual object store, not MinIO. We have the same scenario implemented with the C++ SDK and it works with MinIO. The problem seems to be in the state of the client as if we create a new client the head request doesn't hang. The issue persists if the bucket exists.

The issue is reproducible with version 3.588.0 and the latest 3.651.1

Expected Behavior

The second head request should return an error

Possible Solution

No response

Additional Information/Context

No response

aBurmeseDev commented 1 month ago

Hi @Hristosko - thanks for reaching out.

This is reproducible with MinIO as backed server, It only happens with HTTPS, the test passes if we use HTTP. forcePathStyle doesn't make any difference. However the test passes if we use an actual object store, not MinIO. We have the same scenario implemented with the C++ SDK and it works with MinIO.

It sounds like this issue is specific to MinIO rather than AWS SDK as the problem does not occur when using HTTP or a different object store. Are you able to verify it with other SDKs besides C++?

Hristosko commented 1 month ago

Hi, @aBurmeseDev I could try with different SDK but it will take some time. Do you have any specific one in mind. The problem could be specific to MinIO however there is no way to test every single store out there. This seems to be an issue with the state of the client imo, because if we use a second client to make the second head request it works. As we will have to use this in production, we want to make sure that it works and as of now we have to pretty much delete the tests with HTTPS. Which is a bit risky. It would be fine if we know for sure the issue is caused by MinIO but currently I doubt it. From the tests with the second client and the c++ SDK we know MinIO is responding. And the only difference is in the SDK behavior. Ideally what I would like is to either make 100% sure the issue is from MinIO or fix the issue as this would be a nasty thing to happen in producion.

aBurmeseDev commented 1 month ago

Apologies for the delay here. Is this issue still ongoing for you? Have you had a chance to test with different SDK, for example Python(boto3).

github-actions[bot] commented 4 weeks ago

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

Hristosko commented 4 weeks ago

hi, the issue is ongoing. I tested the issue with different versions of MinIO (as changing the version is something we easily do) and the issue persists for at least 3 years. I can test with python, will take me some time until I get back to this though. I will let you know once I do

github-actions[bot] commented 2 weeks ago