search

awslabs / aws-crt-python

Python bindings for the AWS Common Runtime
Apache License 2.0
87 stars 43 forks source link

AWS_ERROR_PRIORITY_QUEUE_EMPTY when trying to initialise the S3Client #542

Closed ashishkumar468 closed 7 months ago

ashishkumar468 commented 8 months ago

Describe the bug


from awscrt.auth import AwsCredentialsProvider
from awscrt.http import HttpHeaders, HttpRequest
from awscrt.io import ClientBootstrap, DefaultHostResolver, EventLoopGroup
from awscrt.s3 import S3Client, S3RequestType, create_default_s3_signing_config

def _get_s3_client():
    event_loop_group = EventLoopGroup()
    host_resolver = DefaultHostResolver(event_loop_group)
    bootstrap = ClientBootstrap(event_loop_group, host_resolver)
    credential_provider = AwsCredentialsProvider.new_default_chain(bootstrap)
    signing_config = create_default_s3_signing_config(
        region=s3_bucket_region, credential_provider=credential_provider
    )

    return S3Client(
        bootstrap=bootstrap,
        signing_config=signing_config,
    )

RuntimeError: 30 (AWS_ERROR_PRIORITY_QUEUE_EMPTY): Attempt to pop an item from an empty queue.```

The error is thrown when trying to initialise the S3Client

### Expected Behavior

Get the Client

### Current Behavior

RunTime Error

### Reproduction Steps

Added in the description

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### aws-crt-python version used

0.20.2  

### Python version used

3.8

### Operating System and version

LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: Amazon Description:    Amazon Linux release 2 (Karoo) Release:        2 Codename:       Karoo
DmitriyMusatkin commented 8 months ago

Which instance are you running on? Also, could you enable trace logging and attach it to the issue? https://github.com/awslabs/aws-crt-python/blob/main/awscrt/io.py#L28

github-actions[bot] commented 8 months ago

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

ashishkumar468 commented 8 months ago

Hi, sorry for the delay, here is the full stack trace

[INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [channel-bootstrap] - id=0x55ca63f88e40: Initializing client bootstrap with event-loop group 0x55ca63c9fd00
[INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AuthCredentialsProvider] - (id=0x55ca62e09080): TLS context not provided, initializing a new one for credentials provider.
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Creating profile collection from file at "/home/kmariim/.aws/config"
[ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [common-io] - static: Failed to open file. path:'/home/kmariim/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Failed to read file at "/home/kmariim/.aws/config"
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Creating profile collection from file at "/home/kmariim/.aws/credentials"
[ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [common-io] - static: Failed to open file. path:'/home/kmariim/.aws/credentials' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Failed to read file at "/home/kmariim/.aws/credentials"
[ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AuthCredentialsProvider] - static: Profile credentials parser could not load or parse a credentials or config file.
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Creating profile collection from file at "/home/kmariim/.aws/config"
[ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [common-io] - static: Failed to open file. path:'/home/kmariim/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfi######MLflowPluginV0.2#######:-Begin uploading to S3: /shared/kmariim/epoch=000016.ckpt.6aFDF9cD
######MLflowPluginV0.2#######:-Could not find Content Type: but the file is either a .pth or a .ckpt, adding application/octet-stream as the content-type
le] - Failed to read file at "/home/kmariim/.aws/config"
[ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AuthCredentialsProvider] - Failed to build config profile collection from file at (/home/kmariim/.aws/config) : Invalid file path.
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Creating profile collection from file at "/home/kmariim/.aws/config"

Just to add AWS auth credentials are there, aws cli operations are working fine

waahm7 commented 8 months ago

@ashishkumar468 These logs don't look complete, as they have no information about the AWS_ERROR_PRIORITY_QUEUE_EMPTY error. Can you please verify them again? Also, could you share some repro steps, such as the instance type you are running on, and any other helpful information regarding your environment? I am not able to reproduce this issue.

ashishkumar468 commented 8 months ago

Hi @waahm7 , its a bit weird, following this exception - 

[ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [common-io] - static: Failed to open file. path:'/home/kmariim/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Failed to read file at "/home/kmariim/.aws/config"
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Creating profile collection from file at "/home/kmariim/.aws/credentials"
[ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [common-io] - static: Failed to open file. path:'/home/kmariim/.aws/credentials' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Failed to read file at "/home/kmariim/.aws/credentials"
[ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AuthCredentialsProvider] - static: Profile credentials parser could not load or parse a credentials or config file.
[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [AWSProfile] - Creating profile collection from file at "/home/kmariim/.aws/config"
[ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [common-io] - static: Failed to open file. path:'/home/kmariim/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)```
the main script, thows this error

[DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [channel-bootstrap] - id=0x55ca63f88e40: acquiring bootstrap reference [INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [connection-manager] - id=0x55ca64068040: Successfully created [INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [exp-backoff-strategy] - id=0x55ca64055dd0: Initializing exponential backoff retry strategy with scale factor: 0 jitter mode: 0 and max retries 1 [DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [channel-bootstrap] - id=0x55ca63f88e40: acquiring bootstrap reference [INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [event-loop] - id=0x55ca63fb8940: Initializing edge-triggered epoll [INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [event-loop] - id=0x55ca63fb8940: Using eventfd for cross-thread notifications. [TRACE] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [event-loop] - id=0x55ca63fb8940: eventfd descriptor 55. [INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [event-loop] - id=0x55ca63fb8940: Starting event-loop thread. [INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [thread] - id=0x55ca63fb9800: cpu affinity of cpu_id 24 was specified, attempting to honor the value. [ERROR] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [thread] - id=0x55ca63fb9800: pthread_create() failed with 22 [FATAL] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [event-loop] - id=0x55ca63fb8940: thread creation failed. [INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [event-loop] - id=0x55ca63fb8940: Destroying event_loop [INFO] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [event-loop] - id=0x55ca63fb8940: Stopping event-loop thread. [TRACE] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [event-loop] - id=0x55ca63fb8940: Scheduling task 0x55ca63fb98b8 cross-thread for timestamp 0 [TRACE] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [event-loop] - id=0x55ca63fb8940: Waking up event-loop thread [DEBUG] [2024-01-26T04:36:38Z] [00007ffa0a6d4740] [channel-bootstrap] - id=0x55ca63f88e40: releasing bootstrap reference Traceback (most recent call last): File "test-s3-upload.py", line 89, in main() File "test-s3-upload.py", line 85, in main _upload_artifact_to_s3("kmariim-test.ckpt", "/shared/kmariim/epoch=000016.ckpt.6aFDF9cD") File "test-s3-upload.py", line 69, in _upload_artifact_to_s3 s3_request = _get_s3_client().make_request( File "test-s3-upload.py", line 41, in _get_s3_client return S3Client( File "/shared/.conda/envs/kmariim-copy-ldm-sd/lib/python3.8/site-packages/awscrt/s3.py", line 274, in init self._binding = _awscrt.s3_client_new( RuntimeError: 30 (AWS_ERROR_PRIORITY_QUEUE_EMPTY): Attempt to pop an item from an empty queue.``` I have a very basic script, that just creates the client and tries to make a basic S3 upload, attaching the script

import logging
import mimetypes
import os
from urllib.parse import quote
import boto3

import mlflow.store
from awscrt.auth import AwsCredentialsProvider
from awscrt.http import HttpHeaders, HttpRequest
from awscrt.io import (
    ClientBootstrap,
    ClientTlsContext,
    DefaultHostResolver,
    EventLoopGroup,
    TlsConnectionOptions,
    TlsContextOptions,
    init_logging
)

from awscrt.s3 import S3Client, S3RequestType, create_default_s3_signing_config
from mlflow.store.artifact.s3_artifact_repo import S3ArtifactRepository

s3_bucket = os.environ.get("ML_FLOW_S3_BUCKET", "kmariim-test-bucket")
s3_bucket_region = os.environ.get("S3_BUCKET_REGION", "us-east-2")

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')

TAG = "######MLflowPluginV0.2#######:-"

init_logging(6, "stdout")

def _get_s3_client():
    event_loop_group = EventLoopGroup()
    host_resolver = DefaultHostResolver(event_loop_group)
    bootstrap = ClientBootstrap(event_loop_group, host_resolver)
    credential_provider = AwsCredentialsProvider.new_default_chain(bootstrap)
    signing_config = create_default_s3_signing_config(
        region=s3_bucket_region, credential_provider=credential_provider
    )
    return S3Client(
        bootstrap=bootstrap,
        region=s3_bucket_region,
        signing_config=signing_config
    )

def _upload_artifact_to_s3(object_name, local_file_path):
    print(f"{TAG}Begin uploading to S3: {local_file_path}")
    file_stats = os.stat(local_file_path)
    data_len = file_stats.st_size
    headers = HttpHeaders(
        [
            ("host", s3_bucket + ".s3." + "us-east-2" + ".amazonaws.com"),
            ("Content-Length", str(data_len)),
        ]
    )
    content_type = mimetypes.guess_type(local_file_path)[1]
    request = HttpRequest("PUT", object_name, headers)

    s3_request = _get_s3_client().make_request(
        request=request, type=S3RequestType.PUT_OBJECT, send_filepath=local_file_path
    )
    finished_future = s3_request.finished_future
    try:
        finished_future.result()
    except Exception as e:
        print(f"{TAG}Error uploading to s3: {e}")

def main():
    # file_path = input("Enter the file path: ")
    _upload_artifact_to_s3("kmariim-test.ckpt", "/shared/kmariim/epoch=000016.ckpt")
    print(f"Uploading to S3")

if __name__ == "__main__":
    main()
waahm7 commented 8 months ago

@ashishkumar468 We have recently pushed a fix for this error. Can you please try the latest version v0.20.3 to see if it fixes the issue?

ashishkumar468 commented 8 months ago

Okay, let me try using the version 0.20.3 and get back

github-actions[bot] commented 8 months ago

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.