SQS receive_message VisibilityTimeout attribute not returned

LuchiLucs commented 2 months ago

Describe the bug

Calling the receive_message() method of the SQS client with MessageSystemAttributeNames key and value["All", "VisibilityTimeout"] or ["All"] seems not to return a response whose message attribute contains the intented key under its attributes.

Moreover the Python SDK documentation here has that key in the deprecated key AttributeNames and not in MessageSystemAttributeNames. The REST API is here.

Assuming each message has a custom visibility timeout already set. How should I retrieve this information?

Expected Behavior

The returned response should contain the specified attribute.

Current Behavior

The returned response seems not to contain the specified attribute.

Reproduction Steps

def receive_messages(
    self,
    queue_url: str,
    max_number: int = 1,
    wait_time: int = 0,
    visibility_timeout: int = 2 * 60 * 60,
    attribute_names: list[str] = ["All", "VisibilityTimeout"],
):
    response = self.sqs_client.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=max_number,
        WaitTimeSeconds=wait_time,
        VisibilityTimeout=visibility_timeout,
        MessageSystemAttributeNames=attribute_names,
    )
    return response

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.35.2

Environment details (OS name and version, etc.)

Linux (AWS EC2)

adev-code commented 1 month ago

Hi @LuchiLucs, thanks for reaching out. Testing from my side, having "All" as a value and pulling the message from the SQS for the first time by a consumer, the output will show the values for "MessageSystemAttributeNames". When pulling the same message right after you have pulled it, it may not show the list of attributes. This is because the message that you have pulled previously have been taken out already from the SQS Queue and already been delivered to a consumer. This is also shown on the visibility timeout documentation ( https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html ). That said, please try to create a new message then try to pull the message from SQS for the first time with "All" as a value and see the results. If VisibilityTimeout is not specified, it has a 30 seconds default value. So the longer the VisibilityTimeout, the longer the message will not be retrieved again.

If you would like to get the same message again right away, you can set a low "VisibilityTimeout" (VisibilityTimeout=0) - this way you would be able to get the output for the parameters you have included. Also, please take note that "VisibilityTimeout" would not be a value for "MessageSystemAttributeNames".

I have raised an internal ticket to our SQS Documentation team for clarifying the retention of outputs and for the VisibilityTimeout to be added to the API docs.

If you have any questions, please do let me know. For other review, please provide full debug logs by adding boto3.set_stream_logger('') to your code and redacting any sensitive information. Thank you

LuchiLucs commented 1 month ago

@adev-code Thank you for your review. I'm confused whether I should be able to see the set custom value of the key VisibilityTimeoutwith the mentioned version of the Python SDK, or the current REST API. As far as I understood, when I pull a message from the queue for the first time (I'm using FIFO queues) the message becomes in-flight and the queue is blocked while processing the message untill it finishes, by removing the message handle from the queue, or if timeout expires. In my particular use-case, as in the provided example, I'm using the receive_messagemethod to at the same time specified the VisibilityTimeoutdesired value, by setting this per message and not using the default queue value, and also getting the message medadata, i.e. the MessageSystemAttributeNames key which, as far as I understood, should contain the attributes, and one of these is the VisibilityTimeout. In other words, what I expect to date, it to set and get, at the same time the VisibilityTimeoutwith that API call. Am I missing something?

adev-code commented 1 month ago

Hi @LuchiLucs, thanks for the update. As per the documentation for receive_message (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs/client/receive_message.html#SQS.Client.receive_message), the 'VisibilityTimeout' is not part of the response. It is only used as an input to configure the VisibilityTimeout. I have submitted a feature request to the SQS Team about if this can be added to the response. Please refer to an SDK changelog (https://raw.githubusercontent.com/boto/boto3/develop/CHANGELOG.rst) for updates about this going forward. Please let me know if you have any other questions. Thanks!

github-actions[bot] commented 1 month ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

boto / boto3