Azure / azure-storage-php

Microsoft Azure Storage Library for PHP
MIT License
217 stars 200 forks source link

Cannot Fetch Messages from Queue #168

Closed phuze closed 5 years ago

phuze commented 5 years ago

Which service(blob, file, queue, table) does this issue concern?

Queue

Which version of the SDK was used?

1.1.1

What's the PHP/OS version?

7.2.13

What problem was encountered?

Cannot fetch messages from a queue.

Steps to reproduce the issue?

Establish connection using createQueueService(), then attempt to list messages using listMessages() and then attempt to pull out a single message using getQueueMessages()

Have you found a mitigation/solution?

If I query the REST service directly, using cURL, I am able to return messages from the queue service.

phuze commented 5 years ago

Here's the code:

<?php

/**
 * This is the code, using the microsoft/azure-storage-queue package
 * version 1.1.1 as found here: https://github.com/Azure/azure-storage-php
 */

use MicrosoftAzure\Storage\Common\Middlewares\RetryMiddlewareFactory;
use MicrosoftAzure\Storage\Common\Exceptions\ServiceException;
use MicrosoftAzure\Storage\Queue\QueueRestProxy;

try {

    # replace this with the name of your azure storage account
    $storageAccount = "xxxxxxxxxx";

    # replace this with your own account key
    # taken from azure portal --- "Storage accounts" > STORAGE_ACCOUNT_NAME > "Access keys"
    $accountKey = "xxxxxxxxxx";

    # replace with the name of the queue
    $queueName = "xxxxxxxxxx";

    # create queue service
    $queueClient = QueueRestProxy::createQueueService("DefaultEndpointsProtocol=https;AccountName=$storageAccount;AccountKey=$accountKey");

    # list messages
    $listMessagesResult = $queueClient->listMessages($queueName);

    # get messages from list
    $messages = $listMessagesResult->getQueueMessages();

    # !!!!!!! NOTE THIS RETURNS NOTHING (EMPTY OBJECT)
    # yet there are hundreds of messages in the queue at the time
    echo json_encode($messages);

} catch (Exception $e) {
    # catch both Azure and Guzzle exceptions
    if ($e instanceof ServiceException OR $e instanceof RequestException) {
        echo json_encode([
            "status_code" => $e->getCode(),
            "error_message" => $e->getMessage()
        ]);
    }
}
/**
 * ALTERNATIVELY, here's some plain old cURL code which DOES list messages
 * this indicates to me that something has gone wrong with the azure storage PHP SDK
 * however, there have been no updates in months, and everything has been working
 * great up until 09:56 EST --- suddenly stopped being able to list messages
 */

# name of your azure storage account
$storageAccount = "xxxxxxxxxx";

# the azure message queue name
$queueName = "xxxxxxxxxx";

# number of messages to return
$numMessages = "32";

# the access signature which you can generate
# from azure storage explorer --- should begin with "sv="
$accessSignature = "xxxxxxxxxx";

# peek only -- set to TRUE to PREVENT message from being dequeued
# otherwise, set to FALSE to DEQUEUE the message (will come back after 30 seconds)
$peekOnly = "true";

$curl = curl_init();

curl_setopt($curl, CURLOPT_URL, "http://$storageAccount.queue.core.windows.net/$queueName/messages?numofmessages=$numMessages&peekonly=$peekOnly&$accessSignature");

$result = curl_exec($curl);

echo json_encode($result);
XiaoningLiu commented 5 years ago

Hi @purplekrayons

In your curl URL, you actually called REST API PeekMessages https://docs.microsoft.com/en-us/rest/api/storageservices/peek-messages .

But the PHP method $queueClient->listMessages($queueName); will call another API GetMessages https://docs.microsoft.com/en-us/rest/api/storageservices/get-messages

You can get the differences in above REST API documents.

Try with $queueClient->peekMessages($queueName);

phuze commented 5 years ago

Hi @XiaoningLiu

Between Jan 13 starting @ 09:56 EST, through until around Jan 14 @ ~03:30 EST, we were unable to retrieve messages from any queue, across any storage account in Azure, using this SDK.

A Microsoft engineer I had got in touch over the phone suggested trying to pull messages by interfacing with the REST API directly, as opposed to through the SDK, as an alternative to help get our enterprise operations moving forward. I was unaware which endpoints this SDK was using.

Today (Jan 14), starting from at least 03:30 EST, I noticed that I was suddenly able to pull messages again using the SDK, so I reverted my code to use the SDK instead of plain cURL requests I'd written. I have no idea what happened, why it happened, or how the issue came to be resolved.

Since everything is working again, I am marking this issue as resolved. However, if you are interested in investigating, here's an excerpt from what I was seeing on my end while debugging:

2019-01-13 20:49:35-05:00: establishing queue service...
2019-01-13 20:49:35-05:00: debug: azure queue: DefaultEndpointsProtocol=https;AccountName=<hidden>;AccountKey=<hidden>
2019-01-13 20:49:35-05:00: attempting to get queue count...
2019-01-13 20:49:35-05:00: debug: queue count: 167
2019-01-13 20:49:35-05:00: attempting to list messages...
2019-01-13 20:49:35-05:00: debug: message listing: {}
2019-01-13 20:49:35-05:00: attempting to get queue messages...
2019-01-13 20:49:35-05:00: debug: message listing: [{}]
2019-01-13 20:49:35-05:00: debug: azure message: {}

note: i've hidden the account name and key for safety

use MicrosoftAzure\Storage\Common\Middlewares\RetryMiddlewareFactory;
use MicrosoftAzure\Storage\Common\Exceptions\ServiceException;
use MicrosoftAzure\Storage\Queue\QueueRestProxy;
use Monolog\Logger;

class AzureTest {

    public static function logger() { /*...my logger...*/ }

    public static function useNext(string $queueName)
    {
        self::logger()->info("establishing queue service...");
        $queueClient = QueueRestProxy::createQueueService(AZURE_QUEUE);
        self::logger()->info("debug: azure queue: " . AZURE_QUEUE);

        self::logger()->info("attempting to get queue count...");
        $queueMetadata = $queueClient->getQueueMetadata($queueName);
        $msgCount = $queueMetadata->getApproximateMessageCount();
        self::logger()->info("debug: queue count: " . $msgCount);

        self::logger()->info("attempting to list messages...");
        $listMessagesResult = $queueClient->listMessages($queueName);
        self::logger()->info("debug: message listing: " . json_encode($listMessagesResult));

        self::logger()->info("attempting to get queue messages...");
        $messages = $listMessagesResult->getQueueMessages();
        self::logger()->info("debug: message listing: " . json_encode($messages));

        $message = $messages[0];
        self::logger()->info("debug: azure message: " . json_encode($message));

        return $message;
    }

}

# get next message from queue
$message = AzureTest::useNext("shopify-orders-create");

if ( !empty($message) ) {

    $messageId = $message->getMessageId();         # the message ID number
    $popReceipt = $message->getPopReceipt();       # the pop receipt (for dequeing)
    $messageBody = $message->getMessageText();     # the message body
    $dequeueCount = $message->getDequeueCount();   # the dequeue count

    # print results to screen
    print_r($messageId).PHP_EOL;
    print_r($popReceipt).PHP_EOL;
    print_r($messageBody).PHP_EOL;
    print_r($dequeueCount).PHP_EOL;

} else {
    exit;
}

Prior to Jan 13 @ 09:56 EST, this code resulted in the message data being printed to screen. Then suddenly, it began failing and the result was that nothing was printed to screen. This same code then resumed working again sometime after Jan 14 @ 03:30 EST.

Thanks, -Brendon

phuze commented 5 years ago

I'll also note quickly, that this is the second time I've encountered strange behaviour with the Azure Queue service. Note my previous ticket where queue messages randomly weren't being created.

https://github.com/Azure/azure-storage-php/issues/156

XiaoningLiu commented 5 years ago

Sorry for the inconvenience brought to you. As I cannot reproduce both this issue on my dev box and our test environments. It's appreciated you can share the request/response logs collected by HistoryMiddleware or tools like Fiddler.

You can enable fiddle proxy by uncomment these 2 lines code: https://github.com/Azure/azure-storage-php/blob/1afdf1c788a729ed486a6ef1c2e74d3db7b199b9/azure-storage-common/src/Common/Internal/ServiceRestProxy.php#L105