List of files filtered

[fverneau]

Hello, Does the Microsoft REST API on Azure Storage allow you to natively filter the files that are listed? For example with a filter on the file name or their extension. QStringList slNameFilters; ... azure->listFilesSynchronous(containerName, foundListOfFiles, slNameFilters); This would limit the list of all files. This would be especially useful for containers with a lot of files.

[QuentinCG]

You can already do it using marker: https://github.com/QuentinCG/QAzureStorageRestApi/blob/master/lib/include/QAzureStorageRestApi.h#L145

  /*!
   * \brief listContainers List containers in an azure storage account
   *
   * \param[out] foundListOfContainers List of containers retrieved from Azure API (if no error)
   * \param marker (optional) Marker to list specific informations only
   * \param timeoutInMs (optional) Max time to wait an answer from Azure API
   *
   * \return QNetworkReply::NetworkError::NoError if list retrieved successfully on time
   */
  QNetworkReply::NetworkError listContainersSynchronous(QList< QMap<QString,QString> >& foundListOfContainers, const QString& marker = QString(), const int& timeoutInMs = 20000);

  /*!
   * \brief listFiles List files in an azure storage container
   *
   * \param container Container to check
   * \param[out] foundListOfFiles List of files retrieved from Azure API (if no error)
   * \param marker (optional) Marker to check specific informations only
   * \param timeoutInMs (optional) Max time to wait an answer from Azure API
   *
   * \return QNetworkReply::NetworkError::NoError if list retrieved successfully on time
   */
  QNetworkReply::NetworkError listFilesSynchronous(const QString& container, QList< QMap<QString,QString> >& foundListOfFiles, const QString& marker = QString(), const int& timeoutInMs = 20000);

https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs?tabs=microsoft-entra-id

marker	Optional. A string value that identifies the portion of the list to be returned with the next list operation. The operation returns a marker value within the response body if the list returned was not complete. You can then use the marker value in a subsequent call to request the next set of list items.The marker value is opaque to the client.

[QuentinCG]

Sorry, my answer is incorrect. It seems to be possible only to filter by prefix (by specifying subfolder in list request for example, not possible in the lib for the moment). https://stackoverflow.com/questions/54308875/how-is-the-prefix-parameter-used-in-cloudblobcontainer-listblobs-to-get-files

But for other filter, it seems hard, even for official app: https://github.com/microsoft/AzureStorageExplorer/issues/1667

[fverneau]

At the moment I get a 201 error when retrieving the list of files. It's amazing. I'm going to check the permissions on the Azure Storage account. I also tried with a SAS Token on the container with Read and List permissions, but I get the same error code. I'll check on my side...

[fverneau]

I tried to get the list of files on different Azure Storage via the listFilesSynchronous function and I always get an error code 201 (access denied). All my tests, whatever they are, generate a 201 error. Does this work for you?

[QuentinCG]

At the moment I get a 201 error when retrieving the list of files. It's amazing. I'm going to check the permissions on the Azure Storage account. I also tried with a SAS Token on the container with Read and List permissions, but I get the same error code. I'll check on my side...

This was a lib issue I introduced at the same time I added 'synchronous' because I also refactored the class to also allow to connect with 'SAS key' if we want (and I needed to refactor a bit the source code).

It should work now.

[QuentinCG]

To answer you first request, I added some features on listFiles & listFilesSynchronous:

  /*!
   * \brief listFiles List files in an azure storage container
   *
   * \param container Container to check
   * \param[out] foundListOfFiles List of files retrieved from Azure API (if no error)
   * \param marker (optional) Marker to check specific informations only
   * \param prefix (optional) Prefix to filter results
   * \param maxResults (optional, default: default Azure REST API number of results) Max number of elements
   * \param timeoutInSec (optional) Max time to wait answer (in sec)
   *
   * \return QNetworkReply::NetworkError::NoError if list retrieved successfully on time
   */
  QNetworkReply::NetworkError listFilesSynchronous(const QString& container, QList< QMap<QString,QString> >& foundListOfFiles, const QString& marker = QString(), const QString& prefix = QString(), const int& maxResults = -1, const int& timeoutInSec = 10);

You can do to filter:

azure->listFilesSynchronous(container, foundListOfFiles, QString(), QString("folder/subfolder/"));

You can do to that to check if a file exists:

azure->listFilesSynchronous(container, foundListOfFiles, QString(), QString("folder/subfolder/mySuperFile.txt"));

[fverneau]

I downloaded the latest version this morning and I still get error code 201 when I want to retrieve the list of files. And now I also have this same error code when I upload a file... :-/

[QuentinCG]

I don't have access to Azure Storage account those days. The error seems in fact a non error... I'll fix that in the lib: https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob?tabs=microsoft-entra-id#status-code

[QuentinCG]

I added isErrorCodeSuccess(CodeHere) helper to check on your side if return code is valid.

Synchronous method should also now always try to parse result, even if return code is not one found in the 'success code list'.

I don't have access to Qt compiler right now so lib may not compile due to minor error (easy to fix if you need it right now)

[fverneau]

OK, I managed with code 201.

[QuentinCG]

I ended the fix I began few hours ago. It works now.

[fverneau]

For information, in the version I downloaded, I still had messages in qDebug(), i.e. : [QAzureStorageRestApi] Generated URL: "https://MyAccount.blob.core.windows.net/MyContainer/Import/test.txt" QTimer::singleShot: Timers cannot have negative timeouts Returned code "0" , is valid answer: True

[QuentinCG]

For information, in the version I downloaded, I still had messages in qDebug(), i.e. : [QAzureStorageRestApi] Generated URL: "https://MyAccount.blob.core.windows.net/MyContainer/Import/test.txt" QTimer::singleShot: Timers cannot have negative timeouts Returned code "0" , is valid answer: True

This is strange since default timeout are set to positive values in all synchronous methods in lib header:

QNetworkReply::NetworkError downloadFileSynchronous(const QString& container, const QString& blobName, QByteArray& downloadedFile, const int& timeoutInSec = 30, const bool& forceTimeoutOnApi = false);

https://github.com/QuentinCG/QAzureStorageRestApi/blob/master/lib/include/QAzureStorageRestApi.h#L196

Moreover, since no timeout can be triggered in this case, if no response from Azure is received, it will be an infinite loop. This may cause unexpected behavior. There is something wrong in the source code or header you have in local as it is not the same as the one currently in the repository.

I added safeguard right now to prevent infinite loop to happen as library should not allow to send request if timeout parameter is invalid in synchronous request (library was too permissive)... (https://github.com/QuentinCG/QAzureStorageRestApi/commit/b408ee324da60c507f665c73eb4d8caf866cd6f7)

[fverneau]

Okay, thanks for the response. I will check by downloading the latest version again.

[fverneau]

I just tested and everything came back to normal. I no longer have any messages coming out of the application. Thanks again !

[QuentinCG]

You are welcome