SQL Server 2022 Polybase with Azurite - file in use?

kimjamia commented 6 months ago

Which service(blob, file, queue, table) does this issue concern?

Blob

Which version of the Azurite was used?

3.28.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

It comes with Visual Studio Community.

What's the Node.js version?

What problem was encountered?

Attempting to read a blob from Azurite using SQL Server 2022 Polybase results in error:

File 'test-file.csv' cannot be opened because it does not exist or it is used by another process.

Steps to reproduce the issue?

Configure Azurite to use HTTPS using the dotnet dev-cert.
Create a container and set its access level to public.
Upload a csv file to the container.
Create an external data source in a locally running non-containerized SQL Server 2022 and attempt to read the CSV from it:
```
CREATE EXTERNAL DATA SOURCE MyAzureStorage
WITH (
LOCATION = 'abs://localhost:10000/devstoreaccount1/test'
);
```

SELECT * FROM OPENROWSET( BULK 'test-file.csv', DATA_SOURCE = 'MyAzureStorage', SINGLE_CLOB ) AS DataFile;


I can access the blob file with a browser from `https://localhost:10000/devstoreaccount1/test/test-file.csv`.
I got the external data source to work with a publicly available Azure Blob Storage so this issue looks Azurite-related.

kimjamia commented 3 months ago

Any updates? I'm not sure why this was categorized as a question. It feels more like a bug to me.

blueww commented 3 months ago

@kimjamia

We set "question" label to it, since we don't know the root cause for this issue, so can't says this is a bug.

@EmmaZhu Would you please help to follow up this issue? As I see, since https://localhost:10000/devstoreaccount1/test/test-file.csv works, this looks not an Azurite issue. And the error File 'test-file.csv' cannot be opened because it does not exist or it is used by another process. is also a client side error. Since the repro code use protocol abs://, this looks expected since Azurite still not support datalakegen2. (Azurite only support Blob/Queue/Table now, see details.)

kimjamia commented 1 month ago

So does this mean that Azurite can't be used with SQL Server 2022 Polybase?

Azure / Azurite