Please introduce startup option to create containers up front, or to create containers automatically

thomaseyde commented 1 year ago

Please introduce startup option to create containers up front, or to create containers automatically. This will be useful in the scenarios where the production infrastructure takes care of container creation.

Without this feature, it is harder to follow these recommendations:

Let the infrastructure take care of container creation. For apps deployed with Azure DevOps pipelines, this means letting the pipeline take care of container creation.
Use a singleton client for all container access.

Setting up this singleton is usually done in Startup.cs as part of dependency container configuration. This part of the application is not async, and should be non-blocking.

There is no pipelines involved during development, and Azurite doesn't offer a feature to create containers at startup.

Our options are to either create the containers manually, which is a bottleneck during development testing; or to create mitigation code which complicates otherwise simple code.

It would have been nice to start azurite with an argument like --containers "a, b, c" to create three containers in this case, or --auto-create-container=true

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

3.23.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm

What's the Node.js version?

v18.12.1

What problem was encountered?

Not able to preconfigure azurite with known containers.

Steps to reproduce the issue?

Have you found a mitigation/solution?

Write wrapping code around the container client which test for "UseDevelopmentStorage=true" at runtime, then create the container.

piotrbrzuska commented 1 year ago

For me it could be great if I can import folders and files as containers and blobs

blueww commented 1 year ago

@thomaseyde , @piotrbrzuska We already has a similar issue https://github.com/Azure/Azurite/issues/1665.

Data structure of Azurite used to save blob/containers are much different than the structure of blob/container you can see. So currently we don't see an easy way to import folders and files to prepare the test data, instead you need call rest API to Azurite to create them.

Besides that, if the pre-create test data are same for every test runs, you might can first make sure the containers are created, then save the data in the workspace (workspace is specify with -l when start azurite) , and copy the save workspace to a new copy, and start Azurite with the new workspace copy. (However, Azurite doesn't assume backward for the data file format, so you might need to recreate the file after upgrade Azurite version.)

Azurite welcome contribution! It would be great if you have a good idea (better discuss with us and get agreed), and you can raise a PR to add this to Azurite.

thomaseyde commented 1 year ago

No no. I wasn't asking for pre-creation of data.

I ask for pre-creation of an empty container.

Then I can simply assume in my code that given containers will always exist. If they don't, there is an error somewhere.

Benefits:

Simulates the way a container is setup during deploy.
Ensures expected containers are used.
Quality check on container name spelling in code.
Discourage use of random or unexpected containers

blueww commented 1 year ago

@thomaseyde

Azurite currently doesn't support pre-create data (include container) when start. As my before comments, since Data structure of Azurite used to save blob/containers are much different than the structure of blob/container you can see, there's no easy way to pre-create container now.

You can create a container by SDK/script or other ways after Azurite just start up.

Azurite welcome contribution! It would be great if you have a good idea and share with us!

x3ro commented 1 year ago

I also ran into this issue. For the time being, I'm using the following script. Maybe it proves useful to someone else :) Save it with .mjs extension and run it with node. Make sure the @azure/storage-blob package is installed. Since the BlobServiceClient waits for quite a long time until it fails, just starting the provisioning script in parallel with the docker container, without any additional waiting, worked fine for me.

import {
    BlobServiceClient,
  } from '@azure/storage-blob';

const containerToBeCreated = "techdocs";
const connectionString = 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;';
const client = BlobServiceClient.fromConnectionString(connectionString);

console.log("Listing existing containers:");
let containerExists = false;
for await (const container of client.listContainers()) {
  if(container.name === containerToBeCreated) {
    containerExists = true;
  }
  console.log(`* ${container.name}`);
}

if(!containerExists) {
  try {
    console.log(`Container '${containerToBeCreated}' does not exist, creating...`);
    await client.createContainer(containerToBeCreated);
    console.log("Success!");
  } catch(e) {
    console.log("Container creation failed!");
    console.log(e);
  }
}

blueww commented 1 year ago

Another idea for workaround is: use something like a curl (or wget) command to create container after Azurite start as a part of bootstrapping.

Following sample will use curl to create a container on Azurite, it use an account SAS generated with Azurite default storage account key, and with a long expire time.

curl -I -X PUT http://127.0.0.1:10000/devstoreaccount1/newcontainer?restype=container&sv=2022-11-02&ss=bqt&srt=sco&se=2100-01-01T00%3A00%3A00Z&sp=rwdxylacuptfi&sig=cPr3DAuz9MV4wdGwBIGNBteDy%2F5C%2BrTd8tKTduzwDOc%3D

Azure / Azurite