🚀 Feature: Azure Blob Storage entity provider

[drodil]

🔖 Feature description

Just like AWS S3 Entity Provider, there could be support to fetch entities from Azure Blob Storage

🎤 Context

Azure users cannot use Blob Storage to serve catalog info files for Backstage like AWS S3 users can. A new provider would be needed to support this.

✌️ Possible Implementation

No response

👀 Have you spent some time to check if this feature request has been raised before?

• [X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• [X] I have read the Code of Conduct

Are you willing to submit PR?

None

[tudi2d]

Thanks @drodil - sounds good! Would you be interested in contributing the provider or should we open this up for contributions?

[drodil]

I think it's best to open it up if someone has time to look at it. I have my hands full with notifications work already 😆

[numcys]

hey @drodil I am a new contributor here, but have experience with cloud, especially azure. If possible I would like to work on this one, if you you would be willing to point me to the currently implemented AWS S3 provider...

[drodil]

Hey @numcys , that would br great! The S3 provider is located here: https://github.com/backstage/backstage/blob/master/plugins%2Fcatalog-backend-module-aws%2Fsrc%2Fproviders%2FAwsS3EntityProvider.ts

[drodil]

Oh, and @numcys , this would probably be the correct module for that: https://github.com/backstage/backstage/tree/master/plugins/catalog-backend-module-azure

[numcys]

Hi @drodil, I have been tackling some problems while trying to implement this, I didn't had much experience with typescript and even though I tried, I don't think I will be able to complete this one. Really sorry if I wasted yours or anyone's time and really thankful for giving me this chance. I will take my time to understand the project better and then try to go for contributions...

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Nikunj0601]

Hi @drodil @freben If anyone is not working on this I would like to work on this one. Also, you want separate file for azure blob entity provider or you want add to azure devops file?

[drodil]

@Nikunj0601 no I don't think there's anyone working on this yet so you are welcome to take it 👌 What do you think is the best way to proceed?

[Nikunj0601]

I think it should be in separate cause in catalog provider accountName, accountKey, contiainerKey which is from devOps provider. Though I am not sure it is a best approach or not.

[Nikunj0601]

Hi @drodil @freben I was trying to add aws s3 plugin but it seems that i am doing something wrong while integrating it. I followed the backstage docs https://backstage.io/docs/integrations/aws-s3/discovery. Also I have completed entityProvider and integration can you take a look and give a feedback whether I am working on right direction or not. https://github.com/Nikunj0601/backstage/tree/feat/azure-blob-storage-integration

[Nikunj0601]

@drodil @freben any idea why log says one location committed but component is not visible in ui? backend log: [example-backend]: 2024-05-19T22:13:39.371Z catalog info Committed 1 Locations for Azure Blob Storage blobs target=azureBlobStorage-provider:default class=AzureBlobStorageEntityProvider taskId=azureBlobStorage-provider:default:refresh taskInstanceId=5f2b1fae-cc04-44bb-839f-41a9c6d71669 Also, is there a cmd to create new plugin was added in azureDevOps but allows only one module which is set to export as default. Sorry to bug you on weekend.

[Rugvip]

@Nikunj0601 regarding the ingestion, that might be down to trouble reading the actual entity from the location. The provider creates a location entity, but that location then needs to be read and parsed by the UrlReaderProcessor for the actual entity to show up. We currently don't have an azure blob storage URL reader, the existing one is only for Azure DevOps.

[Nikunj0601]

@Rugvip Thanks I will do that for azure blob storage. Also, is there any cmd to create a new plugin cause intially I added this azure blob entity provider in azure DevOps plugin but while exporting catalogModuleAzureBlobStorageEntityProvider in /module/index.ts. for entityProvider to be recoginsed it needs to be exported as default but there is already azureDevopsEntityProvider exported as default. So I think creating new plugin would be better. And one more question this storage account should always be public or it can also be private.

[Nikunj0601]

@Rugvip I am having an issue with registering AzureBlobStorageUrlReader, I have added factory in UrlReaders also exported AzureBlobStorageUrlReader in index.ts as well but I can see any logs or signs that being actually used. Can you guide me I like am I missing export it somewhere as well? Thanks

[freben]

Shouldn't be about a missing export if typescript is happy and not complaining about your code, I think.

Is this on the new backend system? Did you remember to register a replacement coreServices.urlReader in your backend with these additions in it?

Also, what did you do to try to trigger it being called? Some URL registered in the catalog or so?

[Nikunj0601]

@freben Yes I did registered a factory in UrlReaders class. Also, for triggerring I am adding AzureBlobStorageEntityProvider which I created into catalog.ts using builder.addEntityProvider( ...AzureBlobStorageEntityProvider.fromConfig(env.config, { logger: env.logger, schedule: env.scheduler.createScheduledTaskRunner({ frequency: Duration.fromObject({ minutes: 30 }), timeout: Duration.fromObject({ minutes: 3 }), }), }), ); which is similar to importing awsS3 entityProvider.