search

Azure-Samples / azure-search-openai-demo

A sample app for the Retrieval-Augmented Generation pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.
https://azure.microsoft.com/products/search
MIT License
5.61k stars 3.76k forks source link

Automating Data Extraction: My Python Script for Web Scraping and Document Creation #1265

Open arsalanmubeen opened 4 months ago

arsalanmubeen commented 4 months ago

I've successfully crafted a Python script that efficiently scrapes all the data from a website and compiles it into a single doc file. Now, I'm eager to streamline the process by triggering the script whenever I execute 'azd up.' To achieve this, I'm on the lookout for the main class where I can embed this functionality. Any insights on where to find and embed the main class would be greatly appreciated!

pamelafox commented 4 months ago

Take a look at azure.yaml: https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/azure.yaml

It uses "hooks" to execute scripts before or after particular stages. We currently run prepdocs.sh as a postprovision hook, which runs after "azd provision" or after the provisioning stage in "azd up". So you could add your script to the postprovision hooks in azure.yaml, or call your script from prepdocs.sh, whatever you prefer.

Learn more about hooks here:

https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/azd-extensibility