microsoft / PubSec-Info-Assistant

Information Assistant, built with Azure OpenAI Service, Industry Accelerator
MIT License
344 stars 778 forks source link

Files not processing after upload, stuck in Uploaded or Queue State - V1.2 #883

Open ravikhunt opened 1 month ago

ravikhunt commented 1 month ago

After upgrading to version 1.2, newly uploaded files are not being processed. The files remain in the "Uploaded" state, and when resubmitted, they stay in the "Queue" state but do not proceed further for processing. This issue occurs even though the deployment process completes successfully. Logs show that all steps are executed without errors, but the files still don’t get processed as expected.

FIle Upload Not processing

Image: enrichmentapp pushed successfully Cleaning up artifacts directory: /workspaces/kyn-info-assistant/scripts/../artifacts/enrichmentapp Image: enrichmentapp pushed successfully Updating the enrichment webapp with the new image App settings have been redacted. Useaz webapp/logicapp/functionapp config appsettings listto view. App settings have been redacted. Useaz webapp/logicapp/functionapp config appsettings listto view. [ { "name": "DOCKER_REGISTRY_SERVER_PASSWORD", "slotSetting": false, "value": null }, { "name": "DOCKER_REGISTRY_SERVER_URL", "slotSetting": false, "value": "https://********.azurecr.io" }, { "name": "DOCKER_REGISTRY_SERVER_USERNAME", "slotSetting": false, "value": "infoasstacrkaidev" }, { "name": "WEBSITES_ENABLE_APP_SERVICE_STORAGE", "slotSetting": false, "value": "false" }, { "name": "DOCKER_CUSTOM_IMAGE_NAME", "value": "DOCKER|******.azurecr.io/enrichmentapp:**********" } ] Enrichment Webapp deployed successfully

dayland commented 1 month ago

Given they are staying in the first state, it appears as if the Azure Functions that process the files are not running. Please check the Azure Functions in the Azure Portal for any errors. You can also use the Azure Workbook in the same resource group to see App Service and Function Logs for more details.

ravikhunt commented 1 month ago

dayland

I have deployed the Azure Functions multiple times, and the deployment shows success each time. I also checked the logs in the Azure Portal, and there are no errors for the Azure Functions.

However, the issue persists where the Azure Functions do not seem to be triggered in the proper environment. When I run the functions locally and upload the file, the process completes successfully, but on the live environment, the functions do not appear to trigger.

Could you please provide specific steps or debugging suggestions to help identify the root cause of this issue? Any additional guidance would be appreciated.

ravikhunt commented 1 month ago

dayland

When checking back on the diagnosis and solving problem I believe that BLOB_CONNECTION_STRING was used for V1.1.1 or below but with V1.2 I can't find any reference in v1.2 codebase where use BLOB_CONNECTION_STRING in functions, but still gives error in azure functions

Please let me know your thoughts, Thank in advance for the helps and support

Image

bjakems commented 1 month ago

Hi @ravikhunt , Was this a fresh install from 1.2?

ravikhunt commented 1 month ago

No this is not fresh V1.2 but take the latest release pull and upgrade IT

ravikhunt commented 1 month ago

No this is not fresh V1.2 but take the latest release pull and upgrade IT from the V1.1.1 at that time all data var flush for Storage and DB and then try to upload a new file it's stuck with Upload State and when Try to resubmit from UI it's sticking with queue status

bjakems commented 1 month ago

We have not tested upgrades from v1.1.x to v1.2 and do not recommend it.

ravikhunt-kyn commented 1 month ago

Thank you for getting back to me bjakems

As I understand, we currently have V1.1.x running with files already uploaded to the storage account. To upgrade to V1.2, what steps do you recommend?

If upgrading from V1.1.x to V1.2 is not advisable, does that mean we should deploy a fresh V1.2 instance with new resources? If so, will deploying into a new resource group result in losing our old data? Also, based on my understanding, an upgrade would involve pulling the latest release (V1.2) from the codebase and then deploying it using the Make Deploy command. Could you please clarify if this is the correct approach and provide guidance on how to proceed without risking data loss?

bjakems commented 1 month ago

I do not recommend any steps to upgrade to 1.2. I suggest a net new deployment and to re-upload your files. If you really do not wish to do this approach, analyze the terraform plan file, adjust the code so that the resources are no longer deleted (not advised).

In order to deploy a fresh copy of 1.2 without impacting other deployments, please create a NEW Codespace to ensure no files are left over from a previous deployment. This Codespace should be created off of the latest main branch (version 1.2). Create the local.env file and set it up as you did prior, but edit the workspace variable so it deploys to a new RG "export WORKSPACE="changeThisHereToBeUnique"". Save the changes to your local.env. Run make deploy. Then re-import your data via the UI.

ravikhunt commented 1 month ago

All right, Thank you for the support