Azure-Samples / azure-search-openai-demo

A sample app for the Retrieval-Augmented Generation pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.
https://azure.microsoft.com/products/search
MIT License
6.16k stars 4.19k forks source link

failing invoking action 'provision' #195

Closed Ilarom closed 1 year ago

Ilarom commented 1 year ago

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

azd up

Any log messages given by the failure

ERROR: deployment failed: failing invoking action 'provision', planning deployment: planning infrastructure provisioning: creating template: failed to compile bicep template: failed running bicep build: exit code: -1, stdout: , stderr: qemu: uncaught target signal 11 (Segmentation fault) - core dumped (signal: segmentation fault)

Expected/desired behavior

OS and Version?

macOS Ventura 13.3.1

Versions

Mention any other details that might be useful


Thanks! We'll be in touch soon.

acampb commented 1 year ago

Experiencing the same error using the devcontainer on macOS 13.3.1

vhvb1989 commented 1 year ago

Hello guys. @acampb @Ilarom

Can you please mention what azd version do you have run azd version

ellismg commented 1 year ago

From the error message, I would hazard a guess that you are on an M1 mac and for some reason we are downloading the amd64 version of bicep and trying to launch it and that is failing when run under the emulator.

It would be helpful to make sure you are on the latest version of azd and also delete ~/.azd/bin/bicep. This will cause azd to download the newest version of bicep and for the correct architecture.

ellismg commented 1 year ago

Just noticed that, @acampb mpb mentioned he was seeing this in a devcontainer. That's something I haven't tried before on my M1, and I could imagine it could bring in some weird cross architecture issues, but I'll give it a whirl.

A hunch I have here is that we might be trying to launch the macOS arm64 version we previously downloaded from within am amd64 linux dev container and that is unlikely to work great. In that case, you might unstick yourself by deleting ~/.azd/bin/bicep, which will force azd to download the correct version given the context it has when running inside the dev container.

ellismg commented 1 year ago

My hunch turned out to be incorrect - I was able to launch a devcontainer locally and confirm that ~/.azd/bin/ was a path inside the container, and the bicep that was downloaded was the correct amd64 version, as we would expect when running in an amd64 devcontainer.

I can reproduce the segfault as well. It is interesting - bicep --help does work, so we are clearly able to run at least some part of the bicep binary. Trying to get a better handle on what's going on here.

ellismg commented 1 year ago

Some light searching turned me on to the following option in the Docker for Mac. In the Docker Desktop app, if you go to settings and the "Features in development" you can enable using Rosetta for x86/amd64 emulation on Apple Silicon. I turned that on and after installing Rosetta (the act of enabling this feature had macOS prompt me to do so), I now get a different, much more .NET'y error:

node ➜ /workspaces/195 (main) $ ~/.azd/bin/bicep build ./infra/main.bicep --stdout
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Text.RegularExpressions.Regex.CreateRunner()
   at System.Text.RegularExpressions.Regex.RunSingleMatch(RegexRunnerMode, Int32, String, Int32, Int32, Int32)
   at System.Text.RegularExpressions.Regex.Match(String)
   at Bicep.Core.Resources.ResourceTypeReference.TryParse(String resourceType)
   at Bicep.Core.Resources.ResourceTypeReference.Parse(String resourceType)
   at Bicep.Core.TypeSystem.Az.AzResourceTypeLoader.<>c.<.ctor>b__4_0(KeyValuePair`2 kvp)
   at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass9_0`3.<ToImmutableDictionary>b__0(TSource)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1, MutationInput, KeyCollisionBehavior )
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1, Boolean)
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 )
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1, Func`2, Func`2, IEqualityComparer`1 , IEqualityComparer`1 )
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1, Func`2, Func`2)
   at Bicep.Core.TypeSystem.Az.AzResourceTypeLoader..ctor()
   at System.RuntimeMethodHandle.InvokeMethod(Object, Void**, Signature, Boolean)
   at System.Reflection.ConstructorInvoker.Invoke(Object, IntPtr*, BindingFlags)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags, Binder, Object[], CultureInfo)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type, ServiceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider, Type)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider)
   at Bicep.Cli.Program.RunAsync(String[] args)
   at Bicep.Cli.Program.Main(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)

So, obviously not great still, but maybe this means we are making more progress. It's possible we were going down this same error path before but qemu was segfaulting at some point while trying to handle throwing/printing the exception from .NET?

Ilarom commented 1 year ago

Hello guys. @acampb @Ilarom

Can you please mention what azd version do you have run azd version

azd version 0.9.0-beta.2

acampb commented 1 year ago

Hello guys. @acampb @Ilarom Can you please mention what azd version do you have run azd version

azd version 0.9.0-beta.2

Same azd

azd version 0.9.0-beta.2 (commit afa7ac6e839be8304b86b3df6b8525a97e48a532)

eliot-ye commented 1 year ago

Same

azd version 0.9.0-beta.2 (commit afa7ac6e839be8304b86b3df6b8525a97e48a532)

m1 mac 13.3.1

DayuanTan commented 1 year ago

Same here.

azd version 0.9.0-beta.2 (commit afa7ac6e839be8304b86b3df6b8525a97e48a532)

m2pro Mac 13.3.1

Error print out:

 (✓) Done: Resource group: rg-chatgptdemo_dev
  (✓) Done: Storage account: stfuqaba4s2krdy
  (✓) Done: Search service: gptkb-fuqaba4s2krdy
  (✓) Done: App Service plan: plan-fuqaba4s2krdy
  (✓) Done: Form recognizer: cog-fr-fuqaba4s2krdy

ERROR: deployment failed: failing invoking action 'provision', error deploying infrastructure: deploying to subscription:

Deployment Error Details:

  |       | Creating/Updating resourcesTraceID: d6f0ce0867e7eceb3bcbb3d9a4324b59
DayuanTan commented 1 year ago

The same error happened on my windows PC.

HP EliteBook 840. Intel i7-1185G7.

Same azd version. Same error message.

ellismg commented 1 year ago

For folks running into the 'segmentation fault, core dumped' issue, I spent some more time digging into this and it seems like there is an underlying bug with bicep when run on emulated amd64 linux containers. It seems tied some combination of running under the emulator via docker (I tried both qemu and the rosetta based emulation) and using a version of bicep compiled with .NET 7.0 (or this bug was introduced in the same release where the bicep team moved from using .NET 6.0 to build to .NET 7.0).

The dev containers use amd64 because when we authored them we didn't have a great story for azd on linux/arm. We've made some investments here, but the support is still in beta so for folks that want to live on the bleeding edge, you can try installing the linux/arm version of azd and using native arm64 containers.

The installer script does not yet support downloading these binaries, but you can fetch them from the github releases page for our most recent release. I made the following edits to the Dockerfile:

index 2751d6c..55b1852 100644
--- a/.devcontainer/Dockerfile
+++ b/.devcontainer/Dockerfile
@@ -1,6 +1,8 @@
 ARG VARIANT=bullseye
-FROM --platform=amd64 mcr.microsoft.com/vscode/devcontainers/base:0-${VARIANT}
+FROM mcr.microsoft.com/vscode/devcontainers/base:0-${VARIANT}
 RUN export DEBIAN_FRONTEND=noninteractive \
      && apt-get update && apt-get install -y xdg-utils \
      && apt-get clean -y && rm -rf /var/lib/apt/lists/*
-RUN curl -fsSL https://aka.ms/install-azd.sh | bash
+RUN mkdir -p /opt/microsoft/azd \
+     && curl -L https://github.com/Azure/azure-dev/releases/download/azure-dev-cli_0.9.0-beta.2/azd-linux-arm64-beta.tar.gz | tar zxvf - -C /opt/microsoft/azd \
+     && ln -s /opt/microsoft/azd/azd-linux-arm64 /usr/local/bin/azd

The first change removes our flag that was forcing amd64 and the second change replaces the use of the install script with downloading the beta version for arm-linux and then "installs" it as the script would. With these changes, I was able to launch a native version of AZD in a dev container on my M1 MacBook and provision.

@acampb and @Ilarom you might give that a whirl. Note we also publish macos-arm64 versions as well to our github releases, if you wanted to use the arm native versions. We'll be updating the install script to use these versions in a bit after we've gotten some more feedback from users.

acampb commented 1 year ago

@ellismg Confirming the Dockerfile changes you made above worked for me and resolved the issue. Performance is much better using the arm64 container image as well 🚀

ellismg commented 1 year ago

@ellismg Confirming the Dockerfile changes you made above worked for me and resolved the issue. Performance is much better using the arm64 container image as well 🚀

Thanks for letting me know, @acampb. Going to keep working on making this default, glad to hear things are working and that it is much faster (I observed that as well).

AllanChongNC commented 1 year ago

Similar error but running on Windows and PowerShell 7 after 'azd up'.

azd version: 1.0.2 bicep version: Bicep CLI version 0.18.4 python version: Python 3.11.4

`Provisioning Azure resources (azd provision) Provisioning Azure resources can take some time

ERROR: deployment failed: failing invoking action 'provision', planning deployment: planning infrastructure provisioning: creating template: failed to compile bicep template: failed running bicep build: exit code: 1, stdout: , stderr: 'C:\Users[Name]' is not recognized as an internal or external command, operable program or batch file.`

MedusaVM commented 1 year ago

Similar error but running on Windows and PowerShell 7 after 'azd up'.

azd version: 1.0.2 bicep version: Bicep CLI version 0.18.4 python version: Python 3.11.4

`Provisioning Azure resources (azd provision) Provisioning Azure resources can take some time

ERROR: deployment failed: failing invoking action 'provision', planning deployment: planning infrastructure provisioning: creating template: failed to compile bicep template: failed running bicep build: exit code: 1, stdout: , stderr: 'C:\Users[Name]' is not recognized as an internal or external command, operable program or batch file.`

I get the exact same error message. I'm using the same versions.

mratanusarkar commented 1 year ago

I am getting similar errors with ERROR: deployment failed: failing invoking action 'provision' but different full error stack traces. I tried it using "Windows 10 Enterprise" as well as "Azure Cloud Shell --> Bash"

Is this an issue with the "azure-search-openai-demo" itself? or am I doing something wrong? If anyone finds any solution, please do update. Thanks.

pamelafox commented 1 year ago

@mratanusarkar Please share your error / stack trace, as there are many things that could cause an issue. It's typically something with how your local environment is set up. The original error from this thread has since been resolved.

pamelafox commented 1 year ago

I'm closing this issue since the original M1 compatibility issue is resolved, but please open new issues if needed.

mratanusarkar commented 1 year ago

@pamelafox please take a look at this issue #547 then.

I understand that this perticular issue was related to the local environment and M1, so it's better to discuss it on this seperate issue.

developer992 commented 10 months ago

I have the same issue on Intel i7 ( windows 11 )

not quite the same but similar:

ERROR: deployment failed: failing invoking action 'provision', error deploying infrastructure: starting deployment to subscription: PUT https://management.azure.com/subscriptions/2ec52....8116e/providers/Microsoft.Resources/deployments/foo-bar-1704376813
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: AuthorizationFailed
--------------------------------------------------------------------------------
{
  "error": {
    "code": "AuthorizationFailed",
    "message": "The client 'email' with object id 'a9100645-22cb-4bbf-b454-2ed3f4473849' does not have authorization to perform action 'Microsoft.Resources/deployments/write' over scope '/subscriptions/2ec52....8116e/providers/Microsoft.Resources/deployments/foo-bar-1704376813' or the scope is invalid. If access was recently granted, please refresh your credentials."
  }
}
--------------------------------------------------------------------------------

TraceID: 5365da5ea717eabca37f032877fa4049