Todo app fails to connect to database API or start debug session on M1 Mac

[uncvrd]

Hello! I was interested in BTK to debug pods on my M1 Mac so I've been stepping through the tutorial here however I run in to two issues (possibly related).

ISSUE 1. After deploying the app to pods and opening up the frontend app, I see the following error. Which states that the database api failed to start:

ISSUE 2. When starting a debug session, the process hangs on Preparing to run Bridge To Kubernetes configured as pod todo-app/[POD ID]

---

Here are the steps I followed

1. I cloned the todo repo.
2. Created a todo-app namespace, set it as active
3. Applied the deployment (see image below of pods running

4. Ran minikube tunnel and waited for IP to resolve
5. Opened up 127.0.0.1:80 to view frontend, and was presented with the image you see at the very top.
6. I then kubectl logs the database-api and saw the following output:

MongoHelper initialization start
MongoHelper initialization end
Unhandled exception. System.IO.IOException: Function not implemented
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1..ctor(Func`1 changeTokenProducer, Action`1 changeTokenConsumer, TState state)
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at DatabaseApi.Program.Main(String[] args) in /src/Program.cs:line 18
qemu: uncaught target signal 6 (Aborted) - core dumped

I thought I might be able to still run a debug session, so I continued on...

7. Ran BTK - Configure. Selected stats-api, selected PORT 3001, the default profile to debug (which was in the cloned repo) and no isolation.
8. Ran Run Script: dev with Kubernetes
9. Received the following output:

2021-06-10T20:32:58.3262760Z | MindaroCli | TRACE | Initializing progress reporter...
2021-06-10T20:33:00.3682330Z | MindaroCli | TRACE | Remoting started listening on 53455
2021-06-10T20:33:05.0936440Z | MindaroCli | TRACE | Waiting for 'stats-api-685ff8b766-x4vck' in namespace 'todo-app' to reach running state...\n
2021-06-10T20:33:06.8924910Z | MindaroCli | TRACE | Deployment 'todo-app/stats-api' patched to run agent.\n
2021-06-10T20:33:06.9049410Z | MindaroCli | TRACE | Found container 'stats-api' in pod 'stats-api-685ff8b766-x4vck'.\n
2021-06-10T20:33:06.9098190Z | MindaroCli | TRACE | Preparing to run Bridge To Kubernetes configured as pod todo-app/stats-api-685ff8b766-x4vck ...\n

Where it hangs and errors on that step...

10. I then printed out the error log file to this zip, where you can see an error: ERROR | Dependency: Service Run - Port Forward <json>{"target":null,"success":false,"duration":null,"properties": and then some C# connection errors (Log attached below).

mindarocli.txt

I had the same connection issue on my M1 with my personal project which is why I went down this route of running the tutorial and I see that I receive the same problem with the tutorial app so maybe all of this it has something do with the fact that I'm on an M1 environment?

Please let me know if you have any more questions! I know that was a lot, so let me know if I can clarify as well.

Thanks!

Environment Details Client used (VS Code/Visual Studio): VS Code Client's version: BTK Plugin v1.0.120210605

[pragyamehta]

Hi @uncvrd Thanks for reporting this issue! Bridge to Kubernetes will not be able to work if the application is not in a running state and it sounds from what you have shared that the app is not running when you attempt to debug it with Bridge to Kubernetes. I think we need to get the app setup and running first. Can you confirm you are on the latest code in the main branch? Also when you see this error, try restarting the database-api pod and try to hit the endpoint. Let me know if this works. Thanks!

[uncvrd]

@pragyamehta hi! I agree that the app is not running. However I'm not sure what step I missed as it shows that all of the pods are running.

When in the directory with the deployment.yaml, I simply ran: kubectl apply -n todo-app -f deployment.yaml, which should start running all services, right? (The second image in my original post shows all pods running).

I cloned the repo from the master branch a few hours ago and changed directories in to the todo app directory:

Here is my flow that shows me restarting the database-api. Numbers reflect:

1. Listing the deployments and then running kubectl -n todo-app rollout restart deployment database-api to restart the pods in the database-api deployment
2. Getting the pods to show that the pod is terminating and being replaced
3. The pods are fulling running

Hitting the front end, I still receive the same result.

Error: Failed to call API '/api/todos'. Make sure that 'database-api' and 'todos-db' are running properly. Details: Internal Server Error

I've tried on two different M1 computers (My iMac and Macbook) so I believe I'm running the instructions correctly! What other information can I provide?

Thanks!

---

EDIT

Also just a reminder when I run logs in the database-api I have this error:

MongoHelper initialization start
MongoHelper initialization end
Unhandled exception. System.IO.IOException: Function not implemented
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1..ctor(Func`1 changeTokenProducer, Action`1 changeTokenConsumer, TState state)
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at DatabaseApi.Program.Main(String[] args) in /src/Program.cs:line 18
qemu: uncaught target signal 6 (Aborted) - core dumped

So this explains that the database api is not running ...but I'm not quite sure how to get it running since the source code fails haha

[pragyamehta]

@uncvrd Thanks for the info! It looks like you're following the right set of steps. Have you tried on a different kubernetes cluster than minikube? If not, would it be possible to do so? Also are you running the latest version of minikube on your machine? I see you tried on two different Mac machines - do they have the same version of OS? What are the minikube versions running on both of them?

[uncvrd]

@pragyamehta

Have you tried on a different kubernetes cluster than minikube? If not, would it be possible to do so?

Unfortunately I have not as this is my first foray in to Kubernetes. Are there other local alternatives you would suggest? I could try spinning up something on Digital Ocean but the personal goal was not to pay for anything until I understood how to get something running local haha.

Also are you running the latest version of minikube on your machine?

MacBook: minikube version: v1.20.0 iMac: minikube version: v1.20.0

My iMac is running Big Sur 11.3.1 while my iMac is running Big Sur 11.1.

My thought leads me to believe that the image used for the database-api is not a multi-arch image that can be used with arm64 M1 macs as I've had this problem with other images while trying to get all my projects set up on M1. Is that too far off path?

---

Edit

I see that there is this open issue relating to M1 macs and NET images:

https://github.com/dotnet/runtime/issues/44897

I'm in the processes of stripping down a node example to get past this potential image issue to see if I still have issues connecting a debugger as well

[uncvrd]

Alright! @pragyamehta see above and also reporting back with an extremely stripped down Node.js example (using express). So regardless if the database-api image doesn't work, I receive an identical error while trying to debug using BTK on M1.

Example Repo: https://github.com/uncvrd/node-btk-example

Log output file (mindarocli logs):

log.txt

Bridge library logs:

log2.txt

In short, it fails at this line:

2021-06-11T01:15:50.2088290Z | MindaroCli | ERROR | Dependency: Service Run - Port Forward <json>{"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}}</json>

And then proceeds to continue on to a C# style error that looks like something I'd see in dotnet with HttpConnection. Does this help?

[pragyamehta]

@uncvrd Another option for a free kubernetes cluster would be Docker Desktop Kubernetes

The node.js example you tried - did you verify that the app was running before starting debugging with Bridge? From the logs it seems like we lost connectivity to a pod in your environment - it could be because of network issues or if the pod is recycling.

I think to isolate where this problem is coming from, I would recommend running the databases-api image locally as a docker container and see the results. docker run --env MONGO_CONNECTION_STRING=mongodb://todos-db azdspublic/todo-app-database-api

Let me know what error you see when you run the above command.

[uncvrd]

@pragyamehta

Thanks! I'll check that option out shortly. Luckily going to get access to an Intel computer shortly so I can test there as well.

Did I verify the app was running?

Yes I was able to confirm the app was running in the Pod by using Postman to run a GET request and return "Hello World" after exposing the IP using minikube tunnel.

Let me know what error you see when you run the above command (relating to database-api)

When running this command I see following log, which is basically inline with my hypothesis about it not running on M1 (note the WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8))

jordan@Jordans-MacBook-Pro-2 ~ % docker run --env MONGO_CONNECTION_STRING=mongodb://todos-db azdspublic/todo-app-database-api
Unable to find image 'azdspublic/todo-app-database-api:latest' locally
latest: Pulling from azdspublic/todo-app-database-api
6ec7b7d162b2: Pull complete 
f48adbf33222: Pull complete 
0caf687f11cc: Pull complete 
31f7e18202e6: Pull complete 
b2b381c9c354: Pull complete 
9056499c7a9f: Pull complete 
6b28ff76e62c: Pull complete 
Digest: sha256:f5167284a7f37c438404e402ac2b698706f48ce1fdd0b4d0d96102e237d6d324
Status: Downloaded newer image for azdspublic/todo-app-database-api:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
MongoHelper initialization start
MongoHelper initialization end
Unhandled exception. System.IO.IOException: Function not implemented
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.ChangeTokenRegistration`1..ctor(Func`1 changeTokenProducer, Action`1 changeTokenConsumer, TState state)
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at DatabaseApi.Program.Main(String[] args) in /src/Program.cs:line 18
qemu: uncaught target signal 6 (Aborted) - core dumped

[uncvrd]

@pragyamehta hello! Also reporting back with the above that using the same nodejs repo above on an Intel mac works as expected, so I believe there is an issue using the M1 with Bridge to Kubernetes.

For example, when running the node program on intel, I successfully connected with the following logs:

Waiting for 'node-api-deployment-7c8bf84988-xgs8d' in namespace 'default' to reach running state...
Deployment 'default/node-api-deployment' patched to run agent.
Found container 'node-api-container' in pod 'node-api-deployment-7c8bf84988-xgs8d'.
Preparing to run Bridge To Kubernetes configured as pod default/node-api-deployment-7c8bf84988-xgs8d ...
Connection established.
Service 'api-service' is available on 127.1.1.1:8080.
Service 'kubernetes' is available on 127.1.1.2:443.
Hosts file updated.
Container port 8080 is available at localhost:8080.
##################### Environment started. #############################################################

[pragyamehta]

@uncvrd Thanks for reporting this! I will log a bug on our end.

[uncvrd]

@pragyamehta you're welcome! Happy to help, let me know if there's anything I can do to assist

[JamesAskew]

@pragyamehta is there an update on this? I'm running in to the same issue on an M1 mac.

[pkoelbl]

The Unhandled exception. System.IO.IOException: Function not implemented is likely being caused by .net core 3.1 not really working well on an m1 mac. .net6 has better mac m1 compatibility i have found. However once you get past that you will likely run into https://github.com/microsoft/mindaro/issues/280 given the arm64 nature of M1 macs