search

Azure / azure-cosmos-db-emulator-docker

This repo serves as hub for managing issues, gathering feedback, and having discussions regarding the Cosmos DB Emulator Docker.
https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Ccsharp&pivots=api-nosql
MIT License
149 stars 45 forks source link

CosmosDB linux emulator docker container fails to start #77

Open gevoulga opened 9 months ago

gevoulga commented 9 months ago

Describe the bug

Trying to start linux cosmosdb emulator. Getting startup exception.

docker container logs:

023-11-07 13:16:20 This is an evaluation version.  There are [173] days left in the evaluation period.
2023-11-07 13:16:23 Starting
2023-11-07 13:16:23 Failed to create CoreCLR, HRESULT: 0x8007000E
2023-11-07 13:16:41 Started 1/4 partitions
2023-11-07 13:16:42 Started 2/4 partitions
2023-11-07 13:16:43 Started 3/4 partitions
2023-11-07 13:16:54 Started 4/4 partitions
2023-11-07 13:16:54 Started
2023-11-07 13:17:15 
2023-11-07 13:17:15 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-07 13:17:15    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-07 13:17:15    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-07 13:17:15    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-07 13:17:15    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:17:15 --- End of stack trace from previous location where exception was thrown ---
2023-11-07 13:17:15    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-07 13:17:15    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-07 13:17:15    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:17:15    --- End of inner exception stack trace ---
2023-11-07 13:17:15    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-07 13:17:15    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-07 13:17:15    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-07 13:17:15    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-07 13:17:15    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-07 13:17:15    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-07 13:17:15    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-07 13:17:15    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-07 13:17:15 Failed to create CoreCLR, HRESULT: 0x8007000E
2023-11-07 13:18:07 
2023-11-07 13:18:07 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-07 13:18:07    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-07 13:18:07    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-07 13:18:07    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-07 13:18:07    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:18:07 --- End of stack trace from previous location where exception was thrown ---
2023-11-07 13:18:07    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-07 13:18:07    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-07 13:18:07    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:18:07    --- End of inner exception stack trace ---
2023-11-07 13:18:07    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-07 13:18:07    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-07 13:18:07    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-07 13:18:07    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-07 13:18:07    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-07 13:18:07    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-07 13:18:07    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-07 13:18:07    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-07 13:18:07 Failed to create CoreCLR, HRESULT: 0x8007000E
2023-11-07 13:18:58 
2023-11-07 13:18:58 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-07 13:18:58    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-07 13:18:58    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-07 13:18:58    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-07 13:18:58    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:18:58 --- End of stack trace from previous location where exception was thrown ---
2023-11-07 13:18:58    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-07 13:18:58    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-07 13:18:58    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:18:58    --- End of inner exception stack trace ---
2023-11-07 13:18:58    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-07 13:18:58    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-07 13:18:58    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-07 13:18:58    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-07 13:18:58    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-07 13:18:58    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-07 13:18:58    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-07 13:18:58    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-07 13:18:58 Failed to create CoreCLR, HRESULT: 0x8007000E
2023-11-07 13:19:50 
2023-11-07 13:19:50 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-07 13:19:50    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-07 13:19:50    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-07 13:19:50    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-07 13:19:50    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:19:50 --- End of stack trace from previous location where exception was thrown ---
2023-11-07 13:19:50    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-07 13:19:50    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-07 13:19:50    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:19:50    --- End of inner exception stack trace ---
2023-11-07 13:19:50    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-07 13:19:50    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-07 13:19:50    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-07 13:19:50    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-07 13:19:50    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-07 13:19:50    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-07 13:19:50    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-07 13:19:50    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-07 13:19:50 Failed to create CoreCLR, HRESULT: 0x8007000E
2023-11-07 13:20:42 
2023-11-07 13:20:42 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-07 13:20:42    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-07 13:20:42    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-07 13:20:42    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-07 13:20:42    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:20:42 --- End of stack trace from previous location where exception was thrown ---
2023-11-07 13:20:42    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-07 13:20:42    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-07 13:20:42    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:20:42    --- End of inner exception stack trace ---
2023-11-07 13:20:42    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-07 13:20:42    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-07 13:20:42    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-07 13:20:42    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-07 13:20:42    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-07 13:20:42    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-07 13:20:42    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-07 13:20:42    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-07 13:20:42 Failed to create CoreCLR, HRESULT: 0x8007000E
2023-11-07 13:21:33 
2023-11-07 13:21:33 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-07 13:21:33    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-07 13:21:33    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-07 13:21:33    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-07 13:21:33    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:21:33 --- End of stack trace from previous location where exception was thrown ---
2023-11-07 13:21:33    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-07 13:21:33    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-07 13:21:33    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 13:21:33    --- End of inner exception stack trace ---
2023-11-07 13:21:33    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-07 13:21:33    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-07 13:21:33    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-07 13:21:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-07 13:21:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-07 13:21:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-07 13:21:33    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-07 13:21:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-07 13:21:33 Failed to create CoreCLR, HRESULT: 0x8007000E

To Reproduce

use the docker compose below. check container logs.

version: '3.8'

networks:
  test-network:
    external: false
    ipam:
      driver: default
      config:
        - subnet: "172.16.238.0/24"

services:
  cosmosdb:
    image: 'mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest'
    platform: linux
    mem_limit: 4GB
    tty: true
    ports:
      - '8081:8081' # Data Explorer
      - '8900:8900'
      - '8901:8901'
      - '8902:8902'
      - '10250:10250'
      - '10251:10251'
      - '10252:10252'
      - '10253:10253'
      - '10254:10254'
      - '10255:10255'
      - '10256:10256'
      - '10350:10350'
    environment:
      AZURE_COSMOS_EMULATOR_PARTITION_COUNT: 3
      AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE: false
      AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE: 172.16.238.246
    networks:
      test-network:
        ipv4_address: 172.16.238.246

Docker Environment

v1k1 commented 9 months ago

Hi @gevoulga, Thanks for attaching the Docker settings screenshot. Please enable WSL 2 based engine, which is required for CosmosDb Emulator Linux docker.

madhancock commented 9 months ago

Not sure if this is much help but my entire team have been having this issue but one of them has just limited their WSL2 resources to match my machine (which was working) and it resolved the issue for them.

[wsl2]
memory=8GB
processors=2

https://learn.microsoft.com/en-us/windows/wsl/wsl-config#wslconfig

So this might be a race condition of sorts?

jack-edwards-fs commented 9 months ago

I am encountering the same error with WSL2 already enabled.

image

2023-11-07 11:22:33 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-07 11:22:33    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-07 11:22:33    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 11:22:33 --- End of stack trace from previous location where exception was thrown ---
2023-11-07 11:22:33    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-07 11:22:33    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 11:22:33    --- End of inner exception stack trace ---
2023-11-07 11:22:33    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-07 11:22:33    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-07 11:22:33 Failed to create CoreCLR, HRESULT: 0x8007000E
its-mash commented 9 months ago

It was working fine before, just wondering if any new release of cosmosDb causing the issue. Where can I see the previous release of cosmosDB emulator?

vsevolod-mozgovoi-am commented 9 months ago

This work for me too. Just add .wslconfig file if you don't have one and there will be no more errors

its-mash commented 9 months ago

In my case, changing mem limit to 4gb from 3gb in docker-compose file fixed the issue. I was trying with 10 partitions.

mem_limit: 4G

abkolant-MSFT commented 9 months ago

I am encountering the same error with WSL2 already enabled.

image

2023-11-07 11:22:33 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-07 11:22:33    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-07 11:22:33    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 11:22:33 --- End of stack trace from previous location where exception was thrown ---
2023-11-07 11:22:33    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-07 11:22:33    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-07 11:22:33    --- End of inner exception stack trace ---
2023-11-07 11:22:33    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-07 11:22:33    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-07 11:22:33    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-07 11:22:33 Failed to create CoreCLR, HRESULT: 0x8007000E

@jack-edwards-fs - Can you share the params you're using to start the emulator? Is it a docker compose or docker run? If it's docker compose can you try with mem_limit:4g and if it's docker run can you try with docker run -m 4g

jack-edwards-fs commented 9 months ago

@abkolant-MSFT Yes. Here was my existing docker-compose.yml configuration:

  cosmosdb:
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
    expose:
      - "8081"
    mem_limit: 3GB
    ports:
      - "8081:8081"
    profiles:
      - db
      - dev
    environment:
      AZURE_COSMOS_EMULATOR_PARTITION_COUNT: 2
      AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE: true
      AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE: ${COSMOS_EMULATOR_IP_OVERRIDE}

However, adding the .wslconfig file as suggested above does act as a resolution, without changing any parameters in my compose file.

Removing the .wslconfig file and modifying the mem_limit from 3GB to 4g still produces the errors below.

2023-11-09 08:28:23 This is an evaluation version.  There are [171] days left in the evaluation period.
2023-11-09 08:28:25 Starting
2023-11-09 08:28:25 Failed to create CoreCLR, HRESULT: 0x8007000E
2023-11-09 08:28:43 Started 1/3 partitions
2023-11-09 08:28:43 Started 2/3 partitions
2023-11-09 08:28:56 Started 3/3 partitions
2023-11-09 08:28:56 Started
2023-11-09 08:29:16 
2023-11-09 08:29:16 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-09 08:29:16    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-09 08:29:16    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-09 08:29:16    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-09 08:29:16    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-09 08:29:16 --- End of stack trace from previous location where exception was thrown ---
2023-11-09 08:29:16    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-09 08:29:16    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-09 08:29:16    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-09 08:29:16    --- End of inner exception stack trace ---
2023-11-09 08:29:16    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-09 08:29:16    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-09 08:29:16    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-09 08:29:16    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-09 08:29:16    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-09 08:29:16    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-09 08:29:16    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-09 08:29:16    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-09 08:29:17 Failed to create CoreCLR, HRESULT: 0x8007000E
abkolant-MSFT commented 9 months ago

Apologies, I made a typo, it should have been mem_limit: 4GB instead of 4g could you try it with that. Can you confirm if it works with this? You could use this as a reference?

jack-edwards-fs commented 9 months ago

Current docker-compose.yml file with the .wslconfig file still deleted:

  cosmosdb:
    restart: always
    container_name: "azure-cosmosdb-emulator"
    hostname: "azurecosmosemulator"
    image: 'mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest'
    mem_limit: 4GB
    tty: true
    ports:
      - '8081:8081' # Data Explorer
      - '8900:8900'
      - '8901:8901'
      - '8902:8902'
      - '10250:10250'
      - '10251:10251'
      - '10252:10252'
      - '10253:10253'
      - '10254:10254'
      - '10255:10255'
      - '10256:10256'
      - '10350:10350'
    profiles:
      - db
    environment:
      - AZURE_COSMOS_EMULATOR_PARTITION_COUNT=3
      - AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false

Container output:

2023-11-09 08:41:22 This is an evaluation version.  There are [171] days left in the evaluation period.
2023-11-09 08:41:24 Starting
2023-11-09 08:41:24 Failed to create CoreCLR, HRESULT: 0x8007000E
2023-11-09 08:41:42 Started 1/4 partitions
2023-11-09 08:41:43 Started 2/4 partitions
2023-11-09 08:41:43 Started 3/4 partitions
2023-11-09 08:41:55 Started 4/4 partitions
2023-11-09 08:41:55 Started
2023-11-09 08:42:15 
2023-11-09 08:42:15 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
2023-11-09 08:42:15    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
2023-11-09 08:42:15    at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
2023-11-09 08:42:15    at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
2023-11-09 08:42:15    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-09 08:42:15 --- End of stack trace from previous location where exception was thrown ---
2023-11-09 08:42:15    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2023-11-09 08:42:15    at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
2023-11-09 08:42:15    at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
2023-11-09 08:42:15    --- End of inner exception stack trace ---
2023-11-09 08:42:15    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
2023-11-09 08:42:15    at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
2023-11-09 08:42:15    at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
2023-11-09 08:42:15    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
2023-11-09 08:42:15    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
2023-11-09 08:42:15    at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
2023-11-09 08:42:15    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
2023-11-09 08:42:15    at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
2023-11-09 08:42:15 Failed to create CoreCLR, HRESULT: 0x8007000E
adamzest commented 9 months ago

I have the same problem as does a colleague.

docker rmi mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
docker image pull mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
docker run `
    --publish 8081:8081 `
    --publish 10250-10255:10250-10255 `
    --memory 4g --cpus=2.0 `
    --name=cosmos-linux-emulator `
    --env AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 `
    --env AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true `
    --env AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1 `
    --tty `
    --restart unless-stopped `
    -d `
    mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator

Container errors:

2023-11-10 17:58:44 This is an evaluation version. There are [170] days left in the evaluation period. 2023-11-10 17:58:49 Starting 2023-11-10 17:58:49 Failed to create CoreCLR, HRESULT: 0x8007000E 2023-11-10 17:59:16 Started 1/11 partitions 2023-11-10 17:59:16 Started 2/11 partitions 2023-11-10 17:59:17 Started 3/11 partitions 2023-11-10 17:59:18 Started 4/11 partitions 2023-11-10 17:59:19 Started 5/11 partitions 2023-11-10 17:59:20 Started 6/11 partitions 2023-11-10 17:59:21 Started 7/11 partitions 2023-11-10 17:59:22 Started 8/11 partitions 2023-11-10 17:59:22 Started 9/11 partitions 2023-11-10 17:59:23 Started 10/11 partitions 2023-11-10 17:59:29 Started 11/11 partitions 2023-11-10 17:59:29 Started 2023-11-10 17:59:45 2023-11-10 17:59:45 Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified 2023-11-10 17:59:45 at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo) 2023-11-10 17:59:45 at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args) 2023-11-10 17:59:45 at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.b0() 2023-11-10 17:59:45 at Microsoft.Azure.Documents.BackoffRetryUtility`1.d52.MoveNext() 2023-11-10 17:59:45 --- End of stack trace from previous location where exception was thrown --- 2023-11-10 17:59:45 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 2023-11-10 17:59:45 at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) 2023-11-10 17:59:45 at Microsoft.Azure.Documents.BackoffRetryUtility1.d5`2.MoveNext() 2023-11-10 17:59:45 --- End of inner exception stack trace --- 2023-11-10 17:59:45 at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) 2023-11-10 17:59:45 at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile) 2023-11-10 17:59:45 at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile) 2023-11-10 17:59:45 at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath) 2023-11-10 17:59:45 at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context) 2023-11-10 17:59:45 at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.b3_1() 2023-11-10 17:59:45 at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args) 2023-11-10 17:59:45 at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args) 2023-11-10 17:59:45 Failed to create CoreCLR, HRESULT: 0x8007000E

cbayles commented 9 months ago

For me it was a combination of adding the C:\Users\<UserName>\.wslconfig file with content stated above, updating my docker-compose.yml for cosmosdb to have mem_limit: 4GB as stated above, and restarting Docker Desktop.

adamzest commented 9 months ago

I've had to increase the docker container memory to 5g to get it to start (previously was able to run it with 2g). Doesnt matter what I do to .wslconfig - the container needs 5gb to run (thats a completely empty container)... pretty horrific memory usage jump.

Thats despite it only really using: image

madhancock commented 9 months ago

Not sure if this is much help but my entire team have been having this issue but one of them has just limited their WSL2 resources to match my machine (which was working) and it resolved the issue for them.

[wsl2]
memory=8GB
processors=2

https://learn.microsoft.com/en-us/windows/wsl/wsl-config#wslconfig

So this might be a race condition of sorts?

For additional info, on all our dev machines we have the emulator working with wsl.config set to 2 processors and 8GB ram. Our cosmosdb container is limited to 3GB:

cosmosdb:
    container_name: "azurecosmosemulator"
    hostname: "azurecosmosemulator"
    environment:
      AZURE_COSMOS_EMULATOR_KEY: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==
      AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE: true
    image: 'mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator'
    platform: linux
    tty: true
    mem_limit: 3GB
    ports:
        - '8081:8081'
        - '8900:8900'
        - '8901:8901'
        - '8902:8902'
        - '10250:10250'
        - '10251:10251'
        - '10252:10252'
        - '10253:10253'
        - '10254:10254'
        - '10255:10255'
        - '10256:10256'
        - '10350:10350'
    volumes:
      - cosmosdb:/tmp/cosmos/appdata

All on WSL2 Ubuntu 22.04 on Linux containers. The devs that were affected by this had the same setup but without wsl.config - so in their cases their Docker had 30-60GB ram and 8-10 cores to play with. All experienced the error described in this issue until they applied the limits.

ruiqiliu-db commented 9 months ago

If I am running it on Linux machine (not involving Window at all), does the wslconfig matter? How to set that on Linux machine?

Setting memory limit to 4GB worked for me on one type of Linux VM but not on another type, so want to try setting wslconfig.

madhancock commented 9 months ago

If I am running it on Linux machine (not involving Window at all), does the wslconfig matter? How to set that on Linux machine?

Setting memory limit to 4GB worked for me on one type of Linux VM but not on another type, so want to try setting wslconfig.

If your machine is entirely Linux and not running inside WSL (Windows Subsystem for Linux) then WSL config isn't going to be possible.

If there are CPU core count differences between those Linux VMs then that is probably the only thing I could suggest tinkering with to get it working until someone from Microsoft gets to the bottom if the issue.

v1k1 commented 9 months ago

If I am running it on Linux machine (not involving Window at all), does the wslconfig matter? How to set that on Linux machine?

Setting memory limit to 4GB worked for me on one type of Linux VM but not on another type, so want to try setting wslconfig.

Hi @ruiqiliu-db, For Linux systems, wsl config should not matter. By the way can you please elaborate for which Linux system you are facing problem?

ruiqiliu-db commented 9 months ago

Thanks for the reply. @v1k1 I am using EC2 Linux VMs, but different types. On m6i.4xlarge, I need to set memory limit to 4 GiB to make it work (the default value of 3 GiB did not work); on a larger EC2 instance like m5d.16xlarge, 4 GiB did not work either. Is CPU count making a difference here? Is there a way to make a value work regardless of VM type?

v1k1 commented 8 months ago

Hi, we were recently able to reproduce this issue for CosmosDB Linux docker. Here are the steps to fix this - There are two cases-

  1. Host OS is Windows - In case of windows this issue is usually occurring when user had previous versions of emulator Linux docker running and they upgraded. For fresh installations this issue doesn't occur. For this case, please follow these steps -

    1. Make sure in Docker Desktop preference 'Use the WSL 2 based engine' is enabled.
    2. If you still face problems, make sure you have Ubuntu 20.20 distribution installed (From Microsoft App Store) and set it to default.
      • check installed wsl distributions using wsl -l -v
        • Set default wsl distribution to Ubuntu-20.20 - wsl --set-default Ubuntu-22.04
    3. Make sure you quit docker and shutdown wsl afterwards for these settings to be picked again.
    4. If Ubuntu 20.20 is set as default to begin with, Try changing it to some other distribution and re-setting to Ubuntu 20.20.

  2. Host OS is Linux We have been unable to reproduce this if Host is Linux. @ruiqiliu-db, can you please confirm if you are facing the Exactly same exception with Linux is Host OS as well.

adamzest commented 7 months ago

Does this mean that the minimum requirement to run Cosmos on windows is WSL2? Unfortunately our Windows 2019 servers (WSL1) are unable to start the Cosmos container at all. All I get is 'Aborted' when trying to start the container.

sponder808 commented 7 months ago

@v1k1 I can confirm via a colleague that the exact same error is occurring on a Linux host (Ubuntu 20.04.4 LTS). She has tried modifying the mem_limit from 3g to 4g but still reports the error.

Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at Microsoft.Azure.Documents.Common.ProcessUtil.Run(String fileName, String args)
   at Microsoft.Azure.Documents.Common.EtwManifest.<>c__DisplayClass7_0.<UnInstallEtwManifest>b__0()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5`2.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.Azure.Documents.Common.EtwManifest.UnInstallEtwManifest(String manifestFile)
   at Microsoft.Azure.Documents.Common.EtwManifest.InstallEtwManifest(String manifestFile)
   at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceSetupHelper.InstallRuntimeSourceManifest(String manifestPath)
   at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.GatewayInit(CodePackageActivationContext context)
   at Microsoft.Azure.Documents.Services.StartupEntryPoint.GatewayServiceSetupCommand.<>c.<AddCommand>b__3_1()
   at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Microsoft.Azure.Documents.Services.StartupEntryPoint.ServiceStartupEntryPoint.Main(String[] args)
Failed to create CoreCLR, HRESULT: 0x8007000E
version: "3.9"

  database:
    # https://docs.microsoft.com/en-us/azure/cosmos-db/linux-emulator
    # Once running, the CosmosDB emulator can be accessed using a browser on the host at
    # https://localhost:1241/_explorer/index.html
    container_name: cosmos-db-emulator-dev
    hostname: cosmos-db.local
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
    cpus: 2.0
    mem_limit: 3g
    ports:
        # Ports mostly derived from https://docs.microsoft.com/en-us/azure/cosmos-db/emulator-command-line-parameters
        - 8081:8081 # Default emulator port
        - 8901:8901 # Gremlin port
        - 8902:8902 # Table port
        - 10251:10251 # CosmosDB direct port
        - 10252:10252 # CosmosDB direct port
        - 10253:10253 # CosmosDB direct port
        - 10254:10254 # CosmosDB direct port
        - 10255:10255 # MongoDB port
        - 10350:10350 # Cassandra port
    environment:
        # Env variables mostly derived from https://docs.microsoft.com/en-us/azure/cosmos-db/emulator-command-line-parameters
        # and from https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator?tabs=ssl-netstd21
        # Comment in/out CosmosDB features as needed:
        - AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10
        - AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true
        - AZURE_COSMOS_EMULATOR_GREMLIN_ENDPOINT=true
        # - AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE="${IP_ADDRESS}"
        # - AZURE_COSMOS_EMULATOR_CASSANDRA_ENDPOINT=true
        # - AZURE_COSMOS_EMULATOR_MONGODB_ENDPOINT=true
        # - AZURE_COSMOS_EMULATOR_ENABLE_TABLE_ENDPOINT=true
    networks:
        cosmosdb-network_dev:
          ipv4_address: 172.25.0.99 # we use *.*.0.99 to leave some addresses for additional dependencies

networks:
   cosmosdb-network_dev:
      name: cosmos_db_network_dev
      ipam:
         driver: default
         config:
            - subnet: "172.25.0.0/16"
gklijs commented 7 months ago

Running on wsl2 with 20.04.6 and running the container via test containers is working for me.

So maybe the testcontainer source might help. It might also depend on the processor, and thus related to https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/45.

DavidParks8 commented 7 months ago

I found the exact line internally and gave a proposed fix. I will follow up internally until this is resolved.

nikolayarhangelov commented 7 months ago

Running into a similar issue, however the Docker Compose file is generated by Visual Studio using the Connected Services feature for local development. The yml file has a limit of 2G. Unfortunately when I switch it to 4G and start the local dependencies in Visual Studio, it will revert it back to 2G.

Linked issue: https://github.com/MicrosoftDocs/visualstudio-docs/issues/9970

image

version: "3.9"

services:
  cosmosdb:
    image: "mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator"
    ports: 
        - 8081:8081
        - 10251:10251
        - 10252:10252
        - 10253:10253
        - 10254:10254
    tty: true
    restart: always
    mem_limit: 2G
    cpu_count: 2
    environment:
      - AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10
      - AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true
      - AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=192.168.0.10
    volumes:
      - cosmosdb_data:/data/db

volumes: 
  cosmosdb_data:
ProTip commented 2 months ago

I have done quite a bit of testing on this locally and have narrowed it down to the number of CPU cores presented to the WSL VM. My system has 32 available, and I can only assume that the sight of such power causes CosmosDb to explode.

The only thing that worked for me was hiding my system's true power: ~\.wslconfig 

[wsl2]
processors=8

If I see Failed to create CoreCLR, HRESULT: 0x8007000E I will never be able to connect.