search

fsprojects / Paket

A dependency manager for .NET with support for NuGet packages and Git repositories.
https://fsprojects.github.io/Paket/
MIT License
2.02k stars 525 forks source link

Paket crashes on mono 5.0.1.1 #2393

Open matthid opened 7 years ago

matthid commented 7 years ago

Description

See https://s3.amazonaws.com/archive.travis-ci.org/jobs/238510496/log.txt?X-Amz-Expires=30&X-Amz-Date=20170603T212227Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJRYRXRSVGNKPKO5A/20170603/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=7d23056e538543072063a4234e00d1d898dea08db7c5cb98c40ded6fd2c4f903 search for "FATAL UNHANDLED EXCEPTION"

After some analysis of the output my assumption is that we fail on this line: https://github.com/fsprojects/Paket/blob/master/src/Paket/Commands.fs#L445

And reason is mono introduced some race condition. I'd say it should be reproducable by calling paket --version in a loop until the runtime crashes.

This is a reminder to try to reproduce this and report a mono bug. But as I have found already sooo many bugs recently which are much worse (at least from my perspective - mono support is not the most urgent thing atm) this will lay around quite some time (hopefully they have just fixed this until then).

If someone wants to take it feel free :)

Error looks like this:

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.MonoCustomAttrs.GetCustomAttributesInternal (System.Reflection.ICustomAttributeProvider,System.Type,bool) [0x00000] in <a07d6bf484a54da2861691df910339b1>:0
  at System.MonoCustomAttrs.GetCustomAttributesBase (System.Reflection.ICustomAttributeProvider,System.Type,bool) [0x00013] in <a07d6bf484a54da2861691df910339b1>:0
  at System.MonoCustomAttrs.GetCustomAttributes (System.Reflection.ICustomAttributeProvider,System.Type,bool) [0x00037] in <a07d6bf484a54da2861691df910339b1>:0
  at System.Reflection.MonoProperty.GetCustomAttributes (System.Type,bool) [0x00000] in <a07d6bf484a54da2861691df910339b1>:0

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object

and only happens from time to time

matthid commented 7 years ago

Description

original Bugreport


  Stacktrace:

    at <unknown> <0xffffffff>
    at (wrapper managed-to-native) System.Diagnostics.FileVersionInfo.GetVersionInfo_internal (System.Diagnostics.FileVersionInfo,string) <0x00012>
    at System.Diagnostics.FileVersionInfo.GetVersionInfo (string) [0x00019] in <5afdc70d980c4033b3878286f754d091>:0
    at Paket.Bootstrapper.PaketRunner.SetBootstrapperArgument (string,System.Collections.Generic.IEnumerable`1<string>) [0x00001] in <446260d283da49b4a4e02acdda74fe5a>:0
    at Paket.Bootstrapper.PaketRunner.Run (string,System.Collections.Generic.IEnumerable`1<string>) [0x00001] in <446260d283da49b4a4e02acdda74fe5a>:0
    at Paket.Bootstrapper.Program.OnSuccessfulDownload (Paket.Bootstrapper.BootstrapperOptions) [0x00045] in <446260d283da49b4a4e02acdda74fe5a>:0
    at Paket.Bootstrapper.Program/<>c__DisplayClass1_0.<Main>b__0 () [0x00000] in <446260d283da49b4a4e02acdda74fe5a>:0
    at Paket.Bootstrapper.Program.StartPaketBootstrapping (Paket.Bootstrapper.DownloadStrategies.IDownloadStrategy,Paket.Bootstrapper.DownloadArguments,Paket.Bootstrapper.HelperProxies.IFileSystemProxy,System.Action) [0x00358] in <446260d283da49b4a4e02acdda74fe5a>:0
    at Paket.Bootstrapper.Program.Main (string[]) [0x00133] in <446260d283da49b4a4e02acdda74fe5a>:0
    at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) [0x0004e] in <446260d283da49b4a4e02acdda74fe5a>:0

  Native stacktrace:

Debug info from gdb:
  (lldb) command source -s 0 '/tmp/mono-gdb-commands.hNBOfl'
  Executing commands in '/tmp/mono-gdb-commands.hNBOfl'.
  (lldb) process attach --pid 67218
EXEC : warning : (i386) /Library/Frameworks/Mono.framework/Versions/5.0.1/lib/mono/4.5/mscorlib.dll.dylib empty dSYM file detected, dSYM was created with an executable with no debug info. [/Users/joe/Code/Projects/FAKE/src/app/Fake.DotNet.Cli/Fake.DotNet.Cli.fsproj]
  Process 67218 stopped
  * thread #1, name = 'tid_403', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
      frame #0: 0xa17dec4a libsystem_kernel.dylib`__wait4 + 10
  libsystem_kernel.dylib`__wait4:
  ->  0xa17dec4a <+10>: jae    0xa17dec5a                ; <+26>
      0xa17dec4c <+12>: calll  0xa17dec51                ; <+17>
      0xa17dec51 <+17>: popl   %edx
      0xa17dec52 <+18>: movl   0x58de3d7(%edx), %edx

  Executable module set to "/Library/Frameworks/Mono.framework/Commands/mono".
  Architecture set to: i386-apple-macosx.
  (lldb) thread list
  Process 67218 stopped
  * thread #1: tid = 0x23f26cb, 0xa17dec4a libsystem_kernel.dylib`__wait4 + 10, name = 'tid_403', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    thread #2: tid = 0x23f26d1, 0xa17de30e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'SGen worker'
    thread #3: tid = 0x23f26d4, 0xa17d695a libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'Finalizer'
    thread #4: tid = 0x23f26d5, 0xa17decba libsystem_kernel.dylib`__workq_kernreturn + 10
    thread #5: tid = 0x23f26d6, 0xa17decba libsystem_kernel.dylib`__workq_kernreturn + 10
    thread #6: tid = 0x23f26d7, 0xa17decba libsystem_kernel.dylib`__workq_kernreturn + 10
  (lldb) thread backtrace all
  * thread #1, name = 'tid_403', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    * frame #0: 0xa17dec4a libsystem_kernel.dylib`__wait4 + 10
      frame #1: 0xa175d2ef libsystem_c.dylib`waitpid$UNIX2003 + 32
      frame #2: 0x0018ab27 mono`mono_handle_native_crash(signal=<unavailable>, ctx=0x0067ffe0, info=0x0067ffa0) at mini-exceptions.c:2567 [opt]
      frame #3: 0x001e7919 mono`mono_arch_handle_altstack_exception(sigctx=<unavailable>, siginfo=<unavailable>, fault_addr=<unavailable>, stack_ovf=0) at exceptions-x86.c:1115 [opt]
      frame #4: 0x000ccfc2 mono`mono_sigsegv_signal_handler(_dummy=<unavailable>, _info=<unavailable>, context=<unavailable>) at mini-runtime.c:2860 [opt]
      frame #5: 0xa18b8e5b libsystem_platform.dylib`_sigtramp + 43
      frame #6: 0x001f1a5f mono`find_pe_file_resources [inlined] find_pe_file_resources32(res_id=16, lang_id=0) at w32process-unix.c:2786 [opt]
      frame #7: 0x001f196f mono`find_pe_file_resources(res_id=16, lang_id=0, file_map=<unavailable>, map_size=<unavailable>, size=<unavailable>) at w32process-unix.c:2881 [opt]
      frame #8: 0x001f164a mono`mono_w32process_get_fileversion_info_size(filename=0x0087ce94, handle=0xbff429ec) at w32process-unix.c:3422 [opt]
      frame #9: 0x0029b89b mono`mono_w32process_get_fileversion(filever=<unavailable>, filename=<unavailable>, error=0xbff42a28) at w32process.c:330 [opt]
      frame #10: 0x0029b7e9 mono`ves_icall_System_Diagnostics_FileVersionInfo_GetVersionInfo_internal(this_obj=<unavailable>, filename=<unavailable>) at w32process.c:422 [opt]
      frame #11: 0x007ecd00
      frame #12: 0x007ecbb4
      frame #13: 0x007eca6c
      frame #14: 0x007ec960
      frame #15: 0x027b6e28
      frame #16: 0x027b6d64
      frame #17: 0x027b2a83
      frame #18: 0x005e854c
      frame #19: 0x005e86d7
      frame #20: 0x000d0dc5 mono`mono_jit_runtime_invoke(method=<unavailable>, obj=<unavailable>, params=<unavailable>, error=<unavailable>) at mini-runtime.c:2533 [opt]
      frame #21: 0x002e8410 mono`do_runtime_invoke(method=<unavailable>, obj=<unavailable>, params=<unavailable>, exc=<unavailable>, error=<unavailable>) at object.c:2860 [opt]
      frame #22: 0x002ebcc7 mono`do_exec_main_checked [inlined] mono_runtime_invoke_checked(method=0x7d3b40b8, params=0x008002a0, error=0x002e83c2) at object.c:3018 [opt]
      frame #23: 0x002ebc6e mono`do_exec_main_checked(method=<unavailable>, args=<unavailable>, error=0x002e83c2) at object.c:4680 [opt]
      frame #24: 0x002ea999 mono`mono_runtime_run_main_checked [inlined] mono_runtime_exec_main_checked(method=0x7d3b40b8, args=0x008002a0, error=0xbff42e98) at object.c:4781 [opt]
      frame #25: 0x002ea979 mono`mono_runtime_run_main_checked(method=0x7d3b40b8, argc=6, argv=0xbff432b4, error=0xbff42e98) at object.c:4230 [opt]
      frame #26: 0x0014c0e7 mono`mono_jit_exec(domain=0x7b75cbf0, assembly=0x7c960fa0, argc=<unavailable>, argv=<unavailable>) at driver.g.c:1037 [opt]
      frame #27: 0x0014ea52 mono`mono_main [inlined] main_thread_handler at driver.g.c:1106 [opt]
      frame #28: 0x0014ea14 mono`mono_main(argc=<unavailable>, argv=<unavailable>) at driver.g.c:2215 [opt]
      frame #29: 0x000c016b mono`main [inlined] mono_main_with_options(argc=8, argc=8, argc=8, argv=0xbff432ac, argv=0xbff432ac, argv=0xbff432ac) at main.c:45 [opt]
      frame #30: 0x000c014a mono`main(argc=8, argv=0xbff432ac) at main.c:338 [opt]
      frame #31: 0x000bf985 mono`start + 53

    thread #2, name = 'SGen worker'
      frame #0: 0xa17de30e libsystem_kernel.dylib`__psynch_cvwait + 10
      frame #1: 0xa18c2eb0 libsystem_pthread.dylib`_pthread_cond_wait + 647
      frame #2: 0xa18c4844 libsystem_pthread.dylib`pthread_cond_wait$UNIX2003 + 51
      frame #3: 0x00360fb1 mono`thread_func [inlined] mono_os_cond_wait(mutex=0x00456c8c) at mono-os-mutex.h:146 [opt]
      frame #4: 0x00360f9f mono`thread_func(thread_data=0x00000000) at sgen-thread-pool.c:129 [opt]
      frame #5: 0xa18c2047 libsystem_pthread.dylib`_pthread_body + 184
      frame #6: 0xa18c1f8f libsystem_pthread.dylib`_pthread_start + 243
      frame #7: 0xa18c184a libsystem_pthread.dylib`thread_start + 34

    thread #3, name = 'Finalizer'
      frame #0: 0xa17d695a libsystem_kernel.dylib`semaphore_wait_trap + 10
      frame #1: 0x002e0976 mono`finalizer_thread [inlined] mono_os_sem_wait(flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:90 [opt]
      frame #2: 0x002e0968 mono`finalizer_thread [inlined] mono_coop_sem_wait(flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:40 [opt]
      frame #3: 0x002e095e mono`finalizer_thread(unused=0x00000000) at gc.c:907 [opt]
      frame #4: 0x002b0ce4 mono`start_wrapper [inlined] start_wrapper_internal at threads.c:837 [opt]
      frame #5: 0x002b0ba2 mono`start_wrapper(data=<unavailable>) at threads.c:889 [opt]
      frame #6: 0x00377bf0 mono`inner_start_thread(data=<unavailable>) at mono-threads.c:1170 [opt]
      frame #7: 0xa18c2047 libsystem_pthread.dylib`_pthread_body + 184
      frame #8: 0xa18c1f8f libsystem_pthread.dylib`_pthread_start + 243
      frame #9: 0xa18c184a libsystem_pthread.dylib`thread_start + 34

    thread #4
      frame #0: 0xa17decba libsystem_kernel.dylib`__workq_kernreturn + 10
      frame #1: 0xa18c1d06 libsystem_pthread.dylib`_pthread_wqthread + 1210
      frame #2: 0xa18c1826 libsystem_pthread.dylib`start_wqthread + 34

    thread #5
      frame #0: 0xa17decba libsystem_kernel.dylib`__workq_kernreturn + 10
      frame #1: 0xa18c1b95 libsystem_pthread.dylib`_pthread_wqthread + 841
      frame #2: 0xa18c1826 libsystem_pthread.dylib`start_wqthread + 34

    thread #6
      frame #0: 0xa17decba libsystem_kernel.dylib`__workq_kernreturn + 10
      frame #1: 0xa18c1d06 libsystem_pthread.dylib`_pthread_wqthread + 1210
      frame #2: 0xa18c1826 libsystem_pthread.dylib`start_wqthread + 34
  (lldb) detach

  =================================================================
  Got a SIGSEGV while executing native code. This usually indicates
  a fatal error in the mono runtime or one of the native libraries
  used by your application.
  =================================================================

  Process 67218 detached
  (lldb) quit
/var/folders/23/r2b048nj4vn9xvtg7qjsy6980000gn/T/tmpf6917b95623148e7b68dcd50f004b6dd.exec.cmd: line 2: 67218 Abort trap: 6 /Library/Frameworks/Mono.framework/Commands/mono --runtime=v4.0.30319 "/Users/joe/Code/Projects/FAKE/.paket/paket.exe" fix-nuspec file "obj/Fake.DotNet.Cli.5.0.0-alpha010.nuspec" references-file "/Users/joe/Code/Projects/FAKE/src/app/Fake.DotNet.Cli/obj/Fake.DotNet.Cli.fsproj.references"

Looks like just another mono crash

matthid commented 7 years ago 
-> Could not load resources from 'https://api.nuget.org/v3/index.json'
-> WebException: Error: SecureChannelFailure (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
     at /tmp/buildd/mono-5.0.1.1/external/boringssl/ssl/handshake_client.c:1132)
-> MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
     at /tmp/buildd/mono-5.0.1.1/external/boringssl/ssl/handshake_client.c:1132
baronfel commented 7 years ago

May need to run a cert-sync tool on mono 5 to import Mozilla certs into boring 'new cert chain: http://tirania.org/blog/archive/2016/Sep-30.html

matthid commented 7 years ago

@baronfel this is not a error happing every time. We see this from time to time in travis. I'm pretty sure it's a bug.

matthid commented 7 years ago 
The object with ID 2 implements the IObjectReference interface for which all dependencies cannot be resolved. The likely cause is two instances of IObjectReference that have a mutual dependency on each other.

Server stack trace: 
  at System.Runtime.Serialization.ObjectManager.GetCompletionInfo (System.Runtime.Serialization.FixupHolder fixup, System.Runtime.Serialization.ObjectHolder& holder, System.Object& member, System.Boolean bThrowIfMissing) [0x000ae] in <a07d6bf484a54da2861691df910339b1>:0 
  at System.Runtime.Serialization.ObjectManager.CompleteObject (System.Runtime.Serialization.ObjectHolder holder, System.Boolean bObjectFullyComplete) [0x00134] in <a07d6bf484a54da2861691df910339b1>:0 
  at System.Runtime.Serialization.ObjectManager.DoFixups () [0x000f1] in <a07d6bf484a54da2861691df910339b1>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize (System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Runtime.Serialization.Formatters.Binary.__BinaryParser serParser, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) [0x00077] in <a07d6bf484a54da2861691df910339b1>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) [0x000a2] in <a07d6bf484a54da2861691df910339b1>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) [0x00000] in <a07d6bf484a54da2861691df910339b1>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck) [0x00000] in <a07d6bf484a54da2861691df910339b1>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler) [0x00000] in <a07d6bf484a54da2861691df910339b1>:0 
  at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.ProcessMessage (System.Runtime.Remoting.Channels.IServerChannelSinkStack sinkStack, System.Runtime.Remoting.Messaging.IMessage requestMsg, System.Runtime.Remoting.Channels.ITransportHeaders requestHeaders, System.IO.Stream requestStream, System.Runtime.Remoting.Messaging.IMessage& responseMsg, System.Runtime.Remoting.Channels.ITransportHeaders& responseHeaders, System.IO.Stream& responseStream) [0x0009d] in <f586a7d6b774492c991ee7a6eb223d9a>:0 

Exception rethrown at [0]: 
  at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_remoting_wrapper (intptr,intptr)
  at (wrapper remoting-invoke) NUnit.Engine.Agents.RemoteTestAgent:Run (NUnit.Engine.ITestEventListener,NUnit.Engine.TestFilter)
  at NUnit.Engine.Runners.ProcessRunner.RunTests (NUnit.Engine.ITestEventListener listener, NUnit.Engine.TestFilter filter) [0x00025] in <7b0953727751470ab10f0e0b547b85ae>:0
enricosada commented 6 years ago

@matthid @baronfel dunno the initial bug report, but the latest two