Closed ayende closed 4 weeks ago
Assume this issue doesnt repro with W^X disabled? Is there a dump which can be shared to debug further? Appears the original mapping return code issue is fixed in 8 and might be ported to 7 too. Are you testing on the latest servicing release?
I'm afraid that we don't have a dump, only those logs. This is a production instance, so we jump bumped the map_count
to alleviate the issue.
@gregolsky - can you answer regarding the W^X
and the service release?
Runtime version is 7.0.8
. AFAIK WriteXorExecute is enabled there by default? I'm afraid we cannot do any experiments on how it works when it's disabled on the system in question.
We can try to repro on another one though artificially reducing the number of available maps.
Is it possible that you have growing number of dynamically created assemblies like in the https://github.com/dotnet/runtime/issues/80580#issuecomment-1410942438?
Not likely, we aren't really generating many new assemblies on the fly, and not at all in the scenario we tested
@ayende re-reading the issue description, I am not sure I understand this:
The stack trace is really strange, we are attempting to allocate memory and then fail. That is a handled exception, but we are dying with segmentation fault.
The failure to map memory as RW in the executable allocator is fatal fail fast. It is not an exception. The stack trace is a stack trace of the managed code at the time the fail fast happened. At that point, we don't have any other option than to fail fast. There are more than a hundred of places all over the source base when we need to modify or write executable code and there is no way to recover from that at majority of the places.
The fact that you get a sigsegv after the fail fast message is printed is strange though. I wonder if that could be related to the wrong checks for mmap return value that were fixed in .NET 8 (#77952, #78069), but not ported to .NET 7. It is actually quite possible it is the case.
Would it be possible to port these to 7?
wt., 15 sie 2023, 22:47 użytkownik Jan Vorlicek @.***> napisał:
@ayende https://github.com/ayende re-reading the issue description, I am not sure I understand this:
The stack trace is really strange, we are attempting to allocate memory and then fail. That is a handled exception, but we are dying with segmentation fault.
The failure to map memory as RW in the executable allocator is fatal fail fast. It is not an exception. The stack trace is a stack trace of the managed code at the time the fail fast happened. At that point, we don't have any other option than to fail fast. There are more than a hundred of places all over the source base when we need to modify or write executable code and there is no way to recover from that at majority of the places.
The fact that you get a sigsegv after the fail fast message is printed is strange though. I wonder if that could be related to the wrong checks for mmap return value that were fixed in .NET 8 (#77952 https://github.com/dotnet/runtime/pull/77952, #78069 https://github.com/dotnet/runtime/pull/78069), but not ported to .NET
- It is actually quite possible it is the case.
— Reply to this email directly, view it on GitHub https://github.com/dotnet/runtime/issues/90230#issuecomment-1679590208, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALYNLKOEWLUSY6RWU52S5DXVPN6DANCNFSM6AAAAAA3KBPHAE . You are receiving this because you were mentioned.Message ID: @.***>
@gregolsky, have you tried the scenario on 8 to ensure these fixes indeed work in your case?
What would be the expected scenario here? Ideally, I would rather get a proper error message / diagnostics so that we can detect that in production.
The usual metrics (memory usage, etc) are not a problem in this case.
@janvorli, believe there was a change to update the error message for these conditions?
Yes, the message was updated in #102458
Ok closing this issue based on that.
Description
We are tracking what looks like a memory / fragmentation leak, see here: https://github.com/dotnet/runtime/issues/89776
As a result of that, we run into the limit of
vm.map_count
, see:This was set to 65535, and we got several crashes from the finalizer.
I believe that this is related to this: https://github.com/dotnet/runtime/issues/80580
Reproduction Steps
Run for a long while under load, and the number of mapping will increase until you reach the limit
Expected behavior
Should not crash
Actual behavior
It crashes
Regression?
Did not see that in .NET 6.0
Known Workarounds
Increase
vm.map_count
to a very high numberConfiguration
Linux, x64, Ubuntu, .NET 7.0
Other information
The stack trace is really strange, we are attempting to allocate memory and then fail. That is a handled exception, but we are dying with segmentation fault.
Given that this is thrown from
executableallocator
, I wonder if this is possibly trying to JIT the method or maybe tier it up, and then failing.Note that it is possible for
unmap
to fail in Linux (if it will increase the mapping amount. This should probably be handled without killing the process.