microsoftarchive / redis

Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes
http://redis.io
Other
20.81k stars 5.37k forks source link

Redis 3.0.501 Fork Operation Failed #428

Closed SantolNM closed 8 years ago

SantolNM commented 8 years ago

Hi,

I am seeing the exact same behavior as Harachie. I was running version 3.0.500 for a few days but kept on receiving failures with the fork operation after a couple of hours of running successfully. I upgraded to 3.0.501 and saw the exact same thing. I have downgraded to 2.8.24 and everything has been working correctly for 9 hours. On 2.8.24 I have set the maxmemory to 100mb and the maxheap to 1024mb.

There definitely appears to be some type of memory corruption on 3.0.500 and 3.0.501. When running with these two versions I did vary the paging file configurations on my windows server 2012r2 running in Azure. I tried running with a paging file set at 3.5 times the physical memory of 3.5Gb. The service ran good for a couple of hours than began logging errors (Fork Operation Failed). I tried letting windows manage the paging file size and experienced the same behavior.

The error logged in the Windows Application Log:

Faulting application name: redis-server.exe, version: 0.0.0.0, time stamp: 0x5698bfe1 Faulting module name: redis-server.exe, version: 0.0.0.0, time stamp: 0x5698bfe1 Exception code: 0xc0000005 Fault offset: 0x00000000000a4293 Faulting process id: 0x6e8 Faulting application start time: 0x01d18179b3fb1d7c Faulting application path: C:\Program Files\Redis\redis-server.exe Faulting module path: C:\Program Files\Redis\redis-server.exe Report Id: f1ef2f09-ed6c-11e5-80d2-000d3a104340 Faulting package full name: Faulting package-relative application ID:

After reading Harachie comments and the support responses, it appears I am experiencing the same issues. Please let me know if you need any additional information. At some point, I will need to upgrade to 3.0.501 to take advantage of the clustering features.

Any help regarding this matter is greatly appreciated.

qiangbro commented 8 years ago

I also got a bug report in version 3.0.501:

=== REDIS BUG REPORT START: Cut & paste starting from here ===
[2284] 21 Mar 17:29:49.338 # Redis version: 3.0.501
[2284] 21 Mar 17:29:49.338 # --- EXCEPTION_ACCESS_VIOLATION
[2284] 21 Mar 17:29:49.339 # --- STACK TRACE
redis-server.exe!((null):0)(0x0012E6F0, 0x0012FF90, 0x00000001, 0x40124730)
redis-server.exe!((null):0)(0x00000001, 0x00000000, 0x00000001, 0x00000000)
kernel32.dll!UnhandledExceptionFilter((null):0)(0x0012E6F0, 0x00000006, 0x00000000, 0x00000001)
ntdll.dll!EtwEventSetInformation((null):0)(0x0012F330, 0x00000030, 0x4012AC28, 0x00000000)
ntdll.dll!_C_specific_handler((null):0)(0x00130000, 0x0012FF90, 0x0012FF90, 0x7770EAAC)
ntdll.dll!RtlDecodePointer((null):0)(0x00130000, 0x7721DDAC, 0x00012A98, 0x00249F60)
ntdll.dll!RtlUnwindEx((null):0)(0x0012F330, 0x0012EE40, 0x00000000, 0x00000000)
ntdll.dll!KiUserExceptionDispatcher((null):0)(0x76E27B92, 0x4008EAC9, 0x00000000, 0x00000000)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x0012F6E0, 0x00001093, 0x0000108D, 0x03807070)

redis-server.exe!KiUserExceptionDispatcher((null):0)(0x0012F6E0, 0x00000001, 0x2986E5B0, 0x00000001)

redis-server.exe!KiUserExceptionDispatcher((null):0)(0x40150210, 0x021C0000, 0x00000005, 0x025C2754)

redis-server.exe!KiUserExceptionDispatcher((null):0)(0x021C0000, 0x56EE8223, 0x021C0000, 0x00000005)

redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00000005, 0x00000000, 0x00392DF0, 0x00000005)

redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00000006, 0x00000000, 0x00000000, 0x00392870)

redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)

redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)

kernel32.dll!BaseThreadInitThunk((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[2284] 21 Mar 17:29:49.348 # --- INFO OUTPUT
[5908] 21 Mar 17:29:49.456 # fork operation failed
[5908] 21 Mar 17:29:49.653 # Background saving terminated by signal 1
[5908] 21 Mar 17:29:50.060 * 10000 changes in 60 seconds. Saving...
[5908] 21 Mar 17:29:50.068 * Background saving started by pid 14352
[5908] 21 Mar 17:30:12.269 # fork operation complete
[5908] 21 Mar 17:30:12.436 * Background saving terminated with success

and my Java program throw exception

Exception in thread "pool-2-thread-15359" redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
Caused by: java.util.NoSuchElementException: Unable to validate object
enricogior commented 8 years ago

Hi @SantolNM since this issue is already tacked in https://github.com/MSOpenTech/redis/issues/407 it's better to keep only one issue open. Thank you.

SantolNM commented 8 years ago

Hi Enrico,

Thanks for the quick reply.

Nick Santoleri

From: Enrico Giordani notifications@github.com Reply-To: MSOpenTech/redis <reply+0111bb6ca45f68e15cbcb645a432ecd72ead19fc89443b3192cf000000011307892d9 2a169ce0877b360@reply.github.com> Date: Monday, March 21, 2016 at 6:04 AM To: MSOpenTech/redis redis@noreply.github.com Cc: Nicholas Santoleri santolnm@me.com Subject: Re: [redis] Redis 3.0.501 Fork Operation Failed (#428)

Closed #428 https://github.com/MSOpenTech/redis/issues/428 .

‹ You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/MSOpenTech/redis/issues/428#event-596883174