The issue is two fold, the usual case is that a syscall gets ENOMEM as a result of a call to a function declared in kmalloc.h, and the rarer case is that a kernel subsystem runs out of memory and can't continue proper operation. In the former case it would be a lot easier to just quit instead of trying to rollback and keep half-operational. These code paths are practically never tested properly.
The first case is quite often a result of the user passing an absurdly huge buffer size as an argument, in which case we should actually do input validation and not even try to allocate a kernel buffer for the data. Usually instead of allocating a kernel buffer uio buffer wrapping should be preferred, which reads and writes directly from and to the user space. The only problem is that it makes input validation impossible in case it's necessary, because another thready could alter the data while the other thread is executing the syscall code.
The issue is two fold, the usual case is that a syscall gets
ENOMEM
as a result of a call to a function declared inkmalloc.h
, and the rarer case is that a kernel subsystem runs out of memory and can't continue proper operation. In the former case it would be a lot easier to just quit instead of trying to rollback and keep half-operational. These code paths are practically never tested properly.The first case is quite often a result of the user passing an absurdly huge buffer size as an argument, in which case we should actually do input validation and not even try to allocate a kernel buffer for the data. Usually instead of allocating a kernel buffer
uio
buffer wrapping should be preferred, which reads and writes directly from and to the user space. The only problem is that it makes input validation impossible in case it's necessary, because another thready could alter the data while the other thread is executing the syscall code.