Hi,
I maintain ExtUtils::MakeMaker that makes Makefiles to install Perl modules.
Since its an automated thing, and a bit long in the tooth, it sometimes creates
rather long lines. For example, http://gist.github.com/575964
I've been getting complaints about dmake rejecting long lines, even if they're
broken up by escaped newlines. I've just introduced a work around by setting
MAXLINELENGTH.
The whole concept strikes me as a memory optimization hack leaked out to the
user. It looks like dmake is allocating a fixed hunk of memory to concatenate
together lines of input, the size of that allocation is governed by
MAXLINELENGTH. This might have made sense back when 8k meant something.
Bumping the default up to 32k helped, but its still something users should not
have to deal with. I'm not aware of any other mainstream make implementation
that works this way (but boy are there OS' with some stingy exec lengths).
A more dynamic approach would seem to be in order. Allocate a string of size X
and realloc it to size * Y as needed. Perl uses a simple doubling heuristic, it
depends on the CPU cost of realloc vs how much memory you want to potentially waste.
Looking at the code, Buffer is used for a small number of things. Get_line(),
Parse_rule_def() and storing command line options in main() is all I see.
Parse_rule_def() just grabs at the global. The buffer is passed into Get_line()
which has no opportunity to inform the user that its been reallocated. I don't
know if Get_line()'s PUBLIC flag means its use is outside the control of dmake,
but its interface would have to change for this to work. Otherwise only Parse()
uses it.
If Get_line() is really public, as in published and used by code outside the
control of dmake, a new function could be written that does the reallocation as
necessary. Get_line() could be a wrapper which sets a flag that tells the new
function to throw an error rather than reallocate.
I realize this is a bit of rearchitecting, but I thought it should be brought
up. Thank you.
mohawk2
commented
10 years ago
(from schwern)
Hi, I maintain ExtUtils::MakeMaker that makes Makefiles to install Perl modules. Since its an automated thing, and a bit long in the tooth, it sometimes creates rather long lines. For example, http://gist.github.com/575964
I've been getting complaints about dmake rejecting long lines, even if they're broken up by escaped newlines. I've just introduced a work around by setting MAXLINELENGTH.
The whole concept strikes me as a memory optimization hack leaked out to the user. It looks like dmake is allocating a fixed hunk of memory to concatenate together lines of input, the size of that allocation is governed by MAXLINELENGTH. This might have made sense back when 8k meant something. Bumping the default up to 32k helped, but its still something users should not have to deal with. I'm not aware of any other mainstream make implementation that works this way (but boy are there OS' with some stingy exec lengths).
A more dynamic approach would seem to be in order. Allocate a string of size X and realloc it to size * Y as needed. Perl uses a simple doubling heuristic, it depends on the CPU cost of realloc vs how much memory you want to potentially waste.
Looking at the code, Buffer is used for a small number of things. Get_line(), Parse_rule_def() and storing command line options in main() is all I see. Parse_rule_def() just grabs at the global. The buffer is passed into Get_line() which has no opportunity to inform the user that its been reallocated. I don't know if Get_line()'s PUBLIC flag means its use is outside the control of dmake, but its interface would have to change for this to work. Otherwise only Parse() uses it.
If Get_line() is really public, as in published and used by code outside the control of dmake, a new function could be written that does the reallocation as necessary. Get_line() could be a wrapper which sets a flag that tells the new function to throw an error rather than reallocate.
I realize this is a bit of rearchitecting, but I thought it should be brought up. Thank you.