Better indirect argument file handling

[EvanKirshenbaum]

I added the ability to say @file on the command-line to read in a file of arguments (#252), and it works, but some of its behavior is surprising and should be fixed.

In particular, every line of the file is treated, unmodified, as a single argument. This means that

1. Blank lines are a problem.
2. Leading and trailing space is a problem.
3. There's no way to specify comments.
4. There's no way of having one line read as multiple arguments (e.g., --clock-speed 200ms).
5. There's no way to do any sort of conditionality (ala -D symbol and ifdef)

According to the docs, you can subclass ArgumentParser and provide your own convert_arg_line_to_args(), which takes each file line and returns a list of argument strings. This would allow

1. comment stripping
2. leading and trailing whitespace stripping,
3. returning an empty list when the line is empty (after stripping),
4. allowing trailing backslashes (or some other means) for handling multi-line values,
5. splitting on whitespace, at least after the argument, and
6. possibly keeping track of live and dead blocks

The live and dead blocks may be tricky, unless there's also some hook to know when we enter and leave a file (as, presumably, all blocks should close at file exit.) It might be possible to do it within the file if I use

1. some sort of guard prefix to each line or
2. indentation and if/else constructs.

Migrated from internal repository. Originally created by @EvanKirshenbaum on Feb 24, 2023 at 3:37 PM PST.

[EvanKirshenbaum]

No, looking at the code, it looks as though it's all encapsulated ni ArgumentParser._read_args_from_files():

    def _read_args_from_files(self, arg_strings):
        # expand arguments referencing files
        new_arg_strings = []
        for arg_string in arg_strings:

            # for regular arguments, just add them back into the list
            if not arg_string or arg_string[0] not in self.fromfile_prefix_chars:
                new_arg_strings.append(arg_string)

            # replace arguments referencing files with the file content
            else:
                try:
                    with open(arg_string[1:]) as args_file:
                        arg_strings = []
                        for arg_line in args_file.read().splitlines():
                            for arg in self.convert_arg_line_to_args(arg_line):
                                arg_strings.append(arg)
                        arg_strings = self._read_args_from_files(arg_strings)
                        new_arg_strings.extend(arg_strings)
                except OSError:
                    err = _sys.exc_info()[1]
                    self.error(str(err))

        # return the modified argument list
        return new_arg_strings

which does the open directly and doesn't call any hook. Sigh. I could override it, but that would be a support nightmare. I think that for now I'll have to assume that I don't have access to file entry and exit.

Migrated from internal repository. Originally created by @EvanKirshenbaum on Feb 24, 2023 at 3:41 PM PST.