Response file

[awvwgk]

I'm not entirely sure where this exactly originates and where to find a good documentation on this, but some compilers and build tools support a “response file.” Usually the syntax is

prog @file1.rsp @file2.rsp

where file1.rsp, file2.rsp, ... contain the command line arguments separated by newlines and can be combined with normal command lines arguments. Some implementations allow further references to other response files in the response file. I did find some documentation for MSBuild.exe on rsp-files and some notes at the ninja build manpage.

This might be an interesting feature for this library since it would allow to store (maybe even load and/or dump) command lines as a file, which makes it useful for testing or storing frequently needed (complicated) command lines.

[urbanjost]

I actually have done something that might be similiar using a CLI cracker called M_kracken up to the point where the calling program becomes a shell. The one I think is most similar to what you mention here allows you to make your own subcommand name, basically. Since we are adding new subcommands to fpm their could easily be a conflict so using something like the at prefix might be a good idea. I was thinking the way things were already focused on TOML files it might be good to do simple aliases there, but it would be easy using M_calculator and M_logic (old F77 codes with a bit of updating, but as-is a bit daed but ffunction) so you could do things like

fpm @id

and it couldfpm run --compiler ifort --runner 'gdb'. I have a trimmed-down M_calculator (the real one calls a lot of system-dependent stuff like almost all the POSIX routines and such) so the input files can branch and use environment variables and stuff as a "stage two". Would only take a few hours to put a prototype up. Trying to find more info. Some of my favorite aliases are becoming obsolete soon though. My favorite scripts and aliases are "ffpm" which searches for an fpm.toml file in the parent branches an does a cd to the directory, fpm-install which does a mkdir of the directory name, and does

fpm run --release --compiler gfortran --runner 'install -vbp -m 0711 -t ~/.local/bin'

which should shortly be superseded by fpm install and fpm-all which repeats the command for all the compilers I recently got and defaults to "fpm test --compiler NAME` for each compiler. Doing multiple calls to the program might be harder without spawning a subprocess.

I will do some web searching and see if I am interpreting this correctly.

[urbanjost]

After looking around a bit response files seem to mean everthing from system-independent one-per-file way of doing an alias using files in the current directory to journal or replay files. The Intel compiler supports the simplest one of those.This is unstandardized I am a bit hesitant to go down this path, but it would be a lot less code than a parsing shell-like custom language. Unless switching between various platforms a lot not sure aliases and simple shell scripts might not be a better solution.

That being said, combining some of the instances I found and picking and choosing between what I did and did not like I would say the simplest to implement in standard Fortran would be something like ...

PRELIMINARY SPEC

Spec would be if set_args(3f) has the argument response_file=.true. then the following files will be searched until the first match is found:

1. A pathname specified by the environment variable $CLI_RESPONSE
2. A file called $HOME/.cli_response.rsp
3. A file called $HOME/.local/dot/cli_response.rsp

If the environment variable $OS (first) or $OS_TYPE is set the search will be for a line of the form (no leading spaces allowed):

   @$OS_TYPE@basename($arg0)@alias_name

where basename($arg0) means the basename of the name of the program as returned by GET_COMMAND_ARGUMENT(0,...).

If no match is found then look for a line of the form

   @basename($arg0)@alias_name

Lines will then be processed until another line starting with "at" is found or end-of-file is encountered. Column one is special and would have the following meanings

 a leading blank (or >) means echo message to screen

#comment

!system command
#

System commands would be executed as a simple call to system (so a cd(1) would not effect subsequent lines, for example)

- default options

This would be the typical use for a simple alias. For "-" syntax is same as command definitions using SET_ARGS() so double-quote strings; F|T for logical; ... Note easiest to implement is that options are cumulative, might want to change that.

@STOP message
@EXIT message

A special line meaning to STOP or EXIT calling program

EXAMPLES

Resulting in Something like this example file

   @fpm@if
   > RUNNING TESTS USING RELEASE VERSION AND ifort
   - test --compiler ifort
   @fpm@gf
   > RUNNING TESTS USING RELEASE VERSION AND gfortran
   - test --compiler gfortran
   @fpm@nv
   > RUNNING TESTS USING RELEASE VERSION AND nvfortran
   - test --compiler nvfortran
   @fpm@nag
   > RUNNING TESTS USING RELEASE VERSION AND nagfor
   - test --compiler nagfor
   #
   # OS-specific
   @Linux@fpm@install
   #@(#) install executables in directory (assuming install(1) exists)
   !mkdir -p ~/.local/bin
   - run --release T --compiler gfortran --runner "install -vbp -m 0711 -t ~/.local/bin"
   @fpm@install
   @STOP INSTALL NOT SUPPORTED ON THIS PLATFORM OR $OS_TYPE NOT SET
   #
   @fpm@testall
   #
   !fpm test --compiler nvfortran
   !fpm test --compiler ifort
   !fpm test --compiler gfortran
   !fpm test --compiler nagfor
   @EXIT tests complete. Any additional parameters were ignored
   #

Would be used like

   fpm @install
   fpm @nag --release
   fpm @testall

ALTERNATIVE

us M_logic and M_calculator and make look a lot like a bashshell input file allowing for variables and so on

M_logic would provide conditional based on environment if/else/elseif/endif. Could use existing functions and syntax would look Unix-like. ~

[urbanjost]

It does not quite amend itself to a program using subcommands yet; but I added an alpha version of response file support to M_CLI2. Still thinking about it. I have a lot of programs that use M_CLI2 that I am still trying this with. I am trying out a few ideas on how to handle subcommands reliably. I find different documentation for MSWindows indicating in some environments commands can be no longer than 128 characters; but I find other documentation indicating the limit is 32K, so not sure how important the ability of a response file to get around MSWindows line length limits still is, but for fpm(1) this would also be useful for aliasing long commands.

[zoziha]

Hello, @urbanjost ,

I try to query the usage of response file on Windows-msys2-gfortran platform in this repo, but I can't find the correct usage, is the response file not applicable for Windows-msys2-gfortran? It's ok for me to use a response file under Linux.

fpm.rsp:

@run
option run --profile debug

fpm @run

Thanks, looking forward to your reply :)

[urbanjost]

A response file should work on any platform. I currently do not have an MSWindows platform I can test on but it should work anywhere fpm runs. If your response file is not in the top of your project file you have to use environment variables to indicate where to find the files.

What message do you get when you run "fpm @run" ? I will try to get set up with your environment but that environment was one of the original ones being used when response files were being added and tested so I know it worked in the recent path; and I can think of nothing that has changed since that would have broken it so hopefully this is something simple that just has to do with how the file is being searched for.

[zoziha]

Thank you for your reply, @urbanjost . I usually use powershell and I retested it. Under Windows, using powershell, cmd, and bash, it seems that cmd can successfully run fpm @run (Sorry I didn't test cmd before), but powershell and bash cannot run normally. The running results are as follows:

I can run it under Linux with fish shell:

---

Seems like the @ sign has syntactic meaning in powershell, and bash fails unexpectedly. Is there any way to make fpm @run more generic, like available in powershell and bash under Windows?

[urbanjost]

That was very informative. So it works in cmd, but not in bash. I suspect that one or both of the environment variables OSTYPE or OS is being set incorrectly to indicate you are on MSWindows; when in the WSL mode in bash it should be taking the Unix shell. So for the bash shell, probably doing an "unset OS" or "unset OSTYPE" is required for fpm in general, not just for response files. If the @ character is special in powershell maybe another character needs to be allowed as the flag character for a response file. The procedure in fpm that guesses what OS you are in needs a little more work. If you run

 fpm --version

It should show an "OS_type" which I am going to says WIndows instead of Linux; and the work-around to "unset OS" will then cause it to work and "fpm --version" will then show "Linux". I do not use powershell much so I am not sure there is a work-around; that might require making a version where something other than @ be used to indicate to use a response file. I wonder what "fpm --version" shows in powershell as well. What is going on in the bash shell is just a guess till I can get to a MSWIndows machine, but it is likely what is going on. I would have no idea what was going on in the powershell if you had not noted "@" is a special character to it. If there is an environment variable that is only set when in the powershell and/or in the WSL, that might be one way to solve this generically.

[urbanjost]

If you put a backslash in front of the at like

fpm \@run

does that work in powershell? If not, perhaps

fpm `@run

still reading powershell documentation :>

[urbanjost]

Since in bash OSTYPE is a process variable and not by default an environment variable, I am also guessing

export OSTYPE

will also fix the bash shell under WSL.

[zoziha]

WSL-bash should run like normal Linux. Sorry to misunderstand you, my bash here is from mingw64/msys2's bash, similar to git-bash, it is not from WSL.

Thanks for the reminder, when I run fpm \@run in powershell, the following happens:

>> fpm \@run    # In powershell
<ERROR> unknown subcommand [\@run]

 build [--compiler COMPILER_NAME] [--profile PROF] [--flag FFLAGS] [--list]
       [--tests] [--no-prune]
 help [NAME(s)]
 new NAME [[--lib|--src] [--app] [--test] [--example]]|
          [--full|--bare][--backfill]
 update [NAME(s)] [--fetch-only] [--clean] [--verbose]
 list [--list]
 run  [[--target] NAME(s) [--example] [--profile PROF] [--flag FFLAGS] [--all]
      [--runner "CMD"] [--compiler COMPILER_NAME] [--list] [-- ARGS]
 test [[--target] NAME(s)] [--profile PROF] [--flag FFLAGS] [--runner "CMD"]
      [--list] [--compiler COMPILER_NAME] [-- ARGS]
 install [--profile PROF] [--flag FFLAGS] [--no-rebuild] [--prefix PATH]
         [options]
 clean [--skip] [--all]

It seems that fpm `@run in powershell is more correct, and it is consistent with the display effect of fpm @run in bash under Windows:

>> fpm `@run  # In powershell
<ERROR> response name [@run] not found
STOP 1
>> uname -a
MSYS_NT-10.0-22621 windows 3.3.6-341.x86_64 2022-10-24 08:25 UTC x86_64 Msys

>> fpm @run   # In mingw64::bash
<ERROR> response name [@run] not found
STOP 1
>> fpm --version  # In powershell/cmd/bash
Version:     0.7.0, alpha
Program:     fpm(1)
Description: A Fortran package manager and build system
Home Page:   https://github.com/fortran-lang/fpm
License:     MIT
OS Type:     Windows

---

It is possible that M_CLI2 cannot judge the current system, because it uses the msys2 environment after all, not a pure native Windows OS, which leads to the inability to correctly parse fpm.rsp?

It seems that the idea of ​​using fpm `@run in powershell is right 💕, and we can find a way to make M_CLI2 recognize the msys2/mingw64 environment.

[urbanjost]

I am thinking allowing a second prefix character such as colon would make it easier in powershell, such as "fpm :run" as an alternative to "fpm `@run" would make usage easier in powershell; I need to run a few tests from powershell as I see why there might be problems with filenames when \ and / are involved but just a plain "fpm.rsp" file looks like it should be OK, so I am not spotting something. Unfortunately the debug mode in M_CLI2 has to be activated by a code change so I just need to build a test version and what is going on will probably be obvious (hopefully) but it might be a few days.

[urbanjost]

An embarrassingly simple explanation. Make a copy of your fpm.rsp file and name it fpm.exe.rsp. If you run fpm @name it first looks for a file called "name.rsp" but if that is not found it queries the name of the executable and gets the basename and adds ".rsp" So for a program called "fpm" it looks for a file called "fpm.rsp" and assumes that is a compound response file and looks in it for "@name". On MSWindows the files' real name is "fpm.exe", not "fpm" as it is on Linux and MacOS and BSD and Unix. So stripping off the suffix, and also adding : as an alternative for @ will make it far more usable in those environments. There may be other issues if the response files are in other directories searched for when the search path is declared and the wrong OS is being guessed because of the OS variable which I will look at next now that I have an MSWindows test platform; but I will get these changed made first and then test with the new version. I have a PR pending for a change required on 32-bit OpenBSD platforms pending. I might add these changes to that, or make another pull request. Try the work-around of making the file "fpm.exe.rsp" and see if you discover other issues or if that resolves everything if you can.

[zoziha]

So stripping off the suffix, and also adding : as an alternative for @ will make it far more usable in those environments.

I use fpm.exe.rsp in powershell/bash locally and it does work, I think using fpm `@run in powershell is acceptable as long as there are helpful instructions in M_CLI2/fpm that we can find them.

Thank you for your reply and work, @urbanjost :)

[urbanjost]

OK. Will just add stripping off file suffix. Documentation already said it did not; but it was not doing it correctly in the version used by fpm(1). Response files were a test feature that became more popular than expected, particularly because they are barely mentioned in the fpm(1) documentation. So one of the remaining tasks it to add QA into the unit test for M_CLI2 that includes testing response files. The need for them should decrease when user-defined profiles are supported in the fpm.toml file but that feature has been delayed, and other users of M_CLI2 can use the feature as well so it needs a going-over but I will try to get this incorporated soon.

[urbanjost]

The bleeding-edge release of fpm(1) would look for fpm.rsp on all platforms, where it was looking for fpm.exe.rsp in several MSWindows configurations.

https://github.com/fortran-lang/fpm/releases/tag/current