Some ideas & questions

[YPares]

Hi! Nice ideas you've got there :) You've put together something that's pretty close to what I had in mind myself (I'm doing similar stuff in https://github.com/tweag/porcupine , but with vinyl).

I have some questions:

• do you plan to make CLI a regular option source like any other, which you can deactivate? (for now it's mandatory to use it)
• are external file parsers configurable? (I'd like to use TOML or even some typed representations, like Avro or protobuf)

[YPares]

• is it possible to use several config files? (one overloading the others)
• Could it be possible to load config files from remote sources? (HTTP, AWS S3 etc.)

[alexpeits]

Hey @YPares, sorry for the late response - I must have missed a notification and literally just saw the issue! Thanks for the questions and suggestions, I will try to answer them in the same order:

• I have looked into making the cli optional. The only issue is that the whole thing is driven by optparse-applicative. All other options are taken from the defaults, so it would be something like: long "foo" <> value (envVar <|> defaultVal) if that makes sense. So if the command line option does not exist it will take the default, which will be the asum of sources and default values. The main reason is the error reporting and help. I have it in my radar to explore this but right now if you want to have no corresponding command line option you can declare an option without long and short. Not terribly useful though since you cannot turn it off and on by changing a source..

• I would like to think they are. There are 2 classes, one for getting the raw source and one for taking this raw source and converting it to an HKD option (GetSource and RunSource). So you can have an instance for the first that reads the file and the second that turns it into the option using whatever machinery the TOML library provides. Currently the implementation was driven by the way Aeson works and I seem to remember I could not get e.g. dhall to work, but I would like to have something easily extensible. I will take a look at how it would look with TOML (maybe using tomland?) when I am on my computer.

• Yes it is, the order is the same as the sources you define, although not overriding (imagine Last), but rather taking the first available one (imagine First). This means that the order of sources denotes the priority of each source. Would be nice to make this polymorphic over the Monoid (or Alternative since that's how merging currently works) that dictates how sources are merged.

• It can definitely be done, depending on the serialization format for the same reasons in point 2. E.g. if it is JSON then the only thing that changes is how to get the source (reading it from the network instead of the filesystem). It would require a new instance of both GetSource and RunSoure but the implementation of the second one should be identical to the current JSON source

Interestingly all of your points are on things I want to add very soon so I am very interested on all ideas/contributions!

[YPares]

So re. CLI, IMHO the simplest would be not to drive things via optparse-applicative. You could have your own datatype that'd contain field names and default value, and just use the optparse Opt as a backend. That's the way I did it in porcupine.

[alexpeits]

This how it was initially working (if I understand correctly what you mean). The issue was this: if I want to get the CLI value and then the env var and then the default (whatever is available first), I need to make it so that the parser always succeeds, which is easy because the f parameter in my HKD is Maybe so the value would be Nothing if it's not there. This means that optparse never fails for missing options (it only fails when it can't parse an option). But then if my environment variable parsing fails, I'm not in optparse-applicative-land anymore, so I need to render the errors on my own which is a bit inconsistent. An equally good option would be do all of the error reporting myself and not care about being consistent with optparse-applicative - which I am currently considering because having the CLI as a separate source would be sweet.

Do you have a specific example for how you do this?

[YPares]

I do it the other way: I start with the lowest priority environment, and parse only the CLI last. And I merge everything after that, with a flag in an HKD functor layer that indicates the source of the value (if it was from CLI, env vars), and I have an Ord instance over these sources flags to know which one has the priority. So my merging is actually commutative. So priority flags, default options and docstrings are all stored in my vinyl record via functor layers (it's via a HKD trick too, so it's not important it's an extensible record). Additionally, I have a Maybe layer so any field may be missing, but actually missing values seldom occur since I basically always have default values (so in the end, once everything is parsed, I just fail when a field is Nothing).

[alexpeits]

I see. This sounds similar to what I had in the first version and probably what I will be moving to in the future since being so tied to optparse-applicative isn't ideal. Although this might mean dropping the very limited support the library has for subcommands, but I don't see this as a major problem.

The only pain point is configuring the sources. For example, if you want a JSON file as a source, you first have to get the filename from the user somehow, then read it, decode it to your HKD and then get all the rest of the sources (and using optparse-applicative means that you can't just run one parser and then another, because one of them would find more options than it expects). It's done this way currently in harg but the implementation is not very easy to work with/extend..

So to summarize:

• point 3 is already supported
• point 4 is easy to implement, and the sources are extensible so that's good
• point 2 might be possible, depending on the library (I took a quick look into tomland and seems like it can be done if the datatypes have the appropriate instances)
• point 1 is the next thing to do on my roadmap

I'll post a gist with 2 and 4 as soon as I find some free time

[YPares]

http://hackage.haskell.org/package/optparse-generic maps the different constructors of a sum type to subcommands, maybe that could be doable here?

[YPares]

Re. getting the filenames, what I do is run a first simple, non optparse-applicative parsing of the raw list of args (I want the filenames to be the first elem of that list, and if it ends with .json or .yaml I consider it a filename and pop it, else I go straight to optparse phase). But that's a bit ugly.

[alexpeits]

The nice thing about making the first run work in a similar way as the rest is that you can just as well provide the path to the files that form the sources from environment variables. It does require a bit of plumbing though (I am putting a Const String before all the options the user supplies and run the parser with some defaults so it always succeeds).

[solomon-b]

Hi @alexpeits I'm looking into using HKDs for configuration parsing in a project. I would need commands in the arg parser and to make the arg parser able to be disabled per field.

harg unfortunately doesn't match these requirements, but based on your comments here it sounds like there isn't an obvious way to use optparse-applicative with HKDs regardless of harg and hit these 2 requirements?

If one were to greenfield a design with no attachment to existing libraries (such as optparse-applicative), do you see a way to solve these requirements and the other trade-offs mentioned in this issue or is this a deeper issue with using HKDs?

Also, are you still maintaining harg?