Add a configuration system

[melrief]

Currently hsl provides a default API to the user. I propose to extend this system such that the user can still use the default API but can also improve it by adding its own functions and modules or even avoid the default API and define its own. The configuration should be under a certain directory, for example $HOME/.hsl.

Currently HSProces uses the following layout

• $HOME/.hsp/toolkit.hs: a Haskell file containing the user defined functions and everything the user exported
• $HOME/.hsp/modules: contains a list of modules with aliases to be exported. This is due to the fact that in Haskell is not possible to export a module by renaming it. Each line is composed by the module name, a space, and the alias. For example, a line Data.List L means import Data.List as L
• $HOME/.hsp/cache: contains the compiled version of the toolkit to avoid multiple compilations. In this way the bootstrap of HSP should be faster

I propose to follow the same layout in hsl. There are some changes to do:

• a library System.Console.HSL.Utils with the current API of hsl will be created such that the user can use it inside its own configuration. For example, the json utilities will be inside this library.
• the first time hsl will be executed, a default toolkit.hs and module will be installed containing imports from the current API of hsl. In this way,the default usage of hsl will be the same as it is right now.

[gelisam]

I agree with $HOME/.hsl/toolkit.hs, and I agree that we should keep a compiled copy around, but I'm not sure storing that copy next to the configuration file is the best place. I don't really mind, though, and we can easily change that later without breaking anybody's workflow.

Is there any particular reason why $HOME/.hsp/modules uses its own syntax? I would just use whatever is qualified-imported inside $HOME/.hsl/toolkit.hs, I think that would be quite an intuitive behaviour. I did not know that Haskell did not allow you to export a renamed module, but just because Haskell doesn't allow it doesn't mean we can't :) I suggest we parse toolkit.hs and scrape off any qualified (or unqualified?) import.

a library System.Console.HSL.Utils with the current API of hsl will be created such that the user can use it inside its own configuration. For example, the json utilities will be inside this library.

I think making the hist and json stuff optional by stuffing them into an importable module is a great idea! Maybe even organize them into multiple modules, once we get enough?

the first time hsl will be executed, a default toolkit.hs and module will be installed containing imports from the current API of hsl. In this way,the default usage of hsl will be the same as it is right now.

I don't think hsl has a big enough user base for backward-compatibility issues to be of concern at this point. By I agree that it would be nice to pre-populate ~/.hsl, if only to give the user a base on which they can easily build on.

[ssadler]

What about if the first time that HSL was run, we create the file .hsl/default.hs and populate it with a default set of imports? Then, when we actually go to evaluate the user's expression, we can include both default.hs and all of the imports defined in default.hs. The effect would be that the context would be a concatenation of default.hs and main = {user's expression}. Then the user can easily modify default.hs, or create different entry points. We avoid a single modules file, and allow the user to override default.hs to another file with a command line switch. This could look like:

.hsl/default.hs

import qualified Data.ByteString as B
import qualified Data.Text as T
[ Other standard library imports ]
import System.Console.HSL.Core.Types
import System.Console.HSL.Core.Data
import System.Console.HSL.Core.Runners
import System.Console.HSL.Exts.Magic

[ space where user can write functions ]

Given the above, to set up InteractT context, manually parse and add user's imports, and also import file itself for user's function definitions.

[gelisam]

Maybe we should call it $HOME/.hsl/prelude.hs, to make it really clear that this is the code available to all commands?

[ssadler]

I think we all have the same instinct here :) but I also think that $HOME/.hsp/modules is unneccesary.

[gelisam]

I also think that ~/.hsl/modules in unnecessary, we should obtain the information from ~/.hsl/prelude.hs.

[melrief]

I agree on $HOME/.hsl/modules, that was a hack and I was searching for a way to remove it. I'm also ok with the $HOME/.hsl/prelude.hs, it seems the natural solution.

[ssadler]

Wait.. Prelude implies that it's a default API available everywhere. I think that we should provide a default API to the user but allow them to change it, which makes Prelude a bad name, because the Prelude exports a static API, whereas default.hs is the user's playground. I'm not sure if we have the same or a different idea.

[melrief]

I'm for let the user configure everything and that includes the possibility to not use default.hs (or prelude.hs). Do you agree on that or do you think too much freedom could harm the project?

[ssadler]

Prefer as much flexibility and as little convention as possible, so i agree. They should be able to do -f mystuff and use ./hsl/mystuff.hs instead.

[melrief]

No I have to contradict myself: at least the Types.hs file should be always imported or the compiler won't compile it because it will miss some informations. If we aspect a function with type Renderable a => Text -> a then Renderable must be imported always or no compilation is possible. That's why in HSP I force the import of Representable always. It is hardcoded in the bootstrap and there is no way to modify it.

[ssadler]

Right... I think the call to interpret should always return a IO (). That way you don't need to have any knowledge of underlying types. But it does mean you need to prefix and suffix the user's expression with a runner function which is available in his context. These runner functions will depend on command line flags ie --bytestring / --text / --magic. I guess it makes sense for those imports to be added automatically.

[melrief]

Sorry ssadler, I did another mistake: interpret should return IO () also because Renderable a is ambiguous for hint. There is not easy way to evaluate an expression with polymorphic type from hint. So you are right.

[gelisam]

I think the fact that the prelude lives in ~/.hsl and not in /etc is already a good hint that the user is expected to change it.

But really, the name of the configuration file is a minor detail, and I don't want to fuel a bikeshed discussion. I propose that whovever implements a feature should decide on this kind of small details. And then we can have an edit war over it :)

– Samuel

On 2013-08-07, at 5:43 PM, ssadler notifications@github.com wrote:

Wait.. Prelude implies that it's a default API available everywhere. I think that we should provide a default API to the user but allow them to change it, which makes Prelude a bad name, because the Prelude exports a static API, whereas default.hs is the user's playground. I'm not sure if we have the same or a different idea.

— Reply to this email directly or view it on GitHub.

[gelisam]

I did not know about the hint library. Nice, much simpler than parsing ghci's output!

[melrief]

I had some problems with hint by the way. For instance, you can't import single functions from a module instead of the whole module. Another problem is on the errors: not as clear as ghci. Hint is a good start but I think in future we could try to patch it to be a little more powerful.

[gelisam]

Do you think it would be easier to patch hint or to interact with a hidden ghci process, parsing its output when necessary?

[melrief]

I think that for now hint is perfect. If in future we have to patch it then I can assure you that its code is readable and not difficult to modify.

[ssadler]

I had some problems with hint by the way. For instance, you can't import single functions from a module instead of the whole module. Another problem is on the errors: not as clear as ghci. Hint is a good start but I think in future we could try to patch it to be a little more powerful.

You can potentially work around this with re-exports, but you can't re-export a module alias, so it's hacky.

[melrief]

I'm trying to implement the solution with Prelude.hs where the user imports stuff. First of all, hint works on expression, not real Haskell files. If we stick with this technique we should switch to ghc. But then there are two more problems:

• the compilation can be slow due to the fact that it compiles everytime a Haskell file that can potentially contain structures and functions definitions. I think Prelude.hs should contain only imports.
• the user must not change the module name. If Prelude.hs contains a different module then ghc will complain.

Those two reasons were why I decoupled the toolkit from the imports. The toolkit in hsp is a normal haskell module, it doesn't need (but could) be Main. It will automatically imported if it exists. The module file contains only imports to force the user to not set anything else.

For performance reasons we should compile only the user expression with the context. Everything else should be pre-compiled. I see three solutions:

• stay with the current module and toolkit.hs system. This is a sort of hack and I don't like it.
• the user define Prelude.hs as Haskell file and then at runtime parse it to sanitize it and to extract all the definitions to a "cache file" that will be compiled separately. Not easy but we can do it, it requires to change from hint to ghc. It is still a hack and I don't like it.
• let the user define a hawk session in a configuration file Session.hs instead of a Haskell file with imports. This file will be compiled in something that can be used to create a hint session. It can be also used to set some default options of hawk. For example:

[user functions]

main = hawk $ defaultHawkSession {
      modulesImported = [("Data.List",Just "L"), ("Data.Char",Just "C"), ("ByteString",Nothing)],
      onError = reportError >> exitFailure,
}

hawk is the Main function of Hawk while defaultHawkSession is the configuration of hawk. This system is similar to what xmonad does with xmonad.hs. Session.hs will be compiled only one time and the output is the Main module of hawk with that configuration. Note that the user can define functions in this file and then use them. We can easily instrument hint to load this file by default. I was exploring this solution with hsp because it is standard Haskell (not like the modules file) and doesn't require hacks. Still I think this system can be problematic because the user is defining a configuration inside a Haskell file, not a Haskell module.

I don't know what is the best solution...

[gelisam]

hint works on expression, not real Haskell files.

But presumably, those expressions can call functions from existing modules? Couldn't we compile ~/.hawk/prelude.hs as a module, install it somewhere then import it in the same way that we are currently loading ByteString? This way, we will only need to recompile (using ghc) if the prelude file has changed, and we will only use hint to evaluate the user's expression.

the user must not change the module name. If Prelude.hs contains a different module then ghc will complain.

I don't think we should allow the user to write a module declaration in this file. We should prepend our own module declaration before recompiling it.

This system is similar to what xmonad does with xmonad.hs

Ah, I was wondering how xmonad did it! I'm glad one of us knows :) I don't understand your description of it, though. What is defaultHawkSession? Some record which you override with record-update syntax? Is defaultHawkSession created from Session.hs, or does it represent the actual default configuration, when ~/.hawk doesn't exist?

Still I think this system can be problematic because the user is defining a configuration inside a Haskell file, not a Haskell module.

Is there any difference between a Haskell file and a Haskell module beside the module declaration at the top?

[melrief]

hint works on expression, not real Haskell files.

But presumably, those expressions can call functions from existing modules? Couldn't we compile ~/.hawk/prelude.hs as a module, install it somewhere then import it in the same way that we are currently loading ByteString? This way, we > will only need to recompile (using ghc) if the prelude file has changed, and we will only use hint to evaluate the user's > expression.

Like toolkit.hs in hsp? Yes, that's the idea about the user defined functions. But we can't do this with module import.

the user must not change the module name. If Prelude.hs contains a different module then ghc will complain.

I don't think we should allow the user to write a module declaration in this file. We should prepend our own module declaration before recompiling it.

K for me

This system is similar to what xmonad does with xmonad.hs

Ah, I was wondering how xmonad did it! I'm glad one of us knows :) I don't understand your description of it, though. What is defaultHawkSession? Some record which you override with record-update syntax? Is defaultHawkSession created from Session.hs, or does it represent the actual default configuration, when ~/.hawk doesn't exist?

XMonad configuration is very smart: basically the configuration file will be compiled to be the XMonad bootstrap. In this way, when the user starts XMonad, XMonad loads only the dynamic part. For example, my configuration xmonad.hs contains at the end :

main = xmonad $ defaultConfig {
       modMask = mod4Mask
      ,mouseBindings = mouseB
      ,workspaces = show <$> [1 .. 3]
      ,normalBorderColor = "gray"
      ,focusedBorderColor = "cyan"
      ,keys = myKeys
      ,focusFollowsMouse = True
      ,manageHook = myManageHook
      ,layoutHook = customLayouts
      ,logHook = myLogHook <+> dbusLog client
      ,startupHook = myStartupHook
    }

xmonad is the function that configure and run XMonad and takes as first argument a configuration. XMonad library provides a default configuration defaultConfig that can be update using record-update syntax. The compilation will then create an executable that is XMonad with that configuration, already set. We could do the same for Hawk: we could have a file hawk.hs that contains the configuration of Hawk including the modules to import. This file will be compiled to the real executable of Hawk with all the configuration inside. Still the problem is that we declare modules not with import but as tuples...I don't know if people will like it.

Still I think this system can be problematic because the user is defining a configuration inside a Haskell file, not a Haskell module.

Is there any difference between a Haskell file and a Haskell module beside the module declaration at the top?

I didn't explain myself. With this system the user will have to declare modules in this way:

modulesImported = [("Data.List","L")...]

instead of

import qualified Data.List as L

I think the difference is big. What do you think?

[gelisam]

Like toolkit.hs in hsp? Yes, that's the idea about the user defined functions. But we can't do this with module import.

We can easily grep toolkit.hs lines for lines which begin by import qualified, and convert that into the required list of tuples.

For example, my configuration xmonad.hs contains at the end : [...]

Much clearer now, thank you!

This looks like a clever way to configure the frontend+backend part of the tool, if it needs any configuration. We could use this trick to set the characters on which to split lines and columns, for example. However, prelude.sh and the list of modules to import is not used by those parts of the tool, but by the runtime!

I think the difference is big. What do you think?

I agree that the difference is very big! I would much prefer the second syntax.

[melrief]

Grepping imports could be an idea and the result can also be cached. Just to wrap up, now in ~/.hawk we have:

• Prelude.hs contains import lines, everything else will be reported as an error. It will be cached
• Default.hs contains an automatically imported module that contains user functions (In future we could use a directory lib in which the user can defines many files). It will be cached
• Hawk.hs the "main" file with the static configuration.

Do I'm missing anything?

[gelisam]

Why not grep the import line from the same file in which the user functions are?

Which static options do we have?

– Samuel

On 2013-08-08, at 5:16 PM, Mario Pastorelli notifications@github.com wrote:

Grepping imports could be an idea and the result can also be cached. Just to wrap up, now in ~/.hawk we have:

Prelude.hs contains import lines, everything else will be reported as an error. It will be cached Default.hs contains an automatically imported module that contains user functions (In future we could use a directory lib in which the user can defines many files). It will be cached Hawk.hs the "main" file with the static configuration. Do I'm missing anything?

— Reply to this email directly or view it on GitHub.

[melrief]

I think those are different imports: the ones in Default.hs are used for user defined functions, the ones in Prelude.hs are the import for the user expression. I can, for example, use Map in some functions in Default.hs but not expose the module Map itself. Or I can import it in Default.hs without qualification while I would like to have the qualification M in Prelude.hs.

[gelisam]

Yes, you can. But is the extra complexity (from the point of view of the user) worth it? If we use the prelude for both the imports and the definitions, then the semantics are clear and intuitive: whatever expression you could write at the end of prelude.hs, you can write as a hawk expression.

For the same reason, I would keep all the other options as command-line flags instead of using xmonad's complicated-to-understand setup. Unless we plan to have as many commonly-tweaked options as xmonad :)

– Samuel

On 2013-08-08, at 6:45 PM, Mario Pastorelli notifications@github.com wrote:

I think those are different imports: the ones in Default.hs are used for user defined functions, the ones in Prelude.hs are the import for the user expression. I can, for example, use Map in some functions in Default.hs but not expose the module Map itself. Or I can import it in Default.hs without qualification while I would like to have the qualification M in Prelude.hs.

— Reply to this email directly or view it on GitHub.

[melrief]

Even if I prefer the separation of runtime configuration and user utilities, I don't see any problem to use just one file for now and then, eventually, test different configuration systems in new branches and see how they work. But now the priority should be to release a software that can be used everyday.

Let me summarize the ideas to see if we are synchronized about it: inside the directory $HOME/.hawk there will be a file regular Haskell file called Prelude.hs. When hawk is started, this file is parsed and:

• the imports are converted to the hint format, for example import Data.Map is converted to ("Data.Map",Nothing), and cached inside $HOME/.hawk/cache/imports
• the file itself is converted to a module (by adding module <random_generated_name> where at the beginning) and compiled in $HOME/.hawk/cache/Prelude

Hawk then spawns a hint session by setting as $HOME/.hawk/cache/imports plus the $HOME/.hawk/cache/Prelude as imported modules.

The next time Hawk is executed, it checks if the cache is up-to-date. If it is, then it uses the cache and avoid recompilation. If not, it recreates the cache.

An example of Prelude.hs could be:

import Data.Int
import qualified Data.List as L
import Data.Text
import System.Console.Hawk.Representable

at = flip (L.!!) -- infix version of !! with arguments switched

The expected cache should be:

• $HOME/.hawk/cache/imports:

[("Data.Int",Nothing),("Data.List",Just "L"),("Data.Text",Nothing),("System.Console.Hawk.Representable",Nothing)]

• $HOME/.hawk/cache/Prelude: the compiled file

At this point the user should be able to do

hawk -m 'at 1 . words'

where words is from Data.Text and at is the user defined function.

Is this what we have in mind?

[gelisam]

+1

except I would use a format for $HOME/.hawk/cache/imports which I can read instead of interpret, but that's an implementation detail and should therefore be chosen by whoever does the work.

I'm surprised we are spending more time discussing than implementing! That must be a corrolary of "adding more people to a project makes it later".

– Samuel

On 2013-08-09, at 8:53 AM, Mario Pastorelli notifications@github.com wrote:

Even if I prefer the separation of runtime configuration and user utilities, I don't see any problem to use just one file for now and then, eventually, test different configuration systems in new branches and see how they work. But now the priority should be to release a software that can be used everyday.

Let me summarize the ideas to see if we are synchronized about it: inside the directory $HOME/.hawk there will be a file regular Haskell file called Prelude.hs. When hawk is started, this file is parsed and:

the imports are converted to the hint format, for example import Data.Map is converted to ("Data.Map",Nothing), and cached inside $HOME/.hawk/cache/imports the file itself is converted to a module (by adding module where at the beginning) and compiled in $HOME/.hawk/cache/Prelude Hawk then spawns a hint session by setting as $HOME/.hawk/cache/imports plus the $HOME/.hawk/cache/Prelude as imported modules.

The next time Hawk is executed, it checks if the cache is up-to-date. If it is, then it uses the cache and avoid recompilation. If not, it recreates the cache.

An example of Prelude.hs could be:

import Data.Int import qualified Data.List as L import Data.Text import System.Console.Hawk.Representable

at = flip (L.!!) -- infix version of !! with arguments switched The expected cache should be:

$HOME/.hawk/cache/imports: fromList [("Data.Int",Nothing),("Data.List",Just "L"),("Data.Text",Nothing),("System.Console.Hawk.Representable",Nothing)] $HOME/.hawk/cache/Prelude: the compiled file At this point the user should be able to do

hawk -m 'at 1 . words' where words is from Data.Text and at is the user defined function.

Is this what we have in mind?

— Reply to this email directly or view it on GitHub.

[melrief]

Discussing about features with other people takes time, I think, but it also force us to talk about the features before implementing them and that should be good for the project.

About imports, I think files inside $HOME/.hawk/cache shouldn't be (necessarily) readable by the user. Their main purpose is to be fast to load. Ideally the cache directory shouldn't be visible to the user, like the cache of any other program.

The current configuration system of hsp can be adapted to work with Prelude.hs, so if you are ok with it I can start to work on it. I think I should take this assignment because I already know the current system. Give me the green light if you are ok.

[gelisam]

Green light given! I agree that cache doesn't need to be readable; sorry for my previous comment, it was very misleading. I was trying to say that since we need to pass the list of values to hint before it can evaluate expressions, I expected the format to be in a form that can be read (not by me, not by the user, but by the read function) by our program and not in a form which needs to be evaluated by hint in order to be given to hint.

[ssadler]

Looks good. I like the strategy of just having one file, parsing out the imports manually and also importing the file.

[melrief]

I have almost completed the implementation of the new system, now I have to parse and export the modules that the user has imported in Prelude.hs. Do you mind if, instead of using regular expressions, I use haskell-src? I know it is another dependency for hawk but that package contains all the utilities needed for haskell source manipulation and can be useful also for future manipulations. For now we can use it just for module parsing, because it supports every version of module import in Haskell. What do you think? Do you mind if I add another import?

[gelisam]

I really don't mind extra dependencies, since cabal will install them all for the user anyway. And I bet this particular library is going to be very handy when the magic module will need to parse and interpret the type of the user's expression!

– Samuel

On 2013-08-11, at 9:51 AM, Mario Pastorelli notifications@github.com wrote:

I have almost completed the implementation of the new system, now I have to parse and export the modules that the user has imported in Prelude.hs. Do you mind if, instead of using regular expressions, I use haskell-src? I know it is another dependency for hawk but that package contains all the utilities needed for haskell source manipulation and can be useful also for future manipulations. For now we can use it just for module parsing, because it supports every version of module import in Haskell. What do you think? Do you mind if I add another import?

— Reply to this email directly or view it on GitHub.

[melrief]

Ok I have pushed a new branch config_with_prelude with the new configuration system. Now Hawk searches for the file $HOME/.hawk/prelude.hs, compiles it, extract all the modules and creates the cache directory.

Since this is a big change, I haven't merged it yet. Can you both test the branch when you have time and see if it is working in the proper way? On my computer it works apparently. In the meanwhile I'll try to create some tests.

[gelisam]

Great! At least one of us is producing code :) A few remarks:

~/.hawk is created automatically, but ~/.hawk/prelude.hs isn't.

--help stil refers to toolkit.hs instead of prelude.hs

I can't use reverse nor P.reverse from prelude.hs? I would either expect the standard behaviour of importing the prelude by default or, failing that, having at least the same kind of qualified access as is available in the user expressions. In fact, it felt kind of weird and undocumented for those user expressions to have qualified access to Prelude as P without having explicitly requested it to be qualified. Default-populating ~/.hawk/prelude.hs with import qualified Prelude as P would fix both issues at once.

Hawk crashes if ~/.hawk/prelude.hs is empty or contains only newlines:

> touch ~/.hawk/prelude.hs
> cat ~/.hawk/prelude.hs
> hawk -e [1..3]
hawk: Data.ByteString.head: empty ByteString
> printf "\n\n\n" > ~/.hawk/prelude.hs
> cat ~/.hawk/prelude.hs

> hawk -e [1..3]
hawk: Data.ByteString.head: empty ByteString

Importing Prelude from ~/.hawk/prelude.hs makes less functions available, rather than more!

> hawk seq 3 | hawk 'unlines . P.reverse . lines'
3
2
1

> echo "import Prelude" > ~/.hawk/prelude.hs
> seq 3 | hawk 'unlines . P.reverse . lines'

Won't compile:
        Ambiguous occurrence `unlines'
It could refer to either `Data.List.unlines',
                         imported from `Prelude'
                         (and originally defined in `base:Data.List')
                      or `Data.ByteString.Lazy.Char8.unlines',
                         imported from `Data.ByteString.Lazy.Char8'

        Ambiguous occurrence `lines'
It could refer to either `Data.List.lines',
                         imported from `Prelude'
                         (and originally defined in `base:Data.List')
                      or `Data.ByteString.Lazy.Char8.lines',
                         imported from `Data.ByteString.Lazy.Char8'

And since the Data.ByteString import is hardcoded, the user doesn't have the opportunity to make the ByteString version qualified in order to solve the ambiguity, so he is forced to make the prelude qualified.

[gelisam]

I just wanted to mention that adding import qualified hiding (lines, unlines) does not fix the above issue. I kind of understand why that is the case, but that's only because I know how this is implemented. As a user, I would be quite confused as to why there was still a conflict.

I guess it's not possible to tell hint to import a module while hiding some of its symbols. But is it possible to list which symbols are exported by a particular module? If so, we could simulate import..hiding by importing the module with a long name, then defining a bunch of local (unqualified) functions resolving to the visible imported symbols. If those definitions are added to prelude.hs when we add its module declaration, those functions will only need to be generated and compiled once.

[melrief]

The current import system is far from complete :(. There are many way to write an import in Haskell and handle every possibility is complex. So I did a first very primitive system that imports only a few cases:

• import moduleName: should be imported correctly
• import moduleName as alias: the alias should be usable
• import qualified moduleName: should be imported correctly and with the qualification
• import qualified moduleName as alias: should be imported correctly with alias as qualification

All the other cases are still not covered, in particular selective imports and hiding imports. To be honest @gelisam I did the commit to show that I'm doing progress on the configuration side. The code is currently based on the hsp's one and I feel great responsibility in putting the code in a stable state such that you and @ssadler can start working. I understand now it is a mess. Still some progress has been done and now we can improve all the aspects of the new configuration system.

The default file needs to be written and I didn't have time. Yet, the creation of the directory is a first step toward that direction.

[gelisam]

And I am very happy with the progress! Don't worry about the state of the codebase, I think we can refactor as we go along.

I will create issues corresponding to all the missing pieces, this way we can split the remaining work among all three of us.

[gelisam]

And since those new issues will represent all the remaining work, this issue can be closed! Good work, @melrief!