Rework command line args parsing

[luc-tielen]

The commandline args should be refactored in such a way that they make normal development workflows easier.

In the beginning we could focus more on the build related options (fetching code, compiling, ...) But it should not be limited to only building, it could also be used to quickly set up projects (including json files etc).

Maybe also make commands simpler, but put more information inside the JSON files? This would probably be idr-project.json then; containing metadata of the project, while the others are used more for dependency management (package sets etc)? Also I noticed you made things like build directories variable, but maybe this can also be stored in a config file (project.json?).. just an idea.

Example data structure could look as follows:

type PkgName = Text
data CodeType = Library | Executable
data Op = Fetch                 -- Fetches all dependencies as described in json file
        | Compile               -- Compiles all code (fetch needs to be done first to get dependencies code?)
        | Clean                 -- Cleans up build artifacts, fetched code?
        | Run                   -- Runs the executable defined in idr-project.json
        | Repl                  -- Opens up the repl
        -- following are more 'optional', maybe for later?
        | New PkgName CodeType  -- Initializes a new project built on the idream structure
        | Validate              -- Validates the JSON files used by idream
        | MkDoc                 -- Generates the documentation
        | GenerateIpkg          -- Generates a ipkg file from the idream JSON files
        | Test                  -- Runs unit tests for this project
        | Help                  -- Shows the help menu

Maybe also some commands specifically for dependencies need to be added? Also, work could be split out if each of these commands are implemented one by one (regarding your earlier question). Thoughts?

[luc-tielen]

Or maybe better to split up stuff like build dir in a .idream file or something similar, which is only used when that package is built as root package, and other stuff in idr-*.json..

[ejconlon]

Here are some of the constraints I tried to make the project obey:

1) idr-package.json files must represent ipkg files faithfully. 2) It should be easy to describe multiple independent projects in the same directory structure (not relying on auto-discovery of ipkg files). This is done with idr-project.json.

I think it would be fine to put more data into the project config, including build dir. I am only wary of putting more into package configs.

For the most part, the current implementation executes one thing at a time (build OR fetch OR clean, etc) but we should consider automatically building and fetching all dependencies when required. Also, I like the idea of commands to scaffold new projects or add dependencies to existing projects.

[luc-tielen]

Thanks for feedback! Now that I know the reasoning behind it, I'll try and take it into account.. (in my fork I already started experimenting with it). For now I first want the really basic features done in Haskell so that we have something going (in Haskell), and then add the extra features you summed up (multiple libs/bins in 1 repo, etc.) I split the actions (fetch, compile) up into separate files so this could be a point where we split up the work.

[luc-tielen]

Been very busy last week, but just implemented first command to generate a very basic project template (the "New" command above). I initially tried the fetch command, but it is already a fairly involved 1 because of dependency graph (but I have some temporary code already done).

Can you maybe take a look at ba4bfb8 and give some comments? There is obvious room for improvement but for now I just want to get a base laid out so other people can improve on it :smile: Next up is clean command (easy 1), and then will probably continue with fetch.

[ejconlon]

@luc-tielen Looks awesome! Clean style.

I have also been quite busy with the school starting, so I don't know when I will have time to contribute at all this month. (However, I don't think the Idris build landscape is changing that fast...)

[luc-tielen]

No worries! Education is also important :) I will just gradually continue porting the commands I described above. And yes, I think we have some leeway there. ;)

[luc-tielen]

So I finally finished implementing the "fetch" part of Idream in Haskell (see f1248e0921a06219c5c9be1ef8482da3b8af63a7). This is a rough outline so far, might still contain bugs (needs testing!).. It ended up being quite more involved than I initially foresaw. :man_shrugging:

The way I implemented it allows idream to build up a dependency graph along the way as it fetches more dependencies. Cycles should be safe also (for fetching, compiling might be different story...) since there is a check that filters out already fetched deps.

Also had to read up on a (good) paper about graphs in Haskell: http://eprint.ncl.ac.uk/file_store/production/239461/EF82F5FE-66E3-4F64-A1AC-A366D1961738.pdf, mainly used in src/Idream/Graph.hs I used it as a reference when implementing the dependency graph logic.

On a side note, we might need to look into how to debug monad transformer stacks (unless you know?); in some places, stacks are already getting quite large (see https://github.com/luc-tielen/idream/blob/haskell_refactor/src/Idream/Command/Fetch.hs#L97).

For the next 2 weeks or so I won't be able to program much, I got an exam of my own at work but after that I will help with other parts (they should be much easier now that we can just use the graph to figure out build strategy when compiling etc).

Also, feedback welcome.

[ejconlon]

I will take a look! Thanks!

(I tend to be pretty conservative with my transformer stacks; reader, state and IO cover almost everything... I will see if things need to or can be simplified.)

[luc-tielen]

Ok thanks! I added MonadLogger for logging but basically it is IO also. MonadError is used for converting all exceptions into errors that need to be checked (using tryAction etc).. Also MonadError should stop computation on first encountered error. Only used state monad in 1 Function (where it builds the graph along the way)

[ejconlon]

I think realistically you can just put a MonadThrow constraint on and just rethrow exceptions you want to have extra information attached to. I don't see any catch blocks!

[ejconlon]

You could also pick a fixed set of constraints for any command handler with LANGUAGE FlexibleConstraints:

type MonadHandler s m = (MonadThrow m, MonadIO m, MonadState s m, MonadReader Config m, MonadLogger m)

[luc-tielen]

The runExceptT deals with the error handling / runs that part of the transformer stack, actual error handling happens here https://github.com/luc-tielen/idream/blob/haskell_refactor/src/Idream/Command/Fetch.hs#L92 All errors get bubbled up to this line (sometimes wrapped in another type to make typechecker happy). Since all error types are structured in a hierarchical way, logging errors is fairly straight-forward.

Regarding FlexibleConstraints, I stay away from it on purpose since it would loosen the restriction on what each function could do. For example this would allow mutation of state in each of the functions, instead of the single one. Well, atleast that was my reasoning behind it.

Thanks for quick feedback btw :smile:

[luc-tielen]

Well, exam is over now, so I can focus on working on this again. Already been thinking about how to 'optimally' work on the compile step.

For now, I see 2 main options:

• actors, 1 actor per node of the dependency graph, they notify each other when a package is complete until all done.
• a more deterministic approach using the ParIO monad (even if it says it is not deterministic) for dataflow parallellism. I already have an algorithm thought out that separates the nodes of the graph into phases..

For the actors, haskell already has some libs but they feel either overkill or too basic, maybe best to write a small layer on top of the async library. The ParIO approach may be less optimal than the actor approach but it gives us more reasoning ability..

Haven't made my decision yet, maybe it's best to first read up on often used graph algorithms. At the moment I prefer the ParIO approach since it gives slightly more determinism (at the cost of performance).

Maybe also when all commands are implemented, the 2nd option could be implemented anyway and made available as command line option?

[ejconlon]

I think on a first pass it's not terrible to just topologically sort and proceed sequentially! I don't fully trust invoking idris itself in parallel yet.

[luc-tielen]

So. It's been a while, been busy with a lot of other stuff. But, I've also been busy with this repo! (And learned a lot of Haskell things in the process...)

I redesigned the app to use freer-simple, allows really fine grained handling of errors.. and composable effects which was what I was looking for. More commands have been implemented.. and I wrote some tests that build an Idris executable that has some dependencies.

So in summary.. it's going slow, but we're getting there! The next few commands should be much simpler and easy to implement, so that's always nice 😄.

[ejconlon]

Wow, looks so good! I am building now...

[ejconlon]

I get this when I run the integration test:

  1) idream add command fails when no args are provided
     Failure/Error: Dir.chdir File.join(test_dir, proj_name)

     Errno::ENOENT:
       No such file or directory @ dir_chdir - /tmp/idream_integration_tests/test_project
     # ./idream_add_spec.rb:11:in `chdir'
     # ./idream_add_spec.rb:11:in `block (2 levels) in <top (required)>'

I want us to have a simple project people can try the binary on right away so I might try to make the integration test use a source-controlled version.

[ejconlon]

I also pushed some doc changes to remove my name from prominence...

[luc-tielen]

What are you running on? Windows? I made the tests mostly for Linux / Mac. Maybe focus should be on CI first so we can hook it into travis again?

[ejconlon]

Mac. CI my the first goal.

[luc-tielen]

Strange. Ruby version? I got ruby 2.4.1p111 Using High Sierra as OS. Also, can we discuss all open issues somewhere in a central place? Right now it feels a little chaotic 😄 ..