Create Sub-Packages

[samm82]

We need to split the drasil package into the subpackages drasil-lang, drasil-data, drasil-code and drasil-example.

• [x] Create drasil-lang, drasil-data, drasil-code and drasil-example
• [x] Update Makefile to build drasil-code
• [x] Update Makefile to build drasil-data
• [x] Update Makefile to build drasil-example
  • [x] Fix Travis Build
• File Reorganization Update READMEs (#769) drasil-gen (#767) drasil-printers (#766) drasil-docLang (#768)
  • [x] Add Language.Drasil.Development to drasil-lang
• [x] Add stack clean to Makefile
• [x] Check (concept ^. uid) --shortname (see db37504) Clean up make (#771)

[JacquesCarette]

Correct.

[samm82]

Also, where should these files end up? Should they just all be in code? EDIT: Apparently that throws an error - I put them in their respective folders.

[samm82]

@elwazana suggested that the problem might be in the config file. Would someone be able to see if that's a potential problem? @JacquesCarette @szymczdm I'm all out of ideas on how to implement this.

[JacquesCarette]

So what are the outstanding issues?

[samm82]

I've created the four .cabal files (hopefully correctly) and can't seem to get them all to be built in the Makefile. When I talked to @szymczdm, he said that it was an issue with stack and not the Makefile. Some things I have tried:

1. You can't have more than one .cabal file in the same directory, so I moved them to their respective locations. I tried putting them in their appropriate folders in /code, but currently I'm trying them in their own folders (ie. drasil-data is in /Data) within the build folder. I'm not sure if this is correct or not. Regardless, drasil-code.cabal is just in /code, where drasil.cabal used to be.
2. I can't seem to get the Makefile to "find" the directory containing the .cabal files, including by adding cd <dir-name> to the Makefile, and trying to build /Data/drasil-data instead of just drasil-data (so it would know where the file is located).

I haven't committed/pushed anything to my branch since creating the .cabal files because nothing I've done has worked. One thing I have done, which seems to have resolved #2, results in this error:

$ make
mkdir -p build/Data
cd build/Data && stack build -j2  drasil-data
drasil-data-0.0.9: build (lib)
Preprocessing library drasil-data-0.0.9...
Cabal-simple_Z6RU0evB_1.24.2.0_ghc-8.0.2.exe: can't find source for
Data\Drasil\People in ., .stack-work\dist\ca59d0ab\build\autogen

--  While building custom Setup.hs for package drasil-data-0.0.9 using:
      C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_1.24.2.0_ghc-8.0.2.exe --builddir=.stack-work\dist\ca59d0ab build lib:drasil-data --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
make: *** [Dbuild] Error 1

which makes it seems as though drasil-data has been found, and it can't find Data\Drasil\People, which is the first item in the drasil-data.cabal. Do you think I would have to edit the file path in the .cabal to reflect the change in directory (as the .cabal file is now in build)?

[JacquesCarette]

One idea: I think that since drasil-data.cabal is in the Data directory, it will look for Data/Data/Drasil/People. There's something with 'source-dir' that can be done with that. Maybe even set it to .. !

[samm82]

So I changed the file paths in the cabal file in commit 59ea5a6, and now I just need to change the module names in each of the files 🙄

[JacquesCarette]

No, please don't!

[JacquesCarette]

We want the module names to stay as is. I like them, they are good.

We might need to move the source files though. Maybe we need to create sub-directories for each of drasil-code, drasil-data, etc.

[samm82]

Just for completeness' sake, this was the error that resulted:

C:\Users\samcr\Desktop\Summer Research\Drasil\code\Data\Drasil\People.hs:1:8: error:
    File name does not match module name:
    Saw: `Data.Drasil.People'
    Expected: `Drasil.People'

So should I make a Data folder within Data? I think that would be the only way to preserve the module names. EDIT: It seems to be working -> 6c1feb2

[JacquesCarette]

Hmm, if we are willing to upgrade to the latest version of cabal, then I think we can switch to cabal-dev (from cabal) and get this feature in a nice way:

https://stackoverflow.com/questions/18123911/cabal-with-multiple-library-sections

[samm82]

Currently, it is working, except that I had to move one file as per a .cabal file, but now an .hs file needs it to import.... it seems like it needs one file to be two places at once, which can't be good.

[JacquesCarette]

Details? We might have some of our files importing a little too eagerly across 'logical' packages that need to be fixed.

[samm82]

[52 of 75] Compiling Language.Drasil.Chunk.Code ( Language\Drasil\Chunk\Code.hs, .stack-work\dist\5c8418a7\build\Language\Drasil\Chunk\Code.o ) [Language.Drasil.Code.Code changed]

C:\Users\samcr\Desktop\Summer Research\Drasil\code\Language\Language\Drasil\Chunk\Code.hs:21:1: error:
    Could not find module `Language.Drasil.Code.Code'
    Use -v to see a list of the files searched for.
   |
21 | import Language.Drasil.Code.Code as G (CodeType(..))
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

--  While building custom Setup.hs for package drasil-lang-0.0.9 using:
      C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_2.0.1.0_ghc-8.2.2.exe --builddir=.stack-work\dist\5c8418a7 build lib:drasil-lang --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
make: *** [build_lang] Error 1

It cannot find code/Language/Language/Drasil/Code/Code.hs since it was moved to code/Language/Drasil/Code/Code.hs - if I move it back, I get this:

Cabal-simple_Z6RU0evB_1.24.2.0_ghc-8.0.2.exe: can't find source for
Language\Drasil\Code in ., .stack-work\dist\ca59d0ab\build\autogen

--  While building custom Setup.hs for package drasil-code-0.0.9 using:
      C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_1.24.2.0_ghc-8.0.2.exe --builddir=.stack-work\dist\ca59d0ab build lib:drasil-code --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
make: *** [build] Error 1

since the file is no longer where it expects it: code/Language/Drasil/Code

[JacquesCarette]

I think Language.Drasil.Chunk.Code should be moved out of Language. It doesn't need to be a chunk.

[samm82]

If Chunk.Code is moved out of Language, what happens to the files inside of Language that are dependent on it?

[JacquesCarette]

It depends -- which ones are those?

[samm82]

Language.Drasil.Code.Imperative.Import Language.Drasil.CodeSpec Language.Drasil.DataDesc Language.Drasil.Expr.Extract

[JacquesCarette]

The first two should be moveable to Code too (though that is likely to have a small knock-on effect too). I would like to know what pieces in DataDesc and Expr.Extract depend on Chunk.Code. The surgery needed there might be more delicate.

[samm82]

They both use CodeChunk and codevar, which takes a HasAttributes c, Quantity c and returns a CodeChunk.

Note: The import statement for both of these imports the whole module, not just the utilized features - would it be a good idea in the future (perhaps as a time-filler task) to make the import statements throughout be more specific?

Also, what do you mean by "movable to Code"? The problem is that these files are in Language and Code is in Language. Did you mean code, the main folder, and if so, is it OK having loose files in there?

[JacquesCarette]

On the note: yes, definitely.

Can you show me the exact lines in DataDesc and Expr.Extract that are affected?

[samm82]

In DataDesc:

https://github.com/JacquesCarette/Drasil/blob/6c1feb24afcda53008eba6f685728a4debcf0e51/code/Language/Language/Drasil/DataDesc.hs#L10

https://github.com/JacquesCarette/Drasil/blob/6c1feb24afcda53008eba6f685728a4debcf0e51/code/Language/Language/Drasil/DataDesc.hs#L32-L42

In Expr.Extract:

https://github.com/JacquesCarette/Drasil/blob/6c1feb24afcda53008eba6f685728a4debcf0e51/code/Language/Language/Drasil/Expr/Extract.hs#L70-L78

[JacquesCarette]

codevars and codevars' can definitely be moved out of Expr.Extract to where they are used.

And DataDesc is badly named. It should be Code.DataDesc and should be safe to move out (the whole file) to drasil-code.

[samm82]

I now need to rework because of 1d75a42:

Cabal-simple_Z6RU0evB_2.0.1.0_ghc-8.2.2.exe: can't find source for
Language\Drasil\DataDesc in ., .stack-work\dist\5c8418a7\build\autogen,
.stack-work\dist\5c8418a7\build\global-autogen

--  While building custom Setup.hs for package drasil-lang-0.0.9 using:
      C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_2.0.1.0_ghc-8.2.2.exe --builddir=.stack-work\dist\5c8418a7 build lib:drasil-lang --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
make: *** [build_lang] Error 1

I didn't notice this was a straightforward fix until after I pushed 😅

[samm82]

Now I think we're getting a similar problem in a different location now:

$ make
mkdir -p build
stack build -j2  drasil-code
drasil-code-0.0.9: build (lib)
Preprocessing library drasil-code-0.0.9...
[16 of 16] Compiling Language.Drasil.Code.DataDesc ( Language\Drasil\Code\DataDesc.hs, .stack-work\dist\ca59d0ab\build\Language\Drasil\Code\DataDesc.o )

C:\Users\samcr\Desktop\Summer Research\Drasil\code\Language\Drasil\Code\DataDesc.hs:3:1: error:
    Failed to load interface for `Language.Drasil.Chunk.Code'
    it is a hidden module in the package `drasil-0.0.9'
    Use -v to see a list of the files searched for.

C:\Users\samcr\Desktop\Summer Research\Drasil\code\Language\Drasil\Code\DataDesc.hs:4:1: error:
    Failed to load interface for `Language.Drasil.Chunk.Quantity'
    it is a hidden module in the package `drasil-0.0.9'
    Use -v to see a list of the files searched for.

C:\Users\samcr\Desktop\Summer Research\Drasil\code\Language\Drasil\Code\DataDesc.hs:7:1: error:
    Failed to load interface for `Language.Drasil.Classes'
    it is a hidden module in the package `drasil-0.0.9'
    Use -v to see a list of the files searched for.

--  While building custom Setup.hs for package drasil-code-0.0.9 using:
      C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_1.24.2.0_ghc-8.0.2.exe --builddir=.stack-work\dist\ca59d0ab build lib:drasil-code --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
make: *** [build] Error 1

Code.DataDesc can't import what it needs from Language.Drasil because Code.DataDesc was moved.

[JacquesCarette]

Language.Drasil.Chunk.Code needs to move do drasil-code (and be removed from drasil-0.0.9). You'll also need to "stack clean" for all 4 packages too.

[samm82]

@JacquesCarette, I moved Language.Drasil.Chunk.Code to drasil-code, but I'm getting this error:

C:\Users\samcr\Desktop\Summer Research\Drasil\code\Language\Drasil\Chunk\Code.hs:#:1: error:
    Failed to load interface for `module imported from lang'
    it is a hidden module in the package `drasil-0.0.9'
    Use -v to see a list of the files searched for.

I think at this point it would be better to completely rework where files are, and have a module Code.Drasil.xyz... instead of Language.Drasil.Code.xyz.... I could be completely missing the point of a design choice that was made in the placement of these files, but because of the dependent modules, I feel that drasil-lang and drasil-code are too interconnected as of now.

It also seems like Language has been functioning as the "main" part of Drasil, but I've been working as if Code is - that could potentially be causing the issues addressed in the previous paragraph.

Also, you mentioned removing Language.Drasil.Chunk.Code from drasil-0.0.9. What is drasil-0.0.9? When I split the drasil.cabal into subpackages, I renamed it to drasil-code and removed the lang, data, and example components. So where is drasil-0.0.9? Is it a cabal file? Is is a module? The error message above suggests that it is a package, but I can't find a cabal?

I honestly am thinking that deleting the branch and starting over in light of what I've learned and the issues that have come up might be a good idea. I haven't really known what I've been changing, and haven't made a plan before implementing changes. Starting over from a big picture perspective might be advantageous.

[samm82]

Hiding previous comment for now

Now the issue is the same as Language.Drasil.Chunk.Code being removed and breaking other code. Right now, Language.Drasil.Code.DataDesc can't be found by:

Language.Drasil.Code.Imperative.Import

https://github.com/JacquesCarette/Drasil/blob/cd1684178ccb9efcb6c32d56a9416404575f0722/code/Language/Drasil/Code/Imperative/Import.hs#L585-L704 (pretty much everything is used)

Language.Drasil.CodeSpec

https://github.com/JacquesCarette/Drasil/blob/cd1684178ccb9efcb6c32d56a9416404575f0722/code/Language/Language/Drasil/CodeSpec.hs#L180-L181 (DataDesc) https://github.com/JacquesCarette/Drasil/blob/cd1684178ccb9efcb6c32d56a9416404575f0722/code/Language/Language/Drasil/CodeSpec.hs#L233-L249 (getInputs)

[JacquesCarette]

Language.Drasil.Code.Imperative.Import also needs moved to drasil-code.

The breaks after this will be more interesting. I think they may lead us to create another package outside of drasil-lang which is going to run the actual generators.

[samm82]

Language.Drasil.Code.Imperative.Import is already in drasil-code. I think we already need the generator package.

My issue now (after merging the changes from master onto my branch) is that Language\Language\Drasil.hs can't find Language.Drasil.Code.DataDesc or Language.Drasil.Code.Imperative.Lang, since they are now in the outer Language directory.

[JacquesCarette]

Language.Drasil should not re-export either of those. They should be in a Language.Drasil.Code (or some other for-devs-only sub-packaged).

[samm82]

The information from Language.Drasil.Code.Imperative.Lang got moved to CodeSpec - should Lang be deleted now or no?

---

Now, the issue is Language.Drasil.Chunk.Code is in the outer Language, but it needs only Language.Drasil.Code.Code (which it imports as G) from the outer and the rest from the inner one - it makes it seem as if it should be in the inner Language, especially since the only thing from Language.Drasil.Code.Code it imports is used in this one function:

https://github.com/JacquesCarette/Drasil/blob/dbeb5c656fdd9c5dafdd80dcef000da13a06f49b/code/Language/Drasil/Chunk/Code.hs#L152-L164

However, this function also requires Language.Drasil.Space (as S) to function, so this file is right between drasil-code and drasil-lang. How should we remedy this?

---

A cursory look through the outer Language shows that all files require modules from both the outer and the inner Language files. I think that the framework needs a redesign with drasil-code being Code.Drasil instead of Language.Drasil.Code.

[JacquesCarette]

If there's no information left in Lang, then yes. Also, moving that function to Code.Code seems like a sensible thing to try.

[samm82]

I tried that, but very quickly - it doesn't work (see edited comment above)

[JacquesCarette]

The filenames don't actually matter - it is how packages depend on each other that matters. The stuff in Code should be able to depend on Language/Drasil/... (hopefully though Language.Drasil only). The cabal file can be drasil-code depend on drasil-lang.

[samm82]

So do I have to go through the import statements of Language.Drasil.Code and change Language.Drasil.xyz to Language.Language.Drasil.xyz?

[JacquesCarette]

No, you should be able to get rid of most of the imports and just use Language.Drasil itself.

[samm82]

But because of the subfolder, I would have to be importing Language.Language.Drasil, which results in

1. changing the file names in Language.Language.Drasil and the entire folder and
2. a lot of work.

Is there a way to import Language.Drasil from Language\ without changing the module name? I can't seem to find any.

[JacquesCarette]

You need to have the drasil-code package depend on the drasil-lang package -- and not on its individual files. The same way we import various things from many different libraries.

I think there will be something related to 'install' (i.e. cabal install or stack install) that will be needed to make it work.

[samm82]

I'm not sure how to do that - I tried adding drasil-lang to the dependencies of drasil-code in the stack.yaml in commit 9033901, but that didn't work. When I run make on drasil-code, it can't find drasil-lang.

[szymczdm]

Is drasil-lang being built and installed prior to attempting to build drasil-code?

[samm82]

I'm honestly not sure. I think it is being built but I don't think it is being installed. Here is the relevant code:

https://github.com/JacquesCarette/Drasil/blob/90339012bab87fe0cad992b2833cbfb451497d72/code/Makefile#L38-L46

Does drasil-lang get installed if I add this/is this what @JacquesCarette was talking about with stack install?

FORCE: ;

build: FORCE
    mkdir -p $(LANG_DIR)
    cd $(LANG_DIR) && stack build -j2 $(stackArgs) drasil-lang
        **stack install $(stackArgs) drasil-lang**

build_code: build
    mkdir -p build
    stack build -j2 $(stackArgs) drasil-code

[szymczdm]

I believe you would need to run stack install for drasil-lang, yes, but I'm not sure if stack's sand-boxing could cause any problems here.

[samm82]

In 78bdf3e, despite being constructed in the same way as build, build_data results in this error:

stack install  drasil-data

Error: While constructing the build plan, the following exceptions were encountered:

Unknown package: drasil-data

Some potential ways to resolve this:

  * Set 'allow-newer: true' to ignore all version constraints and build anyway.

  * You may also want to try using the 'stack solver' command.

Plan construction failed.
make: *** [build_data] Error 1

Here is the relevant code:

https://github.com/JacquesCarette/Drasil/blob/78bdf3e0ad53ff3a14533a9e13f05947fcf95f8e/code/Makefile#L38-L53

---

Also, there is a difference in the log files: "This section provides general information about the system, identifies the interfaces between the system and its environment, and describes the user characteristics and the system constraints. ---> This section provides general information about the system including identifying the interfaces between the system and its environment (system context), describing the user characteristics and listing the system constraints." Would this be due to the recent merge of #615?

EDIT: Fixed -> I merged master onto SubPackages and no issues with the logs

[JacquesCarette]

The Makefile really has nothing to do with what happens when you do stack install drasil-data. For that, it is the drasil-data.cabal file which matters. The Makefile is in fact not used at all when you invoke stack directly!

[samm82]

So what kinds of things should go in the lang-dev module? I'm just adding everything that needs to be imported in drasil-code to Language.Drasil so that the code runs, and then we can probably split it up after.

The following seem like they should be in lang-dev:

• Language.Drasil.Format
• Language.Drasil.Output.Formats
• Language.Drasil.TeX.Print
• Language.Drasil.HTML.Print
• Language.Drasil.HTML.Helpers
• Language.Drasil.Make.Print

[samm82]

Also, should I make a Data/Drasil.hs file like the Language/Drasil.hs file to make a singular package for the examples to import, or no?

[JacquesCarette]

Hmm, now that I see that list, I think we need drasil-gen (which will have a few of the high-level generators in it) and drasil-printers, which will have the files you list.

A drasil-dev package (better name) would contain those things useful for developers of Drasil infrastructure (like adding to Data.Drasil), such as exporting Language.Drasil.Units.

And no, no big Data.Drasil file. These are really meant to be included one at a time.

[samm82]

Should drasil-printers be located in code/Printers, or code/Language/Drasil/Printers, or somewhere else? And I'm not really sure about drasil-gen and drasil-dev, like what files they needs to have and where they should go.

[JacquesCarette]

I really think that our hierarchy should start at code, and then under that

• drasil-lang
• drasil-code
• drasil-gen
• drasil-dev
• drasil-printers
• drasil-data
• drasil-examples

Under each of these, we can have Language.Drasil.*, Data.Drasil.*, etc, as needed. So parts of Language.Drasil will end up being defined over multiple packages. [The same way many many different haskell packages define things under Data.* and Control.*.