Design: Consistent & Non-Abitrary Module Structure

[Divesh-Otwani]

The internal module structure has some major (not completely disjoint) problems:

• Arbitrariness
  • We shouldn't make arbitrary decisions as to what is or isn't in an internal module.
  • Example: why is flip defined in Prelude.Linear but not in Prelude.Linear.Internal? What is the rule by which we determine what goes in an internal module and what goes together?
  • Example: the same question applies to (<$) in Data.Functor.Linear?
  • What exactly is the reason we put ReaderT in Control.Monad.Linear but not in the internal modules?
  • What is the rule by which we decided to put Consumable and Dupable and Movable in Data.Unrestricted? With the exception of the last, these seem like separate concepts.
• Cohesion
  • Internal modules should have clear boundaries that ideally are reflected in their name. Internal doesn't communicate what is internal. I should have some idea of what ought to be in this module and what ought not to be -- this way, if I see it imported somewhere I know basically what is imported. This ideally comes from each module having a clearly defined and self contained purpose, usually about a singular concept.
  • Example: when I look at Data.Unrestricted.Linear and see the import Data.Functor.Linear.Internal I have no idea what this is importing. Is it importing all of Data.Functor.Linear essentially? It seems the purpose is to avoid a cycle of module dependencies but this still tells me very little. It seems like the original programmer has a clear idea of what that module is and what it does but as a new reader, I do not. This is a problem: there shouldn't be this silent knowledge or "internal wisdom" that makes it hard to do things unless you know all this background that isn't explicitly written down. [Cf., normalization of deviance.] If stuff like this grows, this becomes unsustainable.
  • Example: When I look at imports of Control.Monad.Linear.Internal I have no idea what they are really importing. I don't know the boundaries of what is and is not in this module, nor do I understand it's purpose.
• Consistency
  • The template by which we make internal modules should be consistent. If one part of the codebase just makes internal modules called Internal which just holds anything and everything core that requires no internal dependencies and other parts like Streaming have internal modules like Streaming.Type this is inconsistent.

I propose the following system and subsequent changes:

• System
  • We write this system down in the Design.md document.
  • Internal modules are not named 'Internal' and instead have the name of a single concept, which defines clear boundaries of what one can expect or not expect to be in them.
  • It's good to have a lot of internal modules if each of them is simple. Concrete example: it's good to have Data.Unrestricted.Consumable because it isolates a single self-contained concept, and if the consumable utilities grow, we don't want them to get messy. We also avoid a few module dependency issues this way.
  • Changes
  • We remove the internal Data.Functor and Control.Monad modules, establish Data.Functorial.Linear and Control.Functorial.Linear in which we re-export everything and make internal modules according to the system I've described above.
  • We divide up Data.Unrestricted.Linear
  • We make Data.SymmetricMonoidal.Linear
  • We do as we did with functors to Data.Profunctor.*
  • We make similar changes wherever else there are internal modules or ought to be because there are too many ideas in a file that aren't unified under a clear purpose or concept to the module.

I recognize this system is abstract but hopefully there are enough examples to give a clear enough idea of my proposed solution.

[aspiwack]

I don't agree with all of this.

• Internal module need to be named Internal one way or another (it doesn't need to be the final item in the module path, but it needs to be there) this does signal: this is not public interface, it exists purely for internal, technical reason, use at your own risk: it won't be considered in backwards compatibility. It's a convention that we can't do without in current Haskell.
• It may be worth it making smaller internal modules, though, since it would increase build parallelism, and conceivably makes the imports more predictable.
• I think that you are advocating for moving all the code into such internal modules and only use public module to re-export code from such internal modules: why not. I rather like the idea even.

• We remove the internal Data.Functor and Control.Monad modules, establish Data.Functorial.Linear and Control.Functorial.Linear in which we re-export everything and make internal modules according to the system I've described above.

  Data.Functor is nice: everything in there are functors. That's what I want this module to be named. But Control.Monad should become Control.Functor. If you need an internal name for “just the Functor type class” , you can use Data.Functor.Internal.Functor. A bit redundant, but very regular.

• We make Data.SymmetricMonoidal.Linear

  Likewise, it's Symmetric Monoidal products are in the right place in Data.Bifunctor.Linear, I don't know why we would want to change it. (again, making a dedicated internal module for it is ok)

• We do as we did with functors to Data.Profunctor.*

  I don't know what this means.

• We make similar changes wherever else there are internal modules or ought to be because there are too many ideas in a file that aren't unified under a clear purpose or concept to the module.

  I don't know what this means either.

[Divesh-Otwani]

I don't agree with all of this.

Correct me if I'm wrong, but I think this means what you don't explicitly disagree with you are neutral or in support of.

Internal module need to be named Internal one way or another (it doesn't need to be the final item in the module path, but it needs to be there) this does signal: this is not public interface, it exists purely for internal, technical reason, use at your own risk: it won't be considered in backwards compatibility. It's a convention that we can't do without in current Haskell.

I wasn't advocating for removing the Internal from the path of the name; just that naming the file Internal.hs communicates nothing unless we have a convention of what these Internal.hs files mean and what they stand for. And my main point is "internal modules should have clear boundaries" usually because they are centered around a concept.

It may be worth it making smaller internal modules, though, since it would increase build parallelism, and conceivably makes the imports more predictable.

I would emphasize that it's more than predictable: it makes the use of these readable, sustainable as the project grows, easier to read, etc.

I think that you are advocating for moving all the code into such internal modules and only use public module to re-export code from such internal modules: why not. I rather like the idea even.

That might be a consequence of what I"m saying but what I want is the opposite of my three complaints: non-arbitrariness, cohesion and consistency.

Data.Functor is nice: everything in there are functors. That's what I want this module to be named. But Control.Monad should become Control.Functor. If you need an internal name for “just the Functor type class” , you can use Data.Functor.Internal.Functor. A bit redundant, but very regular.

I don't think a new user will find it intuitive to see Applicative Functors in Data.Functor. If we are going to depart from the way base makes things in a lot of self-contained modules then we can't use the same names. We should call it Data.Functorial.Linear or something. Otherwise this will just be another strange thing a new user has to learn to use linear-base and I'd like as little of that as possible. More fundamentally, linear-base is not about making corrections to things we don't like in base -- we should do as little as possible of that and focus as much as possible on the linear part.

Likewise, it's Symmetric Monoidal products are in the right place in Data.Bifunctor.Linear, I don't know why we would want to change it. (again, making a dedicated internal module for it is ok)

Exactly the point I'm making about Data.Functor applies here. It just makes it harder for a new user who can't read the minds of the writers of linear-base.

  We do as we did with functors to `Data.Profunctor.*`     

I don't know what this means.

This means we put Data.Profunctor.{Profunctor, Wandering, Exchange} each in it's own internal file and have a Data.Kleisli.Linear. However, as I'm writing this I think I'll defer this to when I take a look at optics.

     We make similar changes wherever else there are internal modules or ought to be because there are
      too many ideas in a file that aren't unified under a clear purpose or concept to the module.

I don't know what this means either.

If there isn't a clear boundary or concept that defines an internal module then it should be broken up into more internal modules each of which have clear boundaries and are ideally centered around a concept.

[aspiwack]

I don't think a new user will find it intuitive to see Applicative Functors in Data.Functor. If we are going to depart from the way base makes things in a lot of self-contained modules then we can't use the same names. We should call it Data.Functorial.Linear or something. Otherwise this will just be another strange thing a new user has to learn to use linear-base and I'd like as little of that as possible.

This discussion is not going to be resolved by arguing. So let me just make this a decision: Functor stays.

More fundamentally, linear-base is not about making corrections to things we don't like in base -- we should do as little as possible of that and focus as much as possible on the linear part.

This is not true. There are plenty of corrections, in linear-base, to things in base that I believe haven't stood the test of time. Not everything, one has to choose their combat. But it's not the goal to make a strict base look-alike with different arrows.

[Divesh-Otwani]

Okay, so the things that remain are minimal enough that I think this issue can be closed. The module organization that I think could be made just a bit cleaner is:

• Prelude.Linear should just import things and have no definitions while Prelude.Linear.Internal should have "true prelude functions that belong in no other module, like id"
• The Profunctors could be made cleaner, less entangled with lenses and subdivided (but this makes more sense to tackle when we flush out lenses more)
• The Polarized arrays should perhaps have just 4 modules: Polarized: Internal.Polarized, Internal.Pull, and Internal.Push I'll make this another issue later.

Suffice it to say, all the big things are taken care of.

Also, as an "Author's note": some of the wording above is a bit too emotional and not as grounded it could have been. Let's just say 2020 was a rough year.