make frege build easier to bootstrap

[thkoch2001]

As mentioned in #170, debian (and other distros) have good reasons to build everything from source. This is difficult (but not impossible) if something depends on itself for being built.

Are there ways how Frege could become easier to build without Frege? Some Ideas:

• Frege is a Haskell dialect. Can Frege be built with Haskell?
• Frege compiles to Java source code. Can this Java Source code be shipped as a separate tarball? Than one just needs a JDK to compile the java code and have a Frege Stage 1 to build a fresh Frege from Frege source.
• Is it possible to build a version of frege with fewer features but powerful enough to build the full frege without frege? (I think not, but ...?)

[Ingo60]

I am afraid none of your points would really work, AFAICS:

• A dialect is not the same as a subset. In particular, the Frege FFI targets Java,and Haskell's does not. The compiler uses reflection to extract annotations from class files and to determine subtype relationships. So it is hard to see how a Haskell program could do it.
• The intermediate Java Source Code could indeed be shipped, however, how would this be different from shipping class files? That being said, creating such a tar-file would be a small issue. I just can't see how someone could trust that the Java sources in that tar are indeed the result of the compilation of the signed frege sources with an uncompromised compiler.
• A maintenance nightmare. And you would yet have to start out with a JAR, or Java Sources of that smaller compiler, so the problem is still the same.

How do other self hosting languages (Haskell, Scala?) do it?

[thkoch2001]

I've not yet worked on any compiled package in Debian. So I don't want to give false informations on how other debian packages solve this. However I believe that bootstrapping Frege from Java Source files is still beneficial.

• There is no circular dependency. The Frege build would depend on a JDK.
• The Java Source code could still be edited or patched if necessary. So the goal of being able to build the Debian archive from editable source is still somewhat kept.
• It's harder to hide malicious bits in generated Java source files than in generated bytecode.
• After building the Frege compiler we can test whether the compiled Frege compiler generates the same Java source files. It is possible but not trivial to cheat this test.
• The diff between versions of the Java source files can be inspected.

[Ingo60]

In some sense, yes:

• true, there is no dependency except for the JDK
• theoretically, yes
• well .... we are talking about 600.000 lines of highly unreadabe Java code. So yes. OTOH, class files can be decompiled, so there's not that big a difference.
• yes. though with parallel compilation, the compiler is somewhat nondeterministic. For example, an intermediate variable could be named foo_165 in one run, and foo_564 in another. A textual diff would probably find 100 thousands of differences between two identical (up to variable names) programs.
• see above. It wouldn't make sense, unless all sources are compiled sequentially in a certain order. And even this would not help after changes, because the java code for modules that import the changed one would often differ with respect to the variable names.

I am sorry this all sounds that terrible, It's just that I see no easy way, and don't want to hide the technical difficulties that I see.

I see the value, and appreciate it, in building everything from sources. But it is simply difficult with self-hosting compilers.

[seancorfield]

Also, I would anticipate that at some point Frege's compiler will move from generating Java to generating JVM bytecode, like most other compilers, so any thoughts of blessing the generated Java source code as somehow "official" in any way seems completely wrong-headed to me.

[ghost]

That might open a new issue. Is Frege able to produce reproducible builds? https://wiki.debian.org/ReproducibleBuilds/About

Debian aims to have byte for byte identical builds of its packages independently of the time or machine. Is the bytecode produced by Frege identical in two subsequent builds of the same source code?

Reproducability is not a must have to enter the debian archive but it's a good thing to aim for.

[Ingo60]

No, it's not.

For one, we have a time stamp of the time of compilation.

And then, as mentioned before, the compiler generates a lot of synthetic symbols (not just local variables) using a unique number generator. The state of that number generator is not deterministic in case of parallel builds. But those builds are the way to go if you want to compile some program like the compiler that consists of many modules, since it figures out the dependency order and thus which modules can be compiled when and which ones can be build concurrently.

To be sure, the generated programs are identical up to renamings of certan variables. But I guess that is not what is looked for.

[ForNeVeR]

It seems that guys from GHC will have to do some changes to implement deterministic builds: https://ghc.haskell.org/trac/ghc/ticket/4012

[mat8913]

Would it be possible to put a pre-built compiler in the source package? That's what the rustc Debian package does: https://sources.debian.net/src/rustc/1.6.0%2Bdfsg1-2/dl/

[Ingo60]

You mean the java files? Sure, but this would get fairly big. Surely not less than 100MB.

[mat8913]

Isn't https://github.com/Frege/frege/releases/download/3.23.288/frege3.23.422-ga05a487.jar all that's needed? It's less than 20MB.

[Ingo60]

1. The JAR is binary, but the requirement was to build from source code.
2. The compiler is still in development and self-hosting, this means there is no guarantee that it will be able to compile the latest sources. Hence, one would have to maintain a JAR and a set of sources from whence step by step and commit by commit the latest can be build. (But even this is sometimes impossible to do automatically.)

Now, option 2 is exactly what we have already: Access to all versions of the sources through git, and a recent enough compiler to compile HEAD in the releases area.