Mess in top level directory of project

[lithoxs]

Last commit ([gcc] Close jbruchon#178 : update GCC-IA16) floods top level directory with a plethora of directories used in the intermediate steps of building a compiler. The mail goal of the project is to write a kernel (and its userspace), not the building of compilers.

Look at FUZIX (https://github.com/EtchedPixels/FUZIX), with similar goals than ELKS. You will not find there a "bin" directory to contain binaries of any compiler. And this project uses several compilers: sdcc, cc65, gcc-6809, even ia16-gcc. The documentation gives information on how to get required tools, but getting them is your problem.

It is a gross mistake to require a specific compiler to be able to compile ELKS. Rather, the Documentation must offer a range of options, from precompiled packages to directions on where to get the latest compilers. The top level README should point to the easiest options.

I will appreciate a definition on this issue.

[jbruchon]

Reverted for now: https://github.com/jbruchon/elks/pull/189

[mfld-fr]

Look at BuildRoot (https://buildroot.org), that has exactly the same goal as ELKS, i.e. to build a kernel and a userspace, and where you will find more than a "bin" directory where all the dependencies and their configuration are managed in a consistent and very friendly way, by just selecting options in a menuconfig, including building the cross chain.

It is a big mistake to not put the build chain configuration under control, and to provide user only documented procedure for manual and error-prone installation, not a fully automated one.

And last but not least, it is not very elegant to state things are wrong and reverting without discussing. The "plethora" actually consists of 4 folders with very clear purpose. Just ask if you don't understand, I would explain more extensively that layout.

[jbruchon]

dev86 was never integrated into the project either; despite being required "next door" to build, it also requires installation on the host machine to function. I'd rather distribute whatever GCC we use as a separate repo in the same way as dev86 if possible; that way, if someone can install from a package, they can do that instead of using an in-tree build.

[lithoxs]

The goals of BuildRoot are compiling the linux kernel and userspace. They don't develop the kernel and any user application. And the reason for the need of the buildroot is that every version of the kernel need a certain versions of the userspace and cross-compiler, which in turn needs certain versions of tools to get built.

On the other hand, in www.kernel.org you find many parts of the kernel and userspace. Nowhere are directories to be used by the building process of gcc.

Elks don't make special demands on the compiler (other than the cpu target). Then, elks don't need anything as buildroot.

I agree with mfld on the point that the 4 folders have a very clear purpose, and no futher explanation of their function is needed. I disagree with the very existence of those folders. However, I am open to hear your arguments.

[mfld-fr]

Outcome of the discussion in #166 is that, to be able to compile ELKS userland with GCC-IA16, we need a new (or ported) C library, that is tightly coupled to both the compiler and the kernel.

The current dev86 library depends on both the dev86 tools and some ELKS files (headers and syscall). ELKS depends on the dev86 library and tools. To solve that chicken & egg problem, dev86 either duplicates part of ELKS, or uses a Makefile variable to point to ELKS. There are also other circular dependencies (boot blocks, implicit layout ...) that make the current layout a real "mess", not easily maintainable.

Another point is that the cross compiling options to tell GCC the layout of the build chain (including the target C library) are set at configuration time, so when building GCC.

Building the compiler, the C library and ELKS with the same set of scripts and at the same place is the simplest way to manage the dependencies and to automate the whole. Keeping a kind of "dev-ia16" project (including the C library) synchronized with ELKS would consume more time.

[mfld-fr]

The example of www.kernel.org is not relevant. Building the compiler, the kernel and the userland for the final user is performed not by the kernel guys, but by the distros guys, who ensure consistency between the sourced components. And building the kernel correctly is not an easy thing in term of configuration management : see https://www.kernel.org/doc/html/v4.11/process/changes.html.

ELKS is somewhat a kind of distro, as you assemble a kernel and a userland to produce a bootable image, targetting the final user. And at the heart of any distro, there is a build and, many times, a packaging system.

So either you consider ELKS as a component to be sourced by another image build system, up to that later to perform the configuration management and to produce the final image, or you consider ELKS as a ready to use, self-contained and friendly distro. I do prefer the second approach.

About the dev86 package: in a previous discussion, Jody suggested to take ownership of that project (because not maintained any more in mainstream), and I objected that would be a huge task to update the references in all the distros, because it was still needed for Xen and QEMU.

I recently saw in my Gentoo rolling release that it is not required any more for QEMU through the VgaBIOS package (replaced by SeaVgaBIOS). Only remains Xen as the last user, and I am afraid not for a long time. ELKS user will have to build dev86 himself when the package will be definitively out, as today if he wants the latest version from Jody's fork. So adding the automated build of dev86 would also be a good thing, in addition to GCC-IA16, for a user friendly project.

About the ELKS special demand on the compiler: I don't agree the argument. It is still today tied to dev86. The best evidence is that we have to rewrite all the assembly code to move from AS86 to Gnu AS, and remove the optimization tricks specific to BCC to let GCC optimizer to fully do its job.

[jbruchon]

I have always had a major issue with how Dev86 is attached to ELKS. The fact that it has to be installed on the building system AND sit fully built beside ELKS for ELKS to build is ridiculous. I have also been thinking about the way that the "C library" components are connected; there is a huge amount of waste because there's no shared library system and the kernel and programs all have their own individual copies of the code. That's why I was trying to create that "BusyELKS" monstrosity: one fat binary would greatly increase code sharing without requiring shared libraries.

If we're really keen on adding ia16-gcc to the ELKS source tree, it should be self-contained in elks/ia16-gcc or something similar and a sane default build should be invoked by build.sh and the Makefiles for all other ELKS code should accept CC= LD= CFLAGS= etc. so that the compiler can be used by the build system directly without requiring host installation of the compiler. That would be completely acceptable. I don't mind tying ELKS to a compiler (it's already tied to Dev86 stuff even when non-bcc compilers are invoked) as long as that compiler can be distributed with ELKS in a somewhat isolated manner and built with build.sh so newbies have a shallow barrier to entry.

There is no doubt that GCC is better than bcc and GCC is certainly the direction we should move towards. As long as build tool overrides are available, other compilers can be tried in the future.

Regarding ELKS vs. Buildroot, ELKS is the same thing except the individual program sources are not separate packages that get downloaded and built individually. ELKS is self-contained except for Dev86 and if it weren't for the fact that Dev86 is both an established package on many distributions and used to be properly maintained by someone else until a year or two ago I'd have just shoved it into the ELKS source tree directly.

As far as getting GCC to use whatever C library we want, couldn't we use -nostdinc et cetera and have our build system specify the new standard library and header search locations based on the build configuration?

[lithoxs]

So far, there is not any argument to place the directories for building gcc at the top level of the project. Not even a bad argument.

When I say that ELKS don't make special demands to the compiler, I refer only to the C files. And the tricks to optimize in BCC don't affect GCC abilities for optimization: as everybody knows, GCC ignores the "register" directive and use of integers as pointers do not preclude the optimization as integers because those pointers are never dereferenced. When I say that ELKS don't make special demands to the compiler, I mean that it's not necessary an specific range of versions of the compiler, nonstandard features or a certain version of the language. It suffices to implement plain C. So, no need for a special build for it.

The link between Makefiles and kernel headers is needed only for building userspace. And this link is related with the C library. The FUZIX people mantains their own C library, but don't need any special build procedure for compilers. For the ELKS case matters are even simpler: ia16-elf-gcc compiler is unable to build ELKS applications (yet).

[mfld-fr]

I see... I will then follow Jody's advice as given in his latest post (second paragraph), i.e. insulate the new compiler in a single folder, with no more staging folder outside. But I keep the build of the new compiler and the conversion tool inside ELKS, because I still consider user should has nothing else to do than selecting options and run the build.sh script to get a ready to use image, especially when no package exists for that new compiler.

As I already started to work on the ELKS own C library, I will place that folder at ELKS top (as the link to dev86), because the new compiler is now used for both the kernel, that new library, and the userland.

[lithoxs]

It is false that no package exists for the new compiler. In the following link you can download a fully functional binary package at no cost, for the platform of your choice, for the ia16-elf-gcc compiler. So far, this is also the only ia16-elf-gcc compiler known to be able to produce a working ELKS kernel.

https://sourcery.mentor.com/GNUToolchain/release3298

By the way, any standard binary BCC linux package can be used to succesfully build a working ELKS system. No need to be the latest version, at least as old as 0.16.17 suffices, may be even older.

[mfld-fr]

The Mentor compiler and the one from https://github.com/crtc-demos/gcc-ia16 are actually the same. Andrew Jenner of Mentor (@reenigne) is also the owner of GitHub "crtc-demos" group. The compiler version I selected in my script is the Mentor / Jenner one, plus the far pointer extension currently developed by @tkchia. It is then able to produce a working ELKS kernel. And I intend to use the far pointer extension for next features, starting with the new ELKS C library and TMPFS.

It is indeed true to say there is no package available from any distro (at least the ones tracked by Repology), and installing binaries out of the distro channel can lead to dependency problems for tailored systems, such my Gentoo one. I avoid such problems by rebuilding from sources to get a consistent & optimal platform.

Moreover, chaining from build.sh will be a simpler workflow for the user than manually downloading and installing the Mentor binary distribution, will have a smaller footprint and won't be limited in time (Mentor support ends on 1st april 2018).

About the BCC packages that are still available today in the distros, I already explained that since they are obsolete, non maintained and not needed anymore, it is probable they will be dropped in a near future. We can only rely on Jody's fork today, so it looks logical to source and build from that fork.

[lithoxs]

Of course I know that the @tkchia compiler is a fork of the Mentor compiler. The point is that a binary package exists, is freely available and works for most users.

Now, we don't know if the fork by @tkchia has modifications that hampers the ability of that compiler to build the ELKS kernel. This has yet to be demonstrated.

Regarding BCC, the point is that binary packages exists and any of them perfectly able to build an ELKS system, despite its obsolescence. And nobody can anticipate when an obsolete, unneeded, unmantained package will be dropped. We all know numerous packages like that in current linux distributions.

[lithoxs]

Regarding the end of support by Mentor on Apr, 2018, it is fair to explain that there is no support by Mentor if you simply download the compiler. Exactly the same as if you install from @tkchia or crtc-demos. If you want support from Mentor, you have to pay for it. After Apr, 2018, the software will still be there for download, without support. So, if you need support, the Mentor path is your only option. But hurry.

[jbruchon]

Ideally I would like to package the necessary ia16-gcc source code as either part of ELKS or as a fork that I maintain independently here. I could provide binary releases that are wget downloaded through build.sh instead of building from source for people who don't want to wait on a lengthy GCC build to finish.

I think that the nature of ELKS and the unofficial nature and unproven future maintenance of all working ELKS compiler implementations suggests that we should be keeping compiler code bases known to work for building ELKS "close by."

[lithoxs]

I agree with all that.

[jbruchon]

What is the URL for the working ia16-gcc code base that I can clone or at least download and stuff into a repo?

Edit: I pulled the tar.bz2 from the link provided above; is that one OK?

[lithoxs]

The Mentor site (https://sourcery.mentor.com/GNUToolchain/release3298) has a source package. It seems that the above source comes from (https://github.com/crtc-demos/gcc-ia16) This is a fork from crtc-demos (https://github.com/tkchia/gcc-ia16)

[jbruchon]

Ah, this code is far too big to distribute along with ELKS, but I'm going to push it over to https://github.com/jbruchon/ia16-gcc and track the tkchia code so we don't have to worry so much about it vanishing in the future.

[mfld-fr]

Would be better if you fork from https://github.com/tkchia/gcc-ia16 rather than creating a new project tree, and if you keep the same name (gcc-ia16) as @reenigne and @tkchia...

[jbruchon]

I'll keep the same name but I'm not forking. Github has, in the past, deleted forks when the main repo is deleted. I'd rather my repo not be marked as a Github fork and then get nuked if something goes wrong.

[mfld-fr]

So it means that if @lkundrak drops dev86 main, we would lost your dev86 fork ?

[jbruchon]

Possibly...at least until I re-uploaded it as an independent repository. I want to do that anyway; lkundrak hasn't done anything on Dev86 for a pretty long while now.

[tkchia]

Hello all,

Regarding my gcc-ia16 fork: I have just registered for an account on GNU Savannah; I will see if I can set up a mirror/backup of gcc-ia16 on it. I might also see if I can distribute gcc-ia16 in the form of patches on the official GCC 6 3.0 (which it is based on). I will let you know if it goes well.

Thank you!

[jbruchon]

Hey @tkchia, that's great! I appreciate the info. Thanks for your hard work!

[tkchia]

Hello all,

I have prepared patch files for the "stable" commits of my gcc-ia16 project fork, and uploaded them onto the releases page. The patches are against GNU's official GCC 6.3.0 source release (which will hopefully continued to be mirrored at lots of places). Thank you!

[mfld-fr]

Thank you @tkchia ! Opened the issue https://github.com/jbruchon/gcc-ia16/issues/1 to pick up this latest 2017-12-10 release.