search

jsoftware / jsource

J engine source mirror
Other
657 stars 90 forks source link

make3: Toward a friendlier build system #12

Open xelxebar opened 4 years ago

xelxebar commented 4 years ago

Overview

This PR is part of a larger goal to make J easier to package and share across OSes and Linux distributions.

Currently, the make and make2 build systems provide a particularly non-standard build process. They work well under their intended environments but provide significant munging to work as part of other build systems. This is due to the builds being largely managed by an ad-hoc collection of scripts rather than relying on standard tooling.

My hope is that, at the very least, this initiates discussion toward making J buildable and distributable in a more robust and friendly way.

Project Goal

Anecdotally, the current build system(s) provide significant challenges to users simply wishing to try out J on their system. The overall goal is to provide a build that uses the familiar and build-system-friendly make and, if needed, autotools:

$ ./configure
$ make
$ make install

The collection of commits in this PR shares my initial progress in this regard.

Specific Pain Points

In particular, the following is a collection of issues I have had to work around in order to package J for the Void Linux distribution:

1) Sparse documentation

The build process is specific to J and highly non-standard. However, the documentation introducing said process is limited to overview.txt and make{,2}/make.txt which are both quite sparse. The existing documentation works well under a limited set of build conditions, but there is little information, either within the source tree or online, regarding how to make the build fit your requirements. This is mostly a case of "the code is the documentation."

2) Fragile logic for setting CFLAGS,

Currently, CFLAGS is set with -Werror and a collection of -W flags that depends on whether the compiler is thought to be gcc or not. The gcc-detection scheme relies on ad hoc path parsing and symlink resolution which still breaks when using tools such as ccache. It is possible to short-circuit the detection to your preferred compiler; however, doing so is very non-trivial for someone not familiar with the build scripts. Furthermore, assuming the compiler to be only gcc or clang imposes an unexpected and artificial restriction that is decoupled from where such requirements may actually originate.

3) Fails to respect CFLAGS, LDFLAGS etc. of environment,

When compiling as part of a larger build chain, it is standard to have CFLAGS populated with options you wish all your compiles to pick up. However, the build scripts in make and make2 clobber these environment variables. This is unexpected bevahivour and requires careful modification of the scripts to rectify.

4) Non-friendly "install",

By default, the build system simply builds. There is no equivalent of a make install. Implicitly, the assumption is that the user will either run jconsole directly from the build output directory or copy these to some location, preserving the directory structure. This violates the expectations of the Filesystem Hierarchy Standard (FHS) for installed software on Unices and Linux distributions.

It is possible to manually pull apart the pieces and install J in a way that follows the FHS; however, this requires editing profile.ijs and providing (undocumented) flags to jconsole. This seems like an unreasonably high barrier to entry for someone simply wishing to offer J as a package for their favorite Linux distribution.

5) Intractible use of shell scripting, and

The build scripts, while functional, are overall in need of a lot of TLC. There are deeply-nested if statements, giant case blocks with many repeated elements, and a lot of custom path-munging via cd and pwd. These techniques make the inherent logic within the scripts quite difficult to follow and fail to leverage standard shell facilities that, ostensibly, would perform the same job more concisely and robustly.

6) Undocumented build options.

As far as I can tell, apart from the standard build environment variables the following allow the user to toggle various J-specific build options:

However, the only place these are documented is within thes scripts themselves. Are these inteneded as undocumented/experimental options? If they are inteded to be user-facing, it would make sense to keep track of this kind of flag within make.txt at the very least.

What this PR does

I hope the above makes a case that the build system can be improved. I am willing to lead the charge in that endeavor if upstream is on board.

At the moment, the collection of build scripts here do little to change the above pain points, instead opting for a gradual and incremental evolution of make2 into the inteded target. The HEAD of this PR contains a new make3 directory, which is an exact copy of make2 at the start of this branch (commit cb4d852).

Within make3 I have only refactored build_jconsole.sh and build_libj.sh into code that I hope is "better". I personally feel it's more grokkable, and it reduces the SLOC count to about half of the originals.

I made an effort to ensure the changes are completely transparent by confirming identical hashes of (a tar of) the build products. That said, I have only confirmed this for linux and raspberry (under all j64x targets). My plan is to do the same for android, windows and darwin if this idea gains traction.

Notes

The make3 build should Just Work™ in exactly the same way as make2; however, as per the comments regarding scripting style, I opted to make one change to how build options are passed. Previously, each option was only enabled if the associated variable was set to "1". The scripts now enable each option if they are set to anything at all (other than the empty string). USE_LINENOISE is the only exception, since it defaults to enabled. You can disable as previously it by setting said variable to anything other than "1".

Any other "API" differences are unintended.

Prologue

If you made it this far, thank you for reading through this long PR explanation! If any of the above comes across as overly brash, please accept my apologies. My head has been deep in the code and associated frustrations for a while.

I look forward to hearing your feedback.

lemenkov commented 4 years ago

It seems that meson build-system is getting momentum. It is faster, much easy to read than autotools/cmake, allows cross-platform builds, offers configurability with build options, etc.

Maybe better to switch to it instead?

xelxebar commented 4 years ago

Peter Lemenkov notifications@github.com wrote:

It seems that meson build-system is getting momentum. It is faster, much easy to read than autotools/cmake, allows cross-platform builds, offers configurability with build options, etc.

Maybe better to switch to it instead?

Thank you for the interest and suggestion.

Personally, I am wary of introducing any extra tooling unless there is a strong case for its need. With any build system, the benefits need to be weighed against the drawbacks. Do we really want to ask upstream J developers to learn a whole new build system? Meson replaces make with ninja, so any users wanting to compile J need ninja installed. Is that a reasonable tradeoff? Meson solves a few specific bulid-related problems, but is jsource bumping up against those problems? There is a plethora of options when it comes to build tooling, why choose Meson, specifically?

Those are pretty big questions, especially since we currently only rely on bog standard tooling. My bias in this project is a tad entish. Unless there are concrete reasons to make a major change, it seems a bit hasty to overhaul something that works. In this PR, I do try to make a case that motivates my overhaul of the scripts. Whether or not that's a strong and relevant case is for upstream to decide.

Also, it's important to remember that J isn't run like a community-driven project. Upstream likely has concerns and requirements that are not obvious at first glance. One reason I submitted this PR was to provide a concrete demonstration of my idea so the J developers could have half a chance of weighing it against those concerns of I have no knowledge of. If you feel strongly about the Meson idea, then a similar approach might add valuable material to this discussion.

Anyway, thank you again for the interest.

zhihaoy commented 4 years ago

I think CMake is the only reasonable option. As a cross-platform build system generator, it has builtin support in Visual Studio, which proves its acceptance.

See also https://www.rojtberg.net/1481/do-not-use-meson/

moon-chilled commented 4 years ago

CMake is an abomination, and should be avoided at all costs.

Makefiles and scripts aren't inherently bad; they just need to be made friendlier.

zhihaoy commented 4 years ago

Makefiles and scripts are an abomination, and should be avoided at all costs.

CMake isn't inherently bad; it just needs to be made friendlier.

eiverson commented 3 years ago

Somehow I/we missed your excellent J source PR#12.

My apologies. Is getting back to you after 18 months too ridiculous?

If you are still interested in J and cleaning up our make mess, I would love to talk with you.

As you noted, Jsoftware's relationship to github/PR is close to zero. We want to change that. We are probably going to switch from GPL to MIT or ISC. And become more encouraging and responsive to PRs. But it won't happen overnight.

I loved the bit I found under your reddit handle: Wait no, those perturbative oblate spheroid people are even moreso jackasses. The Earth is a dynamical system of Techtonic plates flowing over a fluid mantle.

I am on unfamiliar ground here and am not sure this email will get to you. Our guy who knows more about this stuff is away right now so I am flying blind.

Would love to hear back from you (regular email).

On Fri, Feb 14, 2020 at 11:32 PM xelxebar @.***> wrote:

Overview

This PR is part of a larger goal to make J easier to package and share across OSes and Linux distributions.

Currently, the make and make2 build systems provide a particularly non-standard build process. They work well under their intended environments but provide significant munging to work as part of other build systems. This is due to the builds being largely managed by an ad-hoc collection of scripts rather than relying on standard tooling.

My hope is that, at the very least, this initiates discussion toward making J buildable and distributable in a more robust and friendly way. Project Goal

Anecdotally, the current build system(s) provide significant challenges to users simply wishing to try out J on their system. The overall goal is to provide a build that uses the familiar and build-system-friendly make and, if needed, autotools:

$ ./configure

$ make

$ make install

The collection of commits in this PR shares my initial progress in this regard. Specific Pain Points

In particular, the following is a collection of issues I have had to work around in order to package J for the Void Linux distribution:

  1. Sparse documentation

The build process is specific to J and highly non-standard. However, the documentation introducing said process is limited to overview.txt and make{,2}/make.txt which are both quite sparse. The existing documentation works well under a limited set of build conditions, but there is little information, either within the source tree or online, regarding how to make the build fit your requirements. This is mostly a case of "the code is the documentation."

  1. Fragile logic for setting CFLAGS,

Currently, CFLAGS is set with -Werror and a collection of -W flags that depends on whether the compiler is thought to be gcc or not. The gcc-detection scheme relies on ad hoc path parsing and symlink resolution which still breaks when using tools such as ccache. It is possible to short-circuit the detection to your preferred compiler; however, doing so is very non-trivial for someone not familiar with the build scripts. Furthermore, assuming the compiler to be only gcc or clang imposes an unexpected and artificial restriction that is decoupled from where such requirements may actually originate.

  1. Fails to respect CFLAGS, LDFLAGS etc. of environment,

When compiling as part of a larger build chain, it is standard to have CFLAGS populated with options you wish all your compiles to pick up. However, the build scripts in make and make2 clobber these environment variables. This is unexpected bevahivour and requires careful modification of the scripts to rectify.

  1. Non-friendly "install",

By default, the build system simply builds. There is no equivalent of a make install. Implicitly, the assumption is that the user will either run jconsole directly from the build output directory or copy these to some location, preserving the directory structure. This violates the expectations of the Filesystem Hierarchy Standard (FHS) for installed software on Unices and Linux distributions.

It is possible to manually pull apart the pieces and install J in a way that follows the FHS; however, this requires editing profile.ijs and providing (undocumented) flags to jconsole. This seems like an unreasonably high barrier to entry for someone simply wishing to offer J as a package for their favorite Linux distribution.

  1. Intractible use of shell scripting, and

The build scripts, while functional, are overall in need of a lot of TLC. There are deeply-nested if statements, giant case blocks with many repeated elements, and a lot of custom path-munging via cd and pwd. These techniques make the inherent logic within the scripts quite difficult to follow and fail to leverage standard shell facilities that, ostensibly, would perform the same job more concisely and robustly.

  1. Undocumented build options.

As far as I can tell, apart from the standard build environment variables the following allow the user to toggle various J-specific build options:

  • USE_OPENMP,
  • USE_LINENOISE,
  • USE_THREAD,
  • VERBOSELOG, and
  • jolecom (?).

However, the only place these are documented is within thes scripts themselves. Are these inteneded as undocumented/experimental options? If they are inteded to be user-facing, it would make sense to keep track of this kind of flag within make.txt at the very least. What this PR does

I hope the above makes a case that the build system can be improved. I am willing to lead the charge in that endeavor if upstream is on board.

At the moment, the collection of build scripts here do little to change the above pain points, instead opting for a gradual and incremental evolution of make2 into the inteded target. The HEAD of this PR contains a new make3 directory, which is an exact copy of make2 at the start of this branch (commit cb4d852 https://github.com/jsoftware/jsource/commit/cb4d852788eaadd8066e228ba59c89fcca07eb2e ).

Within make3 I have only refactored build_jconsole.sh and build_libj.sh into code that I hope is "better". I personally feel it's more grokkable, and it reduces the SLOC count to about half of the originals.

I made an effort to ensure the changes are completely transparent by confirming identical hashes of (a tar of) the build products. That said, I have only confirmed this for linux and raspberry (under all j64x targets). My plan is to do the same for android, windows and darwin if this idea gains traction. Notes

The make3 build should Just Work™ in exactly the same way as make2; however, as per the comments regarding scripting style, I opted to make one change to how build options are passed. Previously, each option was only enabled if the associated variable was set to "1". The scripts now enable each option if they are set to anything at all (other than the empty string). USE_LINENOISE is the only exception, since it defaults to enabled. You can disable as previously it by setting said variable to anything other than "1".

Any other "API" differences are unintended. Prologue

If you made it this far, thank you for reading through this long PR explanation! If any of the above comes across as overly brash, please accept my apologies. My head has been deep in the code and associated frustrations for a while.

I look forward to hearing your feedback.

You can view, comment on, or merge this pull request online at:

https://github.com/jsoftware/jsource/pull/12 Commit Summary

  • Start new make3 build directory
  • make3:libj,jconsole: Print all exported variables
  • make3:libj,jconsole: Replace ... subshells with $(...)
  • make3:libj,jconsole: Double quote parameters
  • make3:libj,jconsole: Replace cd path hopping with explicit path variables
  • make3:libj,jconsole: Document GNUC_MINOR as unused
  • make3:libj,jconsole: Set shell "strict mode"
  • make3:jconsole: Disentagle control flow logic
  • make3:libj: Refactor logic into orthogonal units

File Changes

Patch Links:

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jsoftware/jsource/pull/12?email_source=notifications&email_token=AEBRCHSZHRJVP4ZJAFOS46LRC5V5ZA5CNFSM4KVU4PO2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INX2TDA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBRCHTLIM5Q6DY54D5KQVDRC5V5ZANCNFSM4KVU4POQ .

xelxebar commented 3 years ago

eiverson @.***> wrote:

Somehow I/we missed your excellent J source PR#12.

My apologies. Is getting back to you after 18 months too ridiculous?

Hello Eric,

Thank you for the ping. Actually, I should probably be the one issuing apologies!

IIRC, you responded at the time when I notified Jsource about the PR. Email addresses are different, but I am one and the same person:

http://www.jsoftware.com/pipermail/source/2020-February/001299.html

If you are still interested in J and cleaning up our make mess, I would love to talk with you.

I am definitely still interested! The intervening 18 months haven't changed my intentions, but they have ended up changing my circumstances (new job, getting married, building a house, etc.), so I very much left you all hanging. Sorry about that!

In fact, a past me also started some thread expressing concern over JE's conspicuous lack of contributors:

http://www.jsoftware.com/pipermail/programming/2021-February/057487.html

Hery quickly offered to help me get involved. I am still very interested in this too!

As you noted, Jsoftware's relationship to github/PR is close to zero. We want to change that. We are probably going to switch from GPL to MIT or ISC. And become more encouraging and responsive to PRs. But it won't happen overnight.

Very interesting. I am obviously unfamiliar with the constraints and internal motivations that Jsoftware is dealing with, but I might be able to offer a bit of experience with current norms in OSS dev communities, FWIW.

I loved the bit I found under your reddit handle: Wait no, those perturbative oblate spheroid people are even moreso jackasses. The Earth is a dynamical system of Techtonic plates flowing over a fluid mantle.

Haha. I had completely forgotten about that conversation. I am also slightly mortified that such little coprolites from a past me are lingering around, so easily excavated.

I am on unfamiliar ground here and am not sure this email will get to you. Our guy who knows more about this stuff is away right now so I am flying blind.

Would love to hear back from you (regular email).

Thank you again for expressing interest in that massive PR dump.

I am just on the last stretch of my house build, so I (naively?) expect and hope that I will be able to settle into some semblance of a life-rhythm in the near future. At that point I very much hope to get back to J (and math).

On a more specific note, I recall you mentioning that you and the core team are scattered between Canada, the U.S., and Hong Kong. Do you all directly collaborate much? I badly want to hang out with and collaborate with some hard-core array programmers, but my timezone in Japan makes attending JLUG and similar meetups extremely difficult.

Anyway, if you have any questions, thoughts, or just wish to chat, I always get a jolt of joy seeing a message from you in my inbox.

Cheers, Brandon

On Fri, Feb 14, 2020 at 11:32 PM xelxebar @.***> wrote:

Overview

This PR is part of a larger goal to make J easier to package and share across OSes and Linux distributions.

Currently, the make and make2 build systems provide a particularly non-standard build process. They work well under their intended environments but provide significant munging to work as part of other build systems. This is due to the builds being largely managed by an ad-hoc collection of scripts rather than relying on standard tooling.

My hope is that, at the very least, this initiates discussion toward making J buildable and distributable in a more robust and friendly way. Project Goal

Anecdotally, the current build system(s) provide significant challenges to users simply wishing to try out J on their system. The overall goal is to provide a build that uses the familiar and build-system-friendly make and, if needed, autotools:

$ ./configure

$ make

$ make install

The collection of commits in this PR shares my initial progress in this regard. Specific Pain Points

In particular, the following is a collection of issues I have had to work around in order to package J for the Void Linux distribution:

  1. Sparse documentation

The build process is specific to J and highly non-standard. However, the documentation introducing said process is limited to overview.txt and make{,2}/make.txt which are both quite sparse. The existing documentation works well under a limited set of build conditions, but there is little information, either within the source tree or online, regarding how to make the build fit your requirements. This is mostly a case of "the code is the documentation."

  1. Fragile logic for setting CFLAGS,

Currently, CFLAGS is set with -Werror and a collection of -W flags that depends on whether the compiler is thought to be gcc or not. The gcc-detection scheme relies on ad hoc path parsing and symlink resolution which still breaks when using tools such as ccache. It is possible to short-circuit the detection to your preferred compiler; however, doing so is very non-trivial for someone not familiar with the build scripts. Furthermore, assuming the compiler to be only gcc or clang imposes an unexpected and artificial restriction that is decoupled from where such requirements may actually originate.

  1. Fails to respect CFLAGS, LDFLAGS etc. of environment,

When compiling as part of a larger build chain, it is standard to have CFLAGS populated with options you wish all your compiles to pick up. However, the build scripts in make and make2 clobber these environment variables. This is unexpected bevahivour and requires careful modification of the scripts to rectify.

  1. Non-friendly "install",

By default, the build system simply builds. There is no equivalent of a make install. Implicitly, the assumption is that the user will either run jconsole directly from the build output directory or copy these to some location, preserving the directory structure. This violates the expectations of the Filesystem Hierarchy Standard (FHS) for installed software on Unices and Linux distributions.

It is possible to manually pull apart the pieces and install J in a way that follows the FHS; however, this requires editing profile.ijs and providing (undocumented) flags to jconsole. This seems like an unreasonably high barrier to entry for someone simply wishing to offer J as a package for their favorite Linux distribution.

  1. Intractible use of shell scripting, and

The build scripts, while functional, are overall in need of a lot of TLC. There are deeply-nested if statements, giant case blocks with many repeated elements, and a lot of custom path-munging via cd and pwd. These techniques make the inherent logic within the scripts quite difficult to follow and fail to leverage standard shell facilities that, ostensibly, would perform the same job more concisely and robustly.

  1. Undocumented build options.

As far as I can tell, apart from the standard build environment variables the following allow the user to toggle various J-specific build options:

  • USE_OPENMP,
  • USE_LINENOISE,
  • USE_THREAD,
  • VERBOSELOG, and
  • jolecom (?).

However, the only place these are documented is within thes scripts themselves. Are these inteneded as undocumented/experimental options? If they are inteded to be user-facing, it would make sense to keep track of this kind of flag within make.txt at the very least. What this PR does

I hope the above makes a case that the build system can be improved. I am willing to lead the charge in that endeavor if upstream is on board.

At the moment, the collection of build scripts here do little to change the above pain points, instead opting for a gradual and incremental evolution of make2 into the inteded target. The HEAD of this PR contains a new make3 directory, which is an exact copy of make2 at the start of this branch (commit cb4d852 https://github.com/jsoftware/jsource/commit/cb4d852788eaadd8066e228ba59c89fcca07eb2e ).

Within make3 I have only refactored build_jconsole.sh and build_libj.sh into code that I hope is "better". I personally feel it's more grokkable, and it reduces the SLOC count to about half of the originals.

I made an effort to ensure the changes are completely transparent by confirming identical hashes of (a tar of) the build products. That said, I have only confirmed this for linux and raspberry (under all j64x targets). My plan is to do the same for android, windows and darwin if this idea gains traction. Notes

The make3 build should Just Work™ in exactly the same way as make2; however, as per the comments regarding scripting style, I opted to make one change to how build options are passed. Previously, each option was only enabled if the associated variable was set to "1". The scripts now enable each option if they are set to anything at all (other than the empty string). USE_LINENOISE is the only exception, since it defaults to enabled. You can disable as previously it by setting said variable to anything other than "1".

Any other "API" differences are unintended. Prologue

If you made it this far, thank you for reading through this long PR explanation! If any of the above comes across as overly brash, please accept my apologies. My head has been deep in the code and associated frustrations for a while.

I look forward to hearing your feedback.

You can view, comment on, or merge this pull request online at:

https://github.com/jsoftware/jsource/pull/12 Commit Summary

  • Start new make3 build directory
  • make3:libj,jconsole: Print all exported variables
  • make3:libj,jconsole: Replace ... subshells with $(...)
  • make3:libj,jconsole: Double quote parameters
  • make3:libj,jconsole: Replace cd path hopping with explicit path variables
  • make3:libj,jconsole: Document GNUC_MINOR as unused
  • make3:libj,jconsole: Set shell "strict mode"
  • make3:jconsole: Disentagle control flow logic
  • make3:libj: Refactor logic into orthogonal units

File Changes

Patch Links:

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jsoftware/jsource/pull/12?email_source=notifications&email_token=AEBRCHSZHRJVP4ZJAFOS46LRC5V5ZA5CNFSM4KVU4PO2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INX2TDA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBRCHTLIM5Q6DY54D5KQVDRC5V5ZANCNFSM4KVU4POQ .

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/jsoftware/jsource/pull/12#issuecomment-889204234

moon-chilled commented 2 years ago

@xelxebar you may find my build scripts (modifications to make2) helpful in constructing make3. I broke arm, and never got around to fixing it; but the core logic is largely factored, and hence much easier to change and understand.

xelxebar commented 2 years ago

@moon-chilled Cool. It looks like some of your factoring went in a similar direction as my original (now clobbered) commits. Thanks for sharing. I might have a closer look later.