search

ZOSOpenTools / meta

Meta repository to tie together the various underlying z/OS Open Source tools (ZOT) repositories here
https://zosopentools.github.io/meta/
Apache License 2.0
38 stars 27 forks source link

Define ordering of how to 'bootstrap' from nothing for the z/OS open tools #1

Closed MikeFultonDev closed 1 year ago

MikeFultonDev commented 2 years ago

Tools have dependencies on other tools being present. Defining the order these tools need to be built in, and how, would make it easier for someone wanting to build from scratch

Since this is the 'meta' package, I think this is the natural place to put this information @IgorTodorovskiIBM @perry-ca @ccw-1 @AnthonyGiorgio

Order to build 'from scratch' (assumes 'port' tools have not been built yet) System Pre-reqs:

Both IBM and Rocket provide supported versions of the software above for a fee.

Taking the defaults will mean there are less variables for you to configure. We recommend you structure your sandbox as follows:

MikeFultonDev commented 2 years ago

would appreciate people's thoughts about the 'defaults' and the flow...

perry-ca commented 2 years ago

An important part of this is defining where the built tools should reside to be used by multiple users and how to handle multiple groups on a system needing multiple versions of tools. We could easily say the default location is the latest and greatest and if you depend on a back level that you need to maintain that yourself. I have noticed that on our machines (torolaba) that we have many versions of vi/vim frequently run into the problem of mixing parts of these together and crashing.

I think the boot/prod/dev dirs are kind of addressing this. These are handy for the tool builders. What about for the tool users? I think we should have one install dir for all of the packages so people know where to get things (i.e. add /home/zot/bin to PATH).

perry-ca commented 2 years ago

What if we add into each of these repos a dependency checking script? If the dependency is missing the script clones the missing package and builds it.

AnthonyGiorgio commented 2 years ago

This feels like we're reimplementing what Gentoo did with emerge and the portage tree. There was an entire build system that specified the order and versions of various packages. Each package had an "ebuild" file associated with it which described where to get it from, how to build it, what compiler flags to use, etc.

MikeFultonDev commented 2 years ago

@perry-ca for the tools that use jenkins and github, it would make sense to follow what perl is doing which is to push the binaries to a github tag. See: https://github.com/ZOSOpenTools/perlport/tags for the 'good enough' builds that we are promoting via jenkins CI/CD

MikeFultonDev commented 2 years ago

@perry-ca i like the idea that we can also have some sort of daemon that periodically updates the system with the 'latest' good version of each of the tools. Not sure how we do that without disrupting someone that might be doing work when the update happens...

MikeFultonDev commented 2 years ago

@AnthonyGiorgio i agree it would be good to not re-invent here. I hadn't seen anything explicitly written down for these tools other than text in the readme / install scripts. A lot of this software is so 'deep in the system' that tools just assume it's already there (e.g. make)

MikeFultonDev commented 2 years ago

@perry-ca i wonder if for the 'system wide' version of the tools, we could just use /usr/zot ? And then under that follow the same structure as for /usr/lpp/IBM, e.g. have the vrmf as part of the directory structure and people can pick the one they want to use? (I had /usr/lpp/zot, but i think lpp stands for licensed platform product, and this stuff isn't licensed...)

AnthonyGiorgio commented 2 years ago

It honestly feels like we're implementing a Linux-from-scratch system here, except we already have a running compiler and binutils.

MikeFultonDev commented 2 years ago

no binutils :) see: https://github.com/ZOSOpenTools/coreutilsport

IgorTodorovskiIBM commented 2 years ago

An important part of this is defining where the built tools should reside to be used by multiple users and how to handle multiple groups on a system needing multiple versions of tools. We could easily say the default location is the latest and greatest and if you depend on a back level that you need to maintain that yourself. I have noticed that on our machines (torolaba) that we have many versions of vi/vim frequently run into the problem of mixing parts of these together and crashing.

I think the boot/prod/dev dirs are kind of addressing this. These are handy for the tool builders. What about for the tool users? I think we should have one install dir for all of the packages so people know where to get things (i.e. add /home/zot/bin to PATH).

Another approach would could follow is from Debian, where the tools are installed into a defined location (e.g. /usr/bin) and the binary is suffixed with the version number. For example, /usr/bin/perl5.11.1. To choose the preferred perl, the system or the user would create a symbolic link, e.g.: perl -> perl5.11.1. Debian also provides a tool named update-alternatives that creates, removes, maintains and displays information about the symbolic links comprising the Debian alternatives system. 

We could also have stable (built from stable line)/unstable (built from main/master)/security registry for all the tools we maintain.

AnthonyGiorgio commented 2 years ago

update-alternatives is a handy tool for managing things like Java, where you might want multiple versions installed simultaneously. I've used it to great effect on various Linux distributions before.

IgorTodorovskiIBM commented 2 years ago

One thing we were discussing previously was defining a project template similar to here: https://github.com/IgorTodorovskiIBM/utils

{
  "name": "Project Name",
  "path": "path to project
  "deps": { "perlport", "makeport" }
  "envars": {
    "CC": "xlclang",
    "LINK": "xlclang++",
    "script": "setenv.sh"
  },
  "build": {
    "pre-build": "./configure",
    "build": "make",
    "test": "make check",
    "test-failures": "grep failures",
    "install": "make install"
  },
}

deps would represent the project dependencies and attempt to automatically pull them in (binaries or build from scratch, or use the default location. We could also add fields for prod vs dev or all). The rest of the fields are self-explanatory, but the idea is that this would allow a generic framework for patching, building a project, and testing with other dependencies.

AnthonyGiorgio commented 2 years ago

This sounds very similar to how Arch Linux works with its "ports" system. https://wiki.archlinux.org/title/Arch_Build_System

IgorTodorovskiIBM commented 2 years ago

This sounds very similar to how Arch Linux works with its "ports" system. https://wiki.archlinux.org/title/Arch_Build_System

Thanks, it does look very similar indeed! And the makepkg script appears to be completely shell-based so it may not be too bad to port: https://gitlab.archlinux.org/pacman/pacman/blob/master/scripts/makepkg.sh.in

ccw-1 commented 2 years ago

no binutils :) see: https://github.com/ZOSOpenTools/coreutilsport

binutils relies on something pre-existing binaries, like autoconf, automake .... etc.

MikeFultonDev commented 2 years ago

I'd like to 'clean up' the defaults for the developer side of the house, which I think is simpler than the 'production' install side. What do people think of the organization of:

$HOME:
 └─zot
   └─ dev
   └─ boot
   └─ prod

and of course the developer could use sym links to point wherever, but then we could look for the different tools we need? I'd like to 'clean up' perl, m4, make, automake, autoconf for some self-consistency, but I don't want to do that until we get agreement

MikeFultonDev commented 2 years ago

@AnthonyGiorgio @IgorTodorovskiIBM in addition to git and svn, we would need support to pull a tar ball (e.g. curl/wget) as part of the 'bootstrap' process

MikeFultonDev commented 2 years ago

Just to flesh out @IgorTodorovskiIBM 's example, would we want to also have 'defaults' for the template too? For 'm4' I am thinking we would have:

{
  "name": "m4port",
  "repotype": "tarball" {
    "source": "http://ftp.gnu.org/gnu/m4/",
    "release": "m4-1.4.19",
    "deps": { "m4boot", "curlboot", "makeboot" }
    "build": {
      "bootstrap": NULL, # no bootstrap step required
      "pre-build": "./configure", # configure system
      "build": "make", # build code
      "test": "make check", # test code
      "analyze-failures": "./analyze", # analyze failures
      "install": "make install" # push build to (local) prod
     },
  "repotype": "git" {
    "source": "https://github.com/autotools-mirror",
    "release": "branch-1.4",
    "deps": { "m4prod", "gitprod", "makeprod", "autoconfprod", "automakeprod" }
    "build": {
      "bootstrap": "./bootstrap", # bootstrap step
      "pre-build": "./configure", # configure system
      "build": "make", # build code
      "test": "make check", # test code
      "analyze-failures": "./analyze", # analyze failures
      "install": "make install" # push build to (local) prod
     }
  },
}

@AnthonyGiorgio can you provide an example of what this would look like with https://wiki.archlinux.org/title/Arch_Build_System ?

MikeFultonDev commented 2 years ago

@IgorTodorovskiIBM should CFLAGS be part of the template? What about config options?

AnthonyGiorgio commented 2 years ago

I've never used Arch before. I've just heard that it's the successor to Gentoo, which I used back in the day.

AnthonyGiorgio commented 2 years ago

We'll also need the ability to list patches that will be applied. Gentoo had this with ebuilds, and it was a convenient way to modify an official tarball release with distribution-specific changes.

MikeFultonDev commented 2 years ago

I added another sub-directory for logs... Now that I've been working a bit with m4 and make, if other people were also doing builds, we would clobber log files written to /tmp and I'm thinking it's probably cleaner to write them into the build sub-tree by default rather than a global area like /tmp. Thoughts? The 'log' directory could be finer grained - either in the 'dev' directory or even have one per port (e.g. in the m4port, makeport, ... directories)

perry-ca commented 2 years ago

For tmp dir I always have the following in my profile:

export TMPDIR=$HOME/tmp

in my profile. It saves blotting public space and running into space issues (either small /tmp or people filling /tmp). We can create a log dir but many tools, including xlc/xlclang, create temporary files in $TMPDIR and don't always clean up after themselves. Having a log dir isn't going to solve the problem.

The build scripts can also create a temp dir within $TMPDIR and then delete it after a successful build. That helps clean up after mess tools.

IgorTodorovskiIBM commented 2 years ago

I think a log or workspace directory would be useful. For perl, I let the user dictate where the log should be: https://github.com/ZOSOpenTools/perlport/blob/main/bin/perlbuild.sh#L38, but if not set, defaults to /tmp. We may want to create a default workspace dir or allow the user to override it with an environment variable. We could store logs, builds, temporary files, etc.

IgorTodorovskiIBM commented 2 years ago

@AnthonyGiorgio @IgorTodorovskiIBM in addition to git and svn, we would need support to pull a tar ball (e.g. curl/wget) as part of the 'bootstrap' process

Doesn't look like it supports tar format: https://man.archlinux.org/man/PKGBUILD.5#USING_VCS_SOURCES (fyi, there's an example at the bottom of this link).

IgorTodorovskiIBM commented 2 years ago

@IgorTodorovskiIBM should CFLAGS be part of the template? What about config options?

The way I envisioned it was the quoted value would represent the full command (including options): "pre-build": "./configure --shared --disable-feature", # configure system

Also, I think CFLAGS can be represented as an envar (similar to the envars I listed in my earlier example).

MikeFultonDev commented 2 years ago

@IgorTodorovskiIBM @perry-ca perhaps we combine the two ideas... I like a default that is user-specific for logging. I wonder if we should require that ZOTLOG_ROOT (or something) be set before using the tools and that we recommend it be scoped to the user (e.g. Sean's TMPDIR approach that is in $HOME) ? Alternately, we could default it to $HOME/zot/log. I want to ensure we don't have 2 people running a build and wiping out each other's files in tmp (which will cause failures because they won't have update to the other person's files in tmp)

perry-ca commented 2 years ago

I'm used to thinking of a log dir as a place for build logs & reports. I think it is good to have this too. We should be doing the local TMPDIR too for all of those scratch/temp files created during a build that the user never really needs or wants to look at. The log files and temp files should be stored in separate locations.

MikeFultonDev commented 2 years ago

@perry-ca i took a first crack at the log files that is pretty rudimentary - I just dump them into the clone'd directory. I like your thoughts around separate locations - feel free to jump in and improve: https://github.com/ZOSOpenTools/utils/blob/main/bin/build.sh

MikeFultonDev commented 2 years ago

I've incorporated the general ideas for this into the 'build.sh' script. It's a first take at this. I'd like to close off this issue unless people are opposed. We can open up new issues against the 'build.sh' - maybe in 'utils' repo? @perry-ca @AnthonyGiorgio @IgorTodorovskiIBM @ccw-1 you all had voiced opinions so would like your input before I close this off...

ccw-1 commented 2 years ago

We usually have a ._* in .gitignore and tmp directory set to TMPDIR=${reporoot}/._tmp

IgorTodorovskiIBM commented 2 years ago

We usually have a ._* in .gitignore and tmp directory set to TMPDIR=${reporoot}/._tmp

This would involve updating the community repo's .gitignore file. Would it make more sense to create it under port root directory (${PORT_ROOT} or ${PORT_ROOT}/logs) as opposed to the cloned dir${PORT_ROOT}/${dir}"?

ccw-1 commented 2 years ago

The reason I do it this way is that we work on multiple projects and multiple branches at the same time. The tmp files are kept in the repo itself for problem isolation and forensics. Mixing them together in one spot is a problem.

ccw-1 commented 2 years ago

Of course you can also do it locally without pushing your preference by echo ._\* > .git/info/exclude

MikeFultonDev commented 1 year ago

is this 'done' now @IgorTodorovskiIBM ?