search

sclorg / container-common-scripts

Apache License 2.0
20 stars 45 forks source link

Distgen does different stuff when called several times #154

Closed hhorak closed 3 months ago

hhorak commented 4 years ago

This is quite suspicious, that calling distgen few times in a row results in different output every time. I smell timestamps on files might be the cause here.

(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  DG     9.6/cccp.yml
  DG     9.6/root/usr/share/container-scripts/postgresql/README.md
  DG     9.6/root/usr/share/container-scripts/postgresql/common.sh
  DG     9.6/root/usr/bin/run-postgresql-slave
  DG     9.6/root/usr/share/container-scripts/postgresql/openshift-custom-recovery.conf.template
  DG     10/cccp.yml
  DG     10/root/usr/share/container-scripts/postgresql/README.md
  DG     10/root/usr/share/container-scripts/postgresql/common.sh
  DG     10/root/usr/bin/run-postgresql-slave
  DG     10/root/usr/share/container-scripts/postgresql/openshift-custom-recovery.conf.template
  DG     11/cccp.yml
  DG     11/root/usr/share/container-scripts/postgresql/README.md
  DG     11/root/usr/share/container-scripts/postgresql/common.sh
  DG     11/root/usr/bin/run-postgresql-slave
  DG     11/root/usr/share/container-scripts/postgresql/openshift-custom-recovery.conf.template
  CP     9.6/root/usr/libexec/fix-permissions
  CP     9.6/content_sets.yml
  CP     9.6/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template
  CP     9.6/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template
  CP     9.6/root/usr/share/container-scripts/postgresql/scl_enable
  CP     9.6/root/usr/bin/run-postgresql
  CP     9.6/root/usr/bin/run-postgresql-master
  CP     9.6/root/usr/bin/container-entrypoint
  CP     9.6/root/usr/bin/usage
  CP     9.6/root/usr/libexec/check-container
  CP     9.6/root/usr/share/container-scripts/postgresql/start/set_passwords.sh
  CP     9.6/s2i/bin/assemble
  CP     9.6/s2i/bin/usage
  CP     10/root/usr/libexec/fix-permissions
  CP     10/content_sets.yml
  CP     10/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template
  CP     10/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template
  CP     10/root/usr/share/container-scripts/postgresql/scl_enable
  CP     10/root/usr/bin/run-postgresql
  CP     10/root/usr/bin/run-postgresql-master
  CP     10/root/usr/bin/container-entrypoint
  CP     10/root/usr/bin/usage
  CP     10/root/usr/libexec/check-container
  CP     10/root/usr/share/container-scripts/postgresql/start/set_passwords.sh
  CP     10/s2i/bin/assemble
  CP     10/s2i/bin/usage
  CP     11/root/usr/libexec/fix-permissions
  CP     11/content_sets.yml
  CP     11/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template
  CP     11/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template
  CP     11/root/usr/share/container-scripts/postgresql/scl_enable
  CP     11/root/usr/bin/run-postgresql
  CP     11/root/usr/bin/run-postgresql-master
  CP     11/root/usr/bin/container-entrypoint
  CP     11/root/usr/bin/usage
  CP     11/root/usr/libexec/check-container
  CP     11/root/usr/share/container-scripts/postgresql/start/set_passwords.sh
  CP     11/s2i/bin/assemble
  CP     11/s2i/bin/usage
  LN     9.6/README.md
  LN     9.6/test
  LN     9.6/s2i/bin/run
  LN     10/README.md
  LN     10/test
  LN     10/s2i/bin/run
  LN     11/README.md
  LN     11/test
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
praiskup commented 4 years ago

GNU make by default doesn't check for symbolic link timestamps (see option -L that would help).

  1. We can make the output nicer, always executing the LN but not printing it to stdout to not confuse people. I'd suggest to do some "check" that the symlink points to where it should point to (read the link manually, and compare) and regenerate it only when needed.

  2. We can force make -L execution, but that has drawbacks, per docs:

    '-L'
'--check-symlink-times'
   On systems that support symbolic links, this option causes 'make'
   to consider the timestamps on any symbolic links in addition to the
   timestamp on the file referenced by those links.  When this option
   is provided, the most recent timestamp among the file and the
   symbolic links is taken as the modification time for this target
   file.

  3. We can backup each symlink by some other file (dot-file) in some working directory, and do the symlink creation only if the file itself has outdated timestamp.

phracek commented 3 months ago

The distgen was completely rewritten and we solve it by the latest updates. Also containers that use dist-gen had a test whether source are generated properly.

No issue were discovered in the meantime.