doc: Add octodiff-docker.sh

Krinkle commented 1 year ago

Currently:

PuppetDB Username: krinkle
PuppetDB Password: 
The octocatalog-diff command is now ready for use!

root@ac4aaaa39725:/infrastructure-puppet# octocatalog-diff -n codeorigin-02.stage.ops.jquery.net
#<Thread:0x00007f03a4a00218 /usr/lib/ruby/3.1.0/open3.rb:296 run> terminated with exception
    (report_on_exception is true):
/usr/lib/ruby/3.1.0/open3.rb:296:in `read': stream closed in another thread (IOError)
    from /usr/lib/ruby/3.1.0/open3.rb:296:in `block (2 levels) in capture3'
#<Thread:0x00007f03a4a00088 /usr/lib/ruby/3.1.0/open3.rb:297 run> terminated with exception
    (report_on_exception is true):
/usr/lib/ruby/3.1.0/open3.rb:297:in `read': stream closed in another thread (IOError)
    from /usr/lib/ruby/3.1.0/open3.rb:297:in `block (2 levels) in capture3'
/usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-util/bootstrap.rb:150:in `run_bootstrap':
    bootstrap failed for /tmp/ocd-ipc-20230402-3978-ovvf8x/ocd-bootstrap-checkout-20230402-3979-fu3b95:
        /tmp/ocd-ipc-20230402-3978-ovvf8x/ocd-bootstrap-checkout-20230402-3979-fu3b95
        /test_data/bootstrap.sh: line 4: g10k: command not found (OctocatalogDiff::Errors::BootstrapError)

    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-util/bootstrap.rb:96:in `bootstrap_directory'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog/computed.rb:96:in `bootstrap'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog/computed.rb:113:in `build_catalog'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog.rb:96:in `build'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/catalogs.rb:241:in `build_catalog'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:34:in `call'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:34:in `execute'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:197:in `execute_task'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:119:in `block (2 levels) in run_tasks_parallel'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:117:in `fork'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:117:in `block in run_tasks_parallel'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:114:in `each'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:114:in `each_with_index'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:114:in `run_tasks_parallel'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/parallel.rb:94:in `run_tasks'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/catalogs.rb:92:in `build_catalog_parallelizer'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/util/catalogs.rb:29:in `catalogs'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/api/v1/catalog-diff.rb:34:in `catalog_diff'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/api/v1.rb:19:in `catalog_diff'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/cli.rb:152:in `run_octocatalog_diff'
    from /usr/lib/ruby/vendor_ruby/octocatalog-diff/cli.rb:126:in `cli'
    from /usr/bin/octocatalog-diff:34:in `<main>'

supertassu commented 1 year ago

Looks like we're missing g10k from the container and the list of required packages.

Krinkle commented 1 year ago

@supertassu It's getting a bit further but still doesn't appear to work correctly. Tried three different ways, got three different errors. 😬

root@78f186ba3de3:/infrastructure-puppet# octocatalog-diff -n codeorigin-02.stage.ops.jquery.net
W, [2023-04-11T05:18:41.621540 #4340]  WARN -- : Puppet command failed: STDOUT:
STDERR:
  Error: Evaluation Error: Error while evaluating a Resource Statement, Could not find declared class apt (file: /tmp/ocd-ipc-20230411-4338-j618bx/ocd-builddir-20230411-4340-emubnx/environments/staging/modules/profile/manifests/base/apt.pp, line: 5, column: 3) on node codeorigin-02.stage.ops.jquery.net
  Error: Evaluation Error: Error while evaluating a Resource Statement, Could not find declared class apt (file: /tmp/ocd-ipc-20230411-4338-j618bx/ocd-builddir-20230411-4340-emubnx/environments/staging/modules/profile/manifests/base/apt.pp, line: 5, column: 3) on node codeorigin-02.stage.ops.jquery.net

root@78f186ba3de3:/infrastructure-puppet# octocatalog-diff --environment production -n puppet-03.ops.jquery.net
W, [2023-04-11T05:19:03.317895 #4401]  WARN -- : Puppet command failed: STDOUT:
STDERR:
  Error: Execution of config_version command `/tmp/ocd-ipc-20230411-4399-wh9gzl/ocd-builddir-20230411-4401-z7y1pl/environments/production/bin/config-version.sh production` failed: Execution of '/tmp/ocd-ipc-20230411-4399-wh9gzl/ocd-builddir-20230411-4401-z7y1pl/environments/production/bin/config-version.sh production' returned 128: fatal: ambiguous argument 'production': unknown revision or path not in the working tree.
  Use '--' to separate paths from revisions, like this:
  'git <command> [<revision>...] -- [<file>...]' on node puppet-03.ops.jquery.net
  Error: Execution of config_version command `/tmp/ocd-ipc-20230411-4399-wh9gzl/ocd-builddir-20230411-4401-z7y1pl/environments/production/bin/config-version.sh production` failed: Execution of '/tmp/ocd-ipc-20230411-4399-wh9gzl/ocd-builddir-20230411-4401-z7y1pl/environments/production/bin/config-version.sh production' returned 128: fatal: ambiguous argument 'production': unknown revision or path not in the working tree.

root@78f186ba3de3:/infrastructure-puppet# octocatalog-diff -f production -t staging -n search-02.ops.jquery.net
E, [2023-04-11T05:19:23.846424 #4452] ERROR -- : Git checkout error: Git archive production->/tmp/ocd-ipc-20230411-4451-adf01/ocd-bootstrap-checkout-20230411-4452-n8ljxt failed: STDOUT:
STDERR:
  fatal: not a valid object name: production
  tar: This does not look like a tar archive
  tar: Exiting with failure status due to previous errors
#<Thread:0x00007f2e188e74c8 /usr/lib/ruby/3.1.0/open3.rb:296 run> terminated with exception (report_on_exception is true):
/usr/lib/ruby/3.1.0/open3.rb:296:in `read': stream closed in another thread (IOError)
    from /usr/lib/ruby/3.1.0/open3.rb:296:in `block (2 levels) in capture3'
#<Thread:0x00007f2e188e7360 /usr/lib/ruby/3.1.0/open3.rb:297 run> terminated with exception (report_on_exception is true):
/usr/lib/ruby/3.1.0/open3.rb:297:in `read': stream closed in another thread (IOError)
    from /usr/lib/ruby/3.1.0/open3.rb:297:in `block (2 levels) in capture3'
/usr/lib/ruby/vendor_ruby/octocatalog-diff/catalog-util/bootstrap.rb:109:in `rescue in git_checkout': Git archive production->/tmp/ocd-ipc-20230411-4451-adf01/ocd-bootstrap-checkout-20230411-4452-n8ljxt failed: STDOUT: (OctocatalogDiff::Errors::BootstrapError)
STDERR:
  fatal: not a valid object name: production
  tar: This does not look like a tar archive
  tar: Exiting with failure status due to previous errors

supertassu commented 1 year ago

As long as you run g10k -puppetfile beforehand (to download the vendor_modules/ directory) this seems to roughly work as I'd expect:

PuppetDB Username: taavi
PuppetDB Password:
The octocatalog-diff command is now ready for use!

root@e337d92b3c25:/infrastructure-puppet# git diff
diff --git a/modules/profile/manifests/typesense.pp b/modules/profile/manifests/typesense.pp
index d40f8ff..be0d77c 100644
--- a/modules/profile/manifests/typesense.pp
+++ b/modules/profile/manifests/typesense.pp
@@ -12,6 +12,11 @@ class profile::typesense (
     dport => 443,
   }

+  file { '/etc/test':
+    ensure  => file,
+    content => 'test',
+  }
+
   $backend_port = 8108
   $tls_config = nginx::tls_config()

root@e337d92b3c25:/infrastructure-puppet# git status
On branch octodiff-docker
Your branch and 'origin/octodiff-docker' have diverged,
and have 135 and 1 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   modules/profile/manifests/typesense.pp

no changes added to commit (use "git add" and/or "git commit -a")
root@e337d92b3c25:/infrastructure-puppet# octocatalog-diff -n search-02.ops.jquery.net
I, [2023-08-04T16:31:22.087298 #3978]  INFO -- : Catalogs compiled for search-02.ops.jquery.net
I, [2023-08-04T16:31:22.117871 #3978]  INFO -- : Diffs computed for search-02.ops.jquery.net
I, [2023-08-04T16:31:22.118046 #3978]  INFO -- : Note: you can use --display-detail-add to view details of added resources
+ File[/etc/test]
*******************************************

Krinkle commented 1 year ago

@supertassu Thanks, that got me a step further.

After that, there were three more issues:

vendor_modules/ not writable by g10k (I mount the git repo as read-only).
/tmp and and git repo are on different file systems, wheraas g10k requires hard links between cachedir and vendordir.
when using --environment production,git log as executed by bin/config-version.sh fails due to ambigious "production" argument.

I fixed those by ensuring the directories exist ahead of time, and mounting them separately overtop (can't let docker lazy-create the vendor_modules directory because creating a new directory isn't allowed in a readonly dir).

Merged in https://github.com/jquery/infrastructure-puppet/commit/864f5844b2e01e0a76102a02c1a8af9bd8b36dd4.

jquery / infrastructure-puppet

doc: Add octodiff-docker.sh #3