search

crashappsec / chalk

Chalk allows you to follow code from development, through builds and into production.
https://crashoverride.com/
GNU General Public License v3.0
348 stars 15 forks source link

chalk marks and reports may lack git keys when data is packed #342

Closed ee7 closed 3 months ago

ee7 commented 3 months ago

Example

#!/bin/sh
set +e

tmpdir='/tmp/chalk-git-pack'
rm -rf "${tmpdir}"
mkdir -p "${tmpdir}"
cd "${tmpdir}" || exit

git init
git config user.email "foo@example.com" # In case the user deliberately doesn't set these globally.
git config user.name "Foo Bar"
cp /path/to/chalk ./chalk # To be explicit about the chalk version used.

echo 'hi' > hi.txt
git add hi.txt
git commit -m 'initial commit'
./chalk insert hi.txt
git gc
./chalk insert hi.txt

Expected behavior

The chalk report from each chalk insert command contains git key/value pairs, like:

    "_AUTHOR": "Foo Bar <foo@example.com> 1718368178 +0200",
    "_BRANCH": "main",
    "_COMMITTER": "Foo Bar <foo@example.com> 1718368178 +0200",
    "_COMMIT_ID": "854fe875bc3a2e11d55bc829a592aef8dbe189b9",
    "_COMMIT_MESSAGE": "initial commit",
    "_COMMIT_SIGNED": false,
    "_DATE_AUTHORED": "Fri Jun 14 14:29:38 2024 +0200",
    "_DATE_COMMITTED": "Fri Jun 14 14:29:38 2024 +0200",

Observed behavior

The output after git gc may lack those keys, which points to a general issue. And we see the warning:

warn:  /tmp/chalk-git-pack/.git/refs/heads/main: Git ref file for 'main' doesnt exist. Most likely it's an empty git repo.

Diagnosis

Chalk should handle packfiles better.

Related: https://github.com/crashappsec/chalk/issues/332

miki725 commented 3 months ago

whats the full output with --trace?

ee7 commented 3 months ago

Yeah, I should've mentioned that - sorry. The main thing is the warning:

warn:  /tmp/chalk-git-pack/.git/refs/heads/main: Git ref file for 'main' doesnt exist. Most likely it's an empty git repo.

but for the first chalk insert (using a chalk built from either main, or https://github.com/crashappsec/chalk/commit/53b708656826a8302d078bb6efcbd64fa2bc5c95 on PR #341):

trace: Loaded sink config for 'default_out'
trace: Loaded sink config for 'console_out'
trace: Loaded sink config for 'virtual_chalk_log'
trace: Installed plugin: docker
trace: Installed plugin: elf
trace: Installed plugin: macos
trace: Installed plugin: python_pyc
trace: Installed plugin: source
trace: Installed plugin: zip
trace: Installed plugin: zippeditem
trace: Installed plugin: elf_last_resort
trace: Installed plugin: ci_github
trace: Installed plugin: ci_gitlab
trace: Installed plugin: ci_jenkins
trace: Installed plugin: conffile
trace: Installed plugin: aws_ecs
trace: Installed plugin: aws_lambda
trace: Installed plugin: tool
trace: Installed plugin: cloud_metadata
trace: Installed plugin: authors
trace: Installed plugin: github_codeowners
trace: Installed plugin: procfs
trace: Installed plugin: system
trace: Installed plugin: metsys
trace: Installed plugin: vctl_git
trace: Installed plugin: tech_stack_generic
trace: Checking chalk binary '/tmp/chalk-git-pack/chalk' for embedded config
trace: Codec elf: beginning scan of /tmp/chalk-git-pack/chalk
trace: /tmp/chalk-git-pack/chalk: scanning file
trace: /tmp/chalk-git-pack/chalk: chalk mark magic @ 10874261
trace: No embedded chalk mark.
trace: Using the default user config.  See 'chalk dump' to view.
trace: Looking for config file at: /etc/chalk/chalk.c4m
trace: /etc/chalk/chalk.c4m: No configuration file found.
trace: Looking for config file at: /etc/chalk.c4m
trace: /etc/chalk.c4m: No configuration file found.
trace: Looking for config file at: /tmp/chalk-git-pack/chalk.c4m
trace: /tmp/chalk-git-pack/chalk.c4m: No configuration file found.
trace: Looking for config file at: /home/foo/.config/chalk/chalk.c4m
trace: /home/foo/.config/chalk/chalk.c4m: No configuration file found.
trace: Looking for config file at: /home/foo/chalk.c4m
trace: /home/foo/chalk.c4m: No configuration file found.
trace: Running subscan. Command name is temporarily: extract
trace: Subscan location: @["/home/foo/.local/share/bash_completion/completions/chalk.bash"]
trace: *** Setting log-level = "error" for scan.  Use --debug to turn on
trace: Subscan done. Restored command name to: insert
info:  Invalid chalk mark in autocompletion script. Updating.
info:  Installed bash auto-completion file to: /home/foo/.local/share/bash_completion/completions/chalk.bash
trace: Collecting host-level chalk-time data
trace: Collecting chalk time artifact info
trace: Running plugin: system
trace: Running plugin: ci_jenkins
trace: Running plugin: ci_gitlab
trace: Running plugin: tool
trace: Running plugin: ci_github
trace: Running plugin: aws_ecs
trace: ecs: metadata env var is not defined: no AWS info available
trace: Running plugin: conffile
trace: Called artifacts() -- filepaths = @["/tmp/chalk-git-pack/hi.txt"]; otherPaths = @[]
trace: Asking codec 'python_pyc' to scan artifacts.
trace: Codec python_pyc: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: /tmp/chalk-git-pack/hi.txt: ignored due to matching ignore pattern: .*\.txt
trace: We will NOT report additional path skips.
trace: Asking codec 'zip' to scan artifacts.
trace: Codec zip: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'macos' to scan artifacts.
trace: Codec macos: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'elf' to scan artifacts.
trace: Codec elf: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'source' to scan artifacts.
trace: Codec source: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'elf_last_resort' to scan artifacts.
trace: Codec elf_last_resort: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Collecting runtime host info.
trace: Collecting run time host info
trace: Running plugin: system
trace: Running plugin: docker
trace: Searching PATH for docker
trace: Running subscan. Command name is temporarily: extract
trace: Subscan location: @["/usr/bin/docker"]
trace: *** Setting log-level = "error" for scan.  Use --debug to turn on
trace: Subscan done. Restored command name to: insert
trace: Found 'docker' in PATH: /usr/bin/docker
trace: docker: client version: 26.1.4
trace: docker: server version: 26.1.4
trace: docker: buildx version: 0.15.0
trace: Running plugin: tech_stack_generic
trace: Skipping tech stack runtime detection plugin
trace: Running plugin: vctl_git
trace: Looking for .git directory, from: /tmp/chalk-git-pack/hi.txt
trace: Found version control dir: /tmp/chalk-git-pack/.git
trace: branch: main
trace: commit ID: 854fe875bc3a2e11d55bc829a592aef8dbe189b9
trace: Running plugin: cloud_metadata
trace: Not an EC2 instance - skipping check for IMDSv2
trace: Running plugin: aws_ecs
trace: Running plugin: aws_lambda
trace: lambda: function env vars are not defined: no AWS info available
trace: Running plugin: conffile
trace: Running plugin: metsys
trace: Generating command report.
trace: /tmp/chalk-git-pack/chalk-reports.jsonl: file lock obtained, but no report cache to read.
info:  /home/foo/.local/chalk/chalk.log: Open (sink conf='default_out')
    filename        = "/home/foo/.local/chalk/chalk.log"
    log_search_path = ["/var/log/chalk/", "~/.log/chalk/", "."]
info:  Full chalk report appended to: ~/.local/chalk/chalk.log
trace: Published the report for topic 'report' (1 subscribers)
trace: Loaded sink config for 'json_console_out'
[
  {
    "_OPERATION": "insert",
    "_DATETIME": "2024-06-14T14:29:38.647+02:00",
    "_AUTHOR": "Foo Bar <foo@example.com> 1718368178 +0200",
    "_BRANCH": "main",
    "_COMMITTER": "Foo Bar <foo@example.com> 1718368178 +0200",
    "_COMMIT_ID": "854fe875bc3a2e11d55bc829a592aef8dbe189b9",
    "_COMMIT_MESSAGE": "initial commit",
    "_COMMIT_SIGNED": false,
    "_DATE_AUTHORED": "Fri Jun 14 14:29:38 2024 +0200",
    "_DATE_COMMITTED": "Fri Jun 14 14:29:38 2024 +0200",
    "_ENV": {
      "PWD": "/tmp/chalk-git-pack",
      "XDG_SESSION_TYPE": "wayland",
      "USER": "foo",
      "PATH": "/usr/local/sbin:/usr/local/bin:/usr/bin"
    },
    "_OP_ARGV": [
      "/tmp/chalk-git-pack/chalk",
      "insert",
      "--trace",
      "hi.txt"
    ],
    "_OP_CHALKER_VERSION": "0.4.5-dev",
    "_ORIGIN_URI": "local",
    "_OP_CHALK_COUNT": 0,
    "_OP_UNMARKED_COUNT": 0
  }
]
trace: Published the report for topic 'terminal_chalk_time' (1 subscribers)

and the second (after the git gc):

trace: Loaded sink config for 'default_out'
trace: Loaded sink config for 'console_out'
trace: Loaded sink config for 'virtual_chalk_log'
trace: Installed plugin: docker
trace: Installed plugin: elf
trace: Installed plugin: macos
trace: Installed plugin: python_pyc
trace: Installed plugin: source
trace: Installed plugin: zip
trace: Installed plugin: zippeditem
trace: Installed plugin: elf_last_resort
trace: Installed plugin: ci_github
trace: Installed plugin: ci_gitlab
trace: Installed plugin: ci_jenkins
trace: Installed plugin: conffile
trace: Installed plugin: aws_ecs
trace: Installed plugin: aws_lambda
trace: Installed plugin: tool
trace: Installed plugin: cloud_metadata
trace: Installed plugin: authors
trace: Installed plugin: github_codeowners
trace: Installed plugin: procfs
trace: Installed plugin: system
trace: Installed plugin: metsys
trace: Installed plugin: vctl_git
trace: Installed plugin: tech_stack_generic
trace: Checking chalk binary '/tmp/chalk-git-pack/chalk' for embedded config
trace: Codec elf: beginning scan of /tmp/chalk-git-pack/chalk
trace: /tmp/chalk-git-pack/chalk: scanning file
trace: /tmp/chalk-git-pack/chalk: chalk mark magic @ 10874261
trace: No embedded chalk mark.
trace: Using the default user config.  See 'chalk dump' to view.
trace: Looking for config file at: /etc/chalk/chalk.c4m
trace: /etc/chalk/chalk.c4m: No configuration file found.
trace: Looking for config file at: /etc/chalk.c4m
trace: /etc/chalk.c4m: No configuration file found.
trace: Looking for config file at: /tmp/chalk-git-pack/chalk.c4m
trace: /tmp/chalk-git-pack/chalk.c4m: No configuration file found.
trace: Looking for config file at: /home/foo/.config/chalk/chalk.c4m
trace: /home/foo/.config/chalk/chalk.c4m: No configuration file found.
trace: Looking for config file at: /home/foo/chalk.c4m
trace: /home/foo/chalk.c4m: No configuration file found.
trace: Running subscan. Command name is temporarily: extract
trace: Subscan location: @["/home/foo/.local/share/bash_completion/completions/chalk.bash"]
trace: *** Setting log-level = "error" for scan.  Use --debug to turn on
trace: Subscan done. Restored command name to: insert
info:  Invalid chalk mark in autocompletion script. Updating.
info:  Installed bash auto-completion file to: /home/foo/.local/share/bash_completion/completions/chalk.bash
trace: Collecting host-level chalk-time data
trace: Collecting chalk time artifact info
trace: Running plugin: system
trace: Running plugin: ci_jenkins
trace: Running plugin: ci_gitlab
trace: Running plugin: tool
trace: Running plugin: ci_github
trace: Running plugin: aws_ecs
trace: ecs: metadata env var is not defined: no AWS info available
trace: Running plugin: conffile
trace: Called artifacts() -- filepaths = @["/tmp/chalk-git-pack/hi.txt"]; otherPaths = @[]
trace: Asking codec 'python_pyc' to scan artifacts.
trace: Codec python_pyc: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: /tmp/chalk-git-pack/hi.txt: ignored due to matching ignore pattern: .*\.txt
trace: We will NOT report additional path skips.
trace: Asking codec 'zip' to scan artifacts.
trace: Codec zip: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'macos' to scan artifacts.
trace: Codec macos: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'elf' to scan artifacts.
trace: Codec elf: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'source' to scan artifacts.
trace: Codec source: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'elf_last_resort' to scan artifacts.
trace: Codec elf_last_resort: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Collecting runtime host info.
trace: Collecting run time host info
trace: Running plugin: system
trace: Running plugin: docker
trace: Searching PATH for docker
trace: Running subscan. Command name is temporarily: extract
trace: Subscan location: @["/usr/bin/docker"]
trace: *** Setting log-level = "error" for scan.  Use --debug to turn on
trace: Subscan done. Restored command name to: insert
trace: Found 'docker' in PATH: /usr/bin/docker
trace: docker: client version: 26.1.4
trace: docker: server version: 26.1.4
trace: docker: buildx version: 0.15.0
trace: Running plugin: tech_stack_generic
trace: Skipping tech stack runtime detection plugin
trace: Running plugin: vctl_git
trace: Looking for .git directory, from: /tmp/chalk-git-pack/hi.txt
trace: Found version control dir: /tmp/chalk-git-pack/.git
trace: branch: main
warn:  /tmp/chalk-git-pack/.git/refs/heads/main: Git ref file for 'main' doesnt exist. Most likely it's an empty git repo.
trace: Running plugin: cloud_metadata
trace: Not an EC2 instance - skipping check for IMDSv2
trace: Running plugin: aws_ecs
trace: Running plugin: aws_lambda
trace: lambda: function env vars are not defined: no AWS info available
trace: Running plugin: conffile
trace: Running plugin: metsys
trace: Generating command report.
trace: /tmp/chalk-git-pack/chalk-reports.jsonl: file lock obtained, but no report cache to read.
info:  /home/foo/.local/chalk/chalk.log: Open (sink conf='default_out')
    filename        = "/home/foo/.local/chalk/chalk.log"
    log_search_path = ["/var/log/chalk/", "~/.log/chalk/", "."]
info:  Full chalk report appended to: ~/.local/chalk/chalk.log
trace: Published the report for topic 'report' (1 subscribers)
trace: Loaded sink config for 'json_console_out'
[
  {
    "_OPERATION": "insert",
    "_DATETIME": "2024-06-14T14:29:40.704+02:00",
    "_ENV": {
      "PWD": "/tmp/chalk-git-pack",
      "XDG_SESSION_TYPE": "wayland",
      "USER": "foo",
      "PATH": "/usr/local/sbin:/usr/local/bin:/usr/bin"
    },
    "_OP_ARGV": [
      "/tmp/chalk-git-pack/chalk",
      "insert",
      "--trace",
      "hi.txt"
    ],
    "_OP_CHALKER_VERSION": "0.4.5-dev",
    "_OP_CHALK_COUNT": 0,
    "_OP_UNMARKED_COUNT": 0
  }
]
trace: Published the report for topic 'terminal_chalk_time' (1 subscribers)
miki725 commented 3 months ago

yeah git packs refs themeselves as well

warn: /tmp/chalk-git-pack/.git/refs/heads/main: Git ref file for 'main' doesnt exist. Most likely it's an empty git repo.