search

tern-tools / tern

Tern is a software composition analysis tool and Python library that generates a Software Bill of Materials for container images and Dockerfiles. The SBOM that Tern generates will give you a layer-by-layer view of what's inside your container in a variety of formats including human-readable, JSON, HTML, SPDX and more.
BSD 2-Clause "Simplified" License
967 stars 188 forks source link

Running more than one tern process in parallel in same instance results in errors (problem with $HOME/.tern/temp directory) #1117

Closed raghunathd8 closed 2 years ago

raghunathd8 commented 2 years ago

Describe the bug Running >=2 processes of tern is resulting in error for at least one process. One tern process is finishing and cleaning up temp directory while the other process (which is running in parallel) is also running and expecting that the temp directory is created and required directories/files are in place. This is ending up in various errors depending on the time parallel executions (overlapping periods).

To Reproduce Steps to reproduce the behavior:

  1. Open three terminals.
  2. In terminal-1 run this command: watch $HOME/.tern/temp
  3. In terminal-2 run this command: tern report -f json -i docker.io/library/alpine:latest -o tern-report.json
  4. In terminal-2 run this command: tern report -f json -i docker.io/library/alpine:3.14 -o tern-report1.json

Error in terminal

Terminal-2 output:
$ tern report -f json -i docker.io/library/alpine:latest -o tern-report.json
2022-02-04 12:35:35,977 - DEBUG - __main__ - Starting...
2022-02-04 12:35:35,978 - DEBUG - prep - Setting up...
2022-02-04 12:35:35,979 - DEBUG - rootfs - Running command: sudo chmod +x /usr/local/lib/python3.9/dist-packages/tern/tools/fs_hash.sh
2022-02-04 12:35:35,988 - DEBUG - run - Starting analysis...
2022-02-04 12:35:35,988 - DEBUG - skopeo - Attempting to pull image "docker.io/library/alpine:latest"
2022-02-04 12:35:35,988 - DEBUG - rootfs - Running command: skopeo copy docker://docker.io/library/alpine:latest dir:/home/node/.tern/temp
2022-02-04 12:35:39,637 - DEBUG - skopeo - Inspecting remote image "docker.io/library/alpine:latest"
2022-02-04 12:35:39,637 - DEBUG - rootfs - Running command: skopeo inspect docker://docker.io/library/alpine:latest
2022-02-04 12:35:43,771 - DEBUG - rootfs - Running command: tar -tf /home/node/.tern/temp/59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3
2022-02-04 12:35:43,820 - DEBUG - rootfs - Running command: tar -x -f /home/node/.tern/temp/59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 -C /home/node/.tern/temp/1/contents
2022-02-04 12:35:43,879 - DEBUG - rootfs - Running command: sudo /usr/local/lib/python3.9/dist-packages/tern/tools/fs_hash.sh /home/node/.tern/temp/1/contents
2022-02-04 12:35:44,051 - DEBUG - common - Reading files in filesystem...
2022-02-04 12:35:44,054 - DEBUG - rootfs - Running command: tar -tf /home/node/.tern/temp/59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3
2022-02-04 12:35:44,099 - DEBUG - rootfs - Running command: tar -x -f /home/node/.tern/temp/59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 -C /home/node/.tern/temp/mergedir
2022-02-04 12:35:44,159 - DEBUG - rootfs - Running command: sudo mknod /home/node/.tern/temp/mergedir/dev/urandom c 1 9
2022-02-04 12:35:44,166 - DEBUG - rootfs - Running command: sudo cp /etc/resolv.conf /home/node/.tern/temp/mergedir/etc/resolv.conf
2022-02-04 12:35:44,174 - DEBUG - core - Collecting metadata for image layer...
2022-02-04 12:35:44,174 - DEBUG - rootfs - Running command: sudo chroot /home/node/.tern/temp/mergedir /bin/sh -c pkgs=`apk info 2>/dev/null` && for p in $pkgs; do apk -a info $p 2>/dev/null | tail -2 | head -1; done
2022-02-04 12:35:44,251 - DEBUG - rootfs - Running command: sudo chroot /home/node/.tern/temp/mergedir /bin/sh -c apk info 2>/dev/null
2022-02-04 12:35:44,266 - DEBUG - rootfs - Running command: sudo chroot /home/node/.tern/temp/mergedir /bin/sh -c pkgs=`apk info 2>/dev/null` && for p in $pkgs; do apk info $p 2>/dev/null | head -5 | tail -1; done
2022-02-04 12:35:44,340 - DEBUG - rootfs - Running command: sudo chroot /home/node/.tern/temp/mergedir /bin/sh -c pkgs=`apk info 2>/dev/null` && for p in $pkgs; do files=`apk -L info $p 2>/dev/null`; for file in $files; do if [ -f $file ]; then echo $file; fi; done; echo LICF; done
2022-02-04 12:35:44,413 - DEBUG - rootfs - Running command: sudo chroot /home/node/.tern/temp/mergedir /bin/sh -c pkgs=`apk info 2>/dev/null` && for p in $pkgs; do lic=`apk info $p 2>/dev/null | head -1 | awk '{print $1}'`; echo $lic | sed -e "s/^$p-//"; done
2022-02-04 12:35:44,500 - ERROR - rootfs - Command failed. /bin/sh: sed: not found

2022-02-04 12:35:44,500 - WARNING - rootfs - Error executing command in chroot
2022-02-04 12:35:44,501 - WARNING - collect - Error executing snippets: Command 'pkgs=`apk info 2>/dev/null` && for p in $pkgs; do lic=`apk info $p 2>/dev/null | head -1 | awk '{print $1}'`; echo $lic | sed -e "s/^$p-//"; done' returned non-zero exit status 1.
2022-02-04 12:35:44,501 - ERROR - core - Script invocation error. Unable to collect some metadata
2022-02-04 12:35:44,501 - WARNING - core - Some metadata may be missing
2022-02-04 12:35:44,501 - WARNING - bundle - Inconsistent lengths for key: versions
2022-02-04 12:35:44,521 - DEBUG - rootfs - Running command: sudo rm -rf /home/node/.tern/temp/mergedir
2022-02-04 12:35:44,527 - DEBUG - rootfs - Running command: sudo rm -rf /home/node/.tern/temp/workdir
2022-02-04 12:35:44,535 - DEBUG - rootfs - Running command: sudo rm -rf /home/node/.tern/temp/1/contents
2022-02-04 12:35:44,552 - DEBUG - prep - Tearing down...
2022-02-04 12:35:44,559 - DEBUG - __main__ - Finished
Terminal-3 output:
$ tern report -f json -i docker.io/library/alpine:3.14 -o tern-report1.json
2022-02-04 12:35:36,514 - DEBUG - __main__ - Starting...
2022-02-04 12:35:36,514 - DEBUG - prep - Setting up...
2022-02-04 12:35:36,514 - DEBUG - rootfs - Running command: sudo chmod +x /usr/local/lib/python3.9/dist-packages/tern/tools/fs_hash.sh
2022-02-04 12:35:36,520 - DEBUG - run - Starting analysis...
2022-02-04 12:35:36,520 - DEBUG - skopeo - Attempting to pull image "docker.io/library/alpine:3.14"
2022-02-04 12:35:36,521 - DEBUG - rootfs - Running command: skopeo copy docker://docker.io/library/alpine:3.14 dir:/home/node/.tern/temp
2022-02-04 12:35:40,025 - DEBUG - skopeo - Inspecting remote image "docker.io/library/alpine:3.14"
2022-02-04 12:35:40,026 - DEBUG - rootfs - Running command: skopeo inspect docker://docker.io/library/alpine:3.14
2022-02-04 12:35:44,172 - DEBUG - rootfs - Running command: tar -tf /home/node/.tern/temp/97518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d
2022-02-04 12:35:44,227 - DEBUG - rootfs - Running command: tar -x -f /home/node/.tern/temp/97518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d -C /home/node/.tern/temp/1/contents
2022-02-04 12:35:44,301 - DEBUG - rootfs - Running command: sudo /usr/local/lib/python3.9/dist-packages/tern/tools/fs_hash.sh /home/node/.tern/temp/1/contents
2022-02-04 12:35:44,493 - DEBUG - common - Reading files in filesystem...
2022-02-04 12:35:44,506 - DEBUG - rootfs - Running command: tar -tf /home/node/.tern/temp/97518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d
2022-02-04 12:35:44,556 - DEBUG - rootfs - Running command: tar -x -f /home/node/.tern/temp/97518928ae5f3d52d4164b314a7e73654eb686ecd8aafa0b79acd980773a740d -C /home/node/.tern/temp/mergedir
Traceback (most recent call last):
  File "/usr/local/bin/tern", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/tern/__main__.py", line 286, in main
    do_main(args)
  File "/usr/local/lib/python3.9/dist-packages/tern/__main__.py", line 116, in do_main
    crun.execute_image(args)
  File "/usr/local/lib/python3.9/dist-packages/tern/analyze/default/container/run.py", line 80, in execute_image
    cimage.analyze(full_image, args)
  File "/usr/local/lib/python3.9/dist-packages/tern/analyze/default/container/image.py", line 81, in analyze
    default_analyze(image_obj, options)
  File "/usr/local/lib/python3.9/dist-packages/tern/analyze/default/container/image.py", line 67, in default_analyze
    prereqs = single_layer.analyze_first_layer(image_obj, master_list, options)
  File "/usr/local/lib/python3.9/dist-packages/tern/analyze/default/container/single_layer.py", line 158, in analyze_first_layer
    target_dir = prep_first_layer(image_obj.layers[0])
  File "/usr/local/lib/python3.9/dist-packages/tern/analyze/default/container/single_layer.py", line 104, in prep_first_layer
    rootfs.prep_rootfs(target)
  File "/usr/local/lib/python3.9/dist-packages/tern/utils/rootfs.py", line 172, in prep_rootfs
    os.mkdir(dev_path)

On Terminal-1 watch for temp directory behaviour. The behaviour is different based on when you trigger the parallel tern processes.

Expected behavior Scanning of more than one image "simultaneously" using Tern should be possible. I should be able to run many tern processes on the same instance in parallel (simultaneously).

Environment you are running Tern on Enter all that apply

rnjudge commented 2 years ago

Hi @raghunathd8 this is possible using the -wd PATH cli option.

In terminal one: tern report -f json -i docker.io/library/alpine:latest -o tern-report.json

In terminal two: tern -wd ~/temp1 report -f json -i alpine:3.14 -o tern-report1.json

Tern needs a working directory to perform its comparison operations. If one is not specified, it will default to using the same directory which is the error you're seeing. Note that the temporary working directory does not have to exist prior to running the Tern command (Tern will create it if it doens't exist) so this can be a random directory.