search

docker / build-push-action

GitHub Action to build and push Docker images with Buildx
https://github.com/marketplace/actions/build-and-push-docker-images
Apache License 2.0
4.13k stars 532 forks source link

Command substitution doesn't work via build-args #938

Closed jeankhawand closed 11 months ago

jeankhawand commented 11 months ago

Contributing guidelines

I've found a bug, and:

Description

tried on my local to run the following command and I can see it working as expected 

 docker buildx build --build-arg ARCH=$(uname -m)  --output dest=- ./build-docker-images/wazuh-manager

Expected behaviour

[+] Building 8.3s (20/21)                                                                                                      docker:desktop-linux
[+] Building 8.5s (20/21)                                                                                                      docker:desktop-linux
[+] Building 8.6s (20/21)                                                                                                      docker:desktop-linux
[+] Building 9.4s (20/21)                                                                                                      docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                           0.0s
 => => transferring dockerfile: 2.50kB                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/ubuntu:jammy                                                                                0.5s
 => CACHED https://raw.githubusercontent.com/wazuh/wazuh/4.5/extensions/elasticsearch/7.x/wazuh-template.json                                  0.0s
 => [internal] load build context                                                                                                              0.0s
 => => transferring context: 745B                                                                                                              0.0s
 => [ 1/15] FROM docker.io/library/ubuntu:jammy@sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77                        0.0s
 => CACHED [ 2/15] RUN rm /bin/sh && ln -s /bin/bash /bin/sh                                                                                   0.0s
 => CACHED [ 3/15] RUN apt-get update && apt install curl apt-transport-https lsb-release gnupg -y                                             0.0s
 => CACHED [ 4/15] RUN curl -L -O https://packages.wazuh.com/4.x/apt/pool/main/w/wazuh-manager/wazuh-manager_4.5.0-1_arm64.deb &&    dpkg -i   0.0s
 => CACHED [ 5/15] RUN curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.10.2-arm64.deb &&    dpkg -i filebeat-  0.0s
 => CACHED [ 6/15] RUN curl -L https://github.com/aelsabbahy/goss/releases/latest/download/goss-linux-arm64 -o /usr/local/bin/goss && chmod +  0.0s
 => [ 7/15] RUN if [ "arm64" = "arm64" ]; then ARCH="aarch64"; fi &&     curl --fail --silent -L "https://github.com/just-containers/s6-overl  1.3s
 => [ 8/15] COPY config/etc/ /etc/                                                                                                             0.0s
 => [ 9/15] COPY --chown=root:wazuh config/create_user.py /var/ossec/framework/scripts/create_user.py                                          0.0s
 => [10/15] COPY config/filebeat.yml /etc/filebeat/                                                                                            0.0s
 => [11/15] RUN chmod go-w /etc/filebeat/filebeat.yml                                                                                          0.1s
 => [12/15] ADD https://raw.githubusercontent.com/wazuh/wazuh/4.5/extensions/elasticsearch/7.x/wazuh-template.json /etc/filebeat               0.0s
 => [13/15] RUN chmod go-w /etc/filebeat/wazuh-template.json                                                                                   0.1s
 => [14/15] COPY config/permanent_data.env config/permanent_data.sh /                                                                          0.0s
 => [15/15] RUN chmod 755 /permanent_data.sh &&     sync && /permanent_data.sh &&     sync && rm /permanent_data.sh                            0.3s
 => exporting to client directory                                                                                                              6.8s
 => => copying files 226.40MB                                                                                                                  6.8s

Actual behaviour

using the action I don't see uname -m evaluted https://github.com/jeankhawand/wazuh-docker/actions/runs/5931795158/job/16084421367?pr=1

Repository URL

https://github.com/jeankhawand/wazuh-docker

Workflow run URL

https://github.com/jeankhawand/wazuh-docker/actions/runs/5931795158/job/16084421367?pr=1

YAML workflow

name: Wazuh Docker pipeline

on: [pull_request]

jobs:
  build-docker-images:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        platform:
          - linux/amd64
          - linux/arm64
          - linux/arm/v7
          - linux/ppc64le
          - linux/s390x
    steps:

    - name: Check out code
      uses: actions/checkout@v3

    - name: Create enviroment variables
      run: cat .env > $GITHUB_ENV
# https://docs.docker.com/build/ci/github-actions/multi-platform/
    - name: Setup QEMU
      uses: docker/setup-qemu-action@v2

    - name: Setup Docker Buildx
      uses: docker/setup-buildx-action@v2

    - name: Create images directory
      run: |
        mkdir -p /home/runner/work/wazuh-docker/wazuh-docker/docker-images/
        ls -lah
        pwd

    - name: Build Wazuh manager docker image
      uses: docker/build-push-action@v4
      with:
        context: ./build-docker-images/wazuh-manager
        platforms: ${{ matrix.platform }}
        push: false
        buildkitd-flags: --debug
        outputs: type=tar,dest=/home/runner/work/wazuh-docker/wazuh-docker/docker-images/wazuh-manager.tar
        build-args: |
          ARCH=$(uname -m)
        tags: |
          wazuh/wazuh-manager:${{env.WAZUH_IMAGE_VERSION}}

    - name: Build Wazuh index docker image
      uses: docker/build-push-action@v4
      with:
        context: ./build-docker-images/wazuh-indexer
        platforms: ${{ matrix.platform }}
        push: false
        buildkitd-flags: --debug
        outputs: type=tar,dest=/home/runner/work/wazuh-docker/wazuh-docker/docker-images/wazuh-indexer.tar
        build-args: |
          ARCH=$(uname -m)
        tags: |
          wazuh/wazuh-indexer:${{env.WAZUH_IMAGE_VERSION}}

    - name: Build Wazuh dashboard docker image
      uses: docker/build-push-action@v4
      with:
        context: ./build-docker-images/wazuh-dashboard
        platforms: ${{ matrix.platform }}
        push: false
        buildkitd-flags: --debug
        outputs: type=tar,dest=/home/runner/work/wazuh-docker/wazuh-docker/docker-images/wazuh-dashboard.tar
        build-args: |
          ARCH=$(uname -m)
        tags: |
          wazuh/wazuh-dashboard:${{env.WAZUH_IMAGE_VERSION}}

    - name: Temporarily save Wazuh manager Docker image
      uses: actions/upload-artifact@v3
      with:
        name: docker-artifact-manager
        path: /home/runner/work/wazuh-docker/wazuh-docker/docker-images/wazuh-manager.tar
        retention-days: 1

    - name: Temporarily save Wazuh indexer Docker image
      uses: actions/upload-artifact@v3
      with:
        name: docker-artifact-indexer
        path: /home/runner/work/wazuh-docker/wazuh-docker/docker-images/wazuh-indexer.tar
        retention-days: 1

    - name: Temporarily save Wazuh dashboard Docker image
      uses: actions/upload-artifact@v3
      with:
        name: docker-artifact-dashboard
        path: /home/runner/work/wazuh-docker/wazuh-docker/docker-images/wazuh-dashboard.tar
        retention-days: 1

    - name: Install Goss
      uses: e1himself/goss-installation-action@v1.0.3
      with:
        version: v0.3.16

    - name: Execute Goss tests (wazuh-manager)
      run: dgoss run wazuh/wazuh-manager:${{env.WAZUH_IMAGE_VERSION}}
      env:
        GOSS_SLEEP: 30
        GOSS_FILE: .github/.goss.yaml

  check-single-node:
    runs-on: ubuntu-latest
    needs: build-docker-images
    steps:

    - name: Check out code
      uses: actions/checkout@v3

    - name: Create enviroment variables
      run: cat .env > $GITHUB_ENV

    - name: Retrieve saved Wazuh indexer Docker image
      uses: actions/download-artifact@v3
      with:
        name: docker-artifact-indexer

    - name: Retrieve saved Wazuh manager Docker image
      uses: actions/download-artifact@v3
      with:
        name: docker-artifact-manager

    - name: Retrieve saved Wazuh dashboard Docker image
      uses: actions/download-artifact@v3
      with:
        name: docker-artifact-dashboard

    - name: Docker load
      run: |
        docker load --input ./wazuh-indexer.tar
        docker load --input ./wazuh-dashboard.tar
        docker load --input ./wazuh-manager.tar

    - name: Create single node certficates
      run: docker-compose -f single-node/generate-indexer-certs.yml run --rm generator

    - name: Start single node stack
      run: docker-compose -f single-node/docker-compose.yml up -d

    - name: Check Wazuh indexer start
      run: |
       sleep 60
       status_green="`curl -XGET "https://0.0.0.0:9200/_cluster/health" -u admin:SecretPassword -k -s | grep green | wc -l`"
       if [[ $status_green -eq 1 ]]; then
        curl -XGET "https://0.0.0.0:9200/_cluster/health" -u admin:SecretPassword -k -s
       else
        curl -XGET "https://0.0.0.0:9200/_cluster/health" -u admin:SecretPassword -k -s
        exit 1
       fi
       status_index="`curl -XGET "https://0.0.0.0:9200/_cat/indices" -u admin:SecretPassword -k -s | wc -l`"
       status_index_green="`curl -XGET "https://0.0.0.0:9200/_cat/indices" -u admin:SecretPassword -k -s | grep "green" | wc -l`"
       if [[ $status_index_green -eq $status_index ]]; then
        curl -XGET "https://0.0.0.0:9200/_cat/indices" -u admin:SecretPassword -k -s
       else
        curl -XGET "https://0.0.0.0:9200/_cat/indices" -u admin:SecretPassword -k -s
        exit 1
       fi

    - name: Check Wazuh indexer nodes
      run: |
       nodes="`curl -XGET "https://0.0.0.0:9200/_cat/nodes" -u admin:SecretPassword -k -s | grep -E "indexer" | wc -l`"
       if [[ $nodes -eq 1 ]]; then
        echo "Wazuh indexer nodes: ${nodes}"
       else
        echo "Wazuh indexer nodes: ${nodes}"
        exit 1
       fi

    - name: Check documents into wazuh-alerts index
      run: |
       docs="`curl -XGET "https://0.0.0.0:9200/wazuh-alerts*/_count" -u admin:SecretPassword -k -s | jq -r ".count"`"
       if [[ $docs -gt 100 ]]; then
        echo "wazuh-alerts index documents: ${docs}"
       else
        echo "wazuh-alerts index documents: ${docs}"
        exit 1
       fi

    - name: Check Wazuh templates
      run: |
       qty_templates="`curl -XGET "https://0.0.0.0:9200/_cat/templates" -u admin:SecretPassword -k -s | grep -P "wazuh|wazuh-agent|wazuh-statistics" | wc -l`"
       templates="`curl -XGET "https://0.0.0.0:9200/_cat/templates" -u admin:SecretPassword -k -s | grep -P "wazuh|wazuh-agent|wazuh-statistics"`"
       if [[ $qty_templates -eq 3 ]]; then
        echo "wazuh templates:"
        echo "${templates}"
       else
        echo "wazuh templates:"
        echo "${templates}"
        exit 1
       fi

    - name: Check Wazuh manager start
      run: |
        services="`curl -k -s -X GET "https://0.0.0.0:55000/manager/status?pretty=true" -H  "Authorization: Bearer ${{env.TOKEN}}" | jq -r .data.affected_items | grep running | wc -l`"
        if [[ $services -gt 9 ]]; then
          echo "Wazuh Manager Services: ${services}"
          echo "OK"
        else
          echo "Wazuh indexer nodes: ${nodes}"
          curl -k -X GET "https://0.0.0.0:55000/manager/status?pretty=true" -H  "Authorization: Bearer ${{env.TOKEN}}" | jq -r .data.affected_items
          exit 1
        fi
      env:
        TOKEN: $(curl -s -u wazuh-wui:MyS3cr37P450r.*- -k -X GET "https://0.0.0.0:55000/security/user/authenticate?raw=true")

    - name: Check errors in ossec.log
      run: ./.github/single-node-log-check.sh

    - name: Check filebeat output
      run: ./.github/single-node-filebeat-check.sh

    - name: Check Wazuh dashboard service URL
      run: |
       status=$(curl -XGET --silent  https://0.0.0.0:443/app/status -k -u admin:SecretPassword -I -s | grep -E "^HTTP" | awk  '{print $2}')
       if [[ $status -eq 200 ]]; then
        echo "Wazuh dashboard status: ${status}"
       else
        echo "Wazuh dashboard status: ${status}"
        exit 1
       fi

    - name: Stop single node stack
      run: docker-compose -f single-node/docker-compose.yml down

  check-multi-node:
    runs-on: ubuntu-latest
    needs: build-docker-images
    steps:

    - name: Check out code
      uses: actions/checkout@v3

    - name: Create enviroment variables
      run: cat .env > $GITHUB_ENV

    - name: Retrieve saved Wazuh dashboard Docker image
      uses: actions/download-artifact@v3
      with:
        name: docker-artifact-dashboard

    - name: Retrieve saved Wazuh manager Docker image
      uses: actions/download-artifact@v3
      with:
        name: docker-artifact-manager

    - name: Retrieve saved Wazuh indexer Docker image
      uses: actions/download-artifact@v3
      with:
        name: docker-artifact-indexer

    - name: Docker load
      run: |
        docker load --input ./wazuh-manager.tar
        docker load --input ./wazuh-indexer.tar
        docker load --input ./wazuh-dashboard.tar

    - name: Create multi node certficates
      run: docker-compose -f multi-node/generate-indexer-certs.yml run --rm generator

    - name: Start multi node stack
      run: docker-compose -f multi-node/docker-compose.yml up -d

    - name: Check Wazuh indexer start
      run: |
       sleep 120
       status_green="`curl -XGET "https://0.0.0.0:9200/_cluster/health" -u admin:SecretPassword -k -s | grep green | wc -l`"
       if [[ $status_green -eq 1 ]]; then
        curl -XGET "https://0.0.0.0:9200/_cluster/health" -u admin:SecretPassword -k -s
       else
        curl -XGET "https://0.0.0.0:9200/_cluster/health" -u admin:SecretPassword -k -s
        exit 1
       fi
       status_index="`curl -XGET "https://0.0.0.0:9200/_cat/indices" -u admin:SecretPassword -k -s | wc -l`"
       status_index_green="`curl -XGET "https://0.0.0.0:9200/_cat/indices" -u admin:SecretPassword -k -s | grep -E "green" | wc -l`"
       if [[ $status_index_green -eq $status_index ]]; then
        curl -XGET "https://0.0.0.0:9200/_cat/indices" -u admin:SecretPassword -k -s
       else
        curl -XGET "https://0.0.0.0:9200/_cat/indices" -u admin:SecretPassword -k -s
        exit 1
       fi

    - name: Check Wazuh indexer nodes
      run: |
       nodes="`curl -XGET "https://0.0.0.0:9200/_cat/nodes" -u admin:SecretPassword -k -s | grep -E "indexer" | wc -l`"
       if [[ $nodes -eq 3 ]]; then
        echo "Wazuh indexer nodes: ${nodes}"
       else
        echo "Wazuh indexer nodes: ${nodes}"
        exit 1
       fi

    - name: Check documents into wazuh-alerts index
      run: |
       docs="`curl -XGET "https://0.0.0.0:9200/wazuh-alerts*/_count" -u admin:SecretPassword -k -s | jq -r ".count"`"
       if [[ $docs -gt 100 ]]; then
        echo "wazuh-alerts index documents: ${docs}"
       else
        echo "wazuh-alerts index documents: ${docs}"
        exit 1
       fi

    - name: Check Wazuh templates
      run: |
       qty_templates="`curl -XGET "https://0.0.0.0:9200/_cat/templates" -u admin:SecretPassword -k -s | grep "wazuh" | wc -l`"
       templates="`curl -XGET "https://0.0.0.0:9200/_cat/templates" -u admin:SecretPassword -k -s | grep "wazuh"`"
       if [[ $qty_templates -eq 3 ]]; then
        echo "wazuh templates:"
        echo "${templates}"
       else
        echo "wazuh templates:"
        echo "${templates}"
        exit 1
       fi

    - name: Check Wazuh manager start
      run: |
        services="`curl -k -s -X GET "https://0.0.0.0:55000/manager/status?pretty=true" -H  "Authorization: Bearer ${{env.TOKEN}}" | jq -r .data.affected_items | grep running | wc -l`"
        if [[ $services -gt 10 ]]; then
          echo "Wazuh Manager Services: ${services}"
          echo "OK"
        else
          echo "Wazuh indexer nodes: ${nodes}"
          curl -k -s -X GET "https://0.0.0.0:55000/manager/status?pretty=true" -H  "Authorization: Bearer ${{env.TOKEN}}" | jq -r .data.affected_items
          exit 1
        fi
        nodes=$(curl -k -s -X GET "https://0.0.0.0:55000/cluster/nodes" -H "Authorization: Bearer ${{env.TOKEN}}" | jq -r ".data.affected_items[].name" | wc -l)
        if [[ $nodes -eq 2 ]]; then
         echo "Wazuh manager nodes: ${nodes}"
        else
         echo "Wazuh manager nodes: ${nodes}"
         exit 1
        fi
      env:
        TOKEN: $(curl -s -u wazuh-wui:MyS3cr37P450r.*- -k -X GET "https://0.0.0.0:55000/security/user/authenticate?raw=true")

    - name: Check errors in ossec.log
      run: ./.github/multi-node-log-check.sh

    - name: Check filebeat output
      run: ./.github/multi-node-filebeat-check.sh

    - name: Check Wazuh dashboard service URL
      run: |
       status=$(curl -XGET --silent  https://0.0.0.0:443/app/status -k -u admin:SecretPassword -I | grep -E "^HTTP" | awk  '{print $2}')
       if [[ $status -eq 200 ]]; then
        echo "Wazuh dashboard status: ${status}"
       else
        echo "Wazuh dashboard status: ${status}"
        exit 1
       fi

Workflow logs

No response

BuildKit logs

No response

Additional info

No response

crazy-max commented 11 months ago 
    - name: Build Wazuh manager docker image
      uses: docker/build-push-action@v4
      with:
        context: ./build-docker-images/wazuh-manager
        platforms: ${{ matrix.platform }}
        push: false
        buildkitd-flags: --debug
        outputs: type=tar,dest=/home/runner/work/wazuh-docker/wazuh-docker/docker-images/wazuh-manager.tar
        build-args: |
          ARCH=$(uname -m)
        tags: |
          wazuh/wazuh-manager:${{env.WAZUH_IMAGE_VERSION}}

Input values in GitHub Actions are not evaluated within a Unix-like shell environment. You can set the value in a run step for example using an env var like:

  - name: Set the value
    run: |
      echo "ARCH=$(uname -m)" >> "$GITHUB_ENV"

Then you can use it like this:

        build-args: |
          ARCH=${{ env.ARCH }}

Or just:

        build-args: |
          ARCH

Also looking at this step, buildkitd-flags is not a valid input for this action. I think you want to use it in the setup-buildx-action instead: https://github.com/docker/setup-buildx-action

Not sure why you want to pass the machine architecture to your Dockerfile though. You can just use one of the automatic platform ARGs in the global scope for this purpose. Suggest to read our docs about multi-platform image: https://docs.docker.com/build/building/multi-platform/