Output file is not getting exported

[aditivarade]

Hi, I am using steampipe docker image(turbot/steampipe:latest). I followed this documentation and built the docker image based on the example dockerfile provided in the documentation.

Below is the docker run command used docker run \ -it \ --rm \ -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \ -e AWS_REGION=us-east-1 \ -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \ --name steampipe-compliance \ --mount type=bind,source="${PWD}",target=/output \ steampipe-aws-compliance check benchmark.cis_v140_2_1 --export /output/myoutput.json

After running this command I can see the output on the console but the file is not getting exported on the host machine. Can you please let me know how can I resolve this or if I am missing something?

[pskrbasu]

Hi @aditivarade, were you following the Running a batch job with derived image documentation? Per this guide(and the docker run command you used), the file should be exported on your host machine at ${PWD}.

Can you send us the contents of the Dockerfile that you built?

[aditivarade]

@pskrbasu - yes I was following "Running a batch job with derived image" documentation. I checked the current directory(${PWD}), also tried giving physical path(/tmp/test) of the host machine, but it did not export the file on any of the mentioned paths. Below is the Dockerfile I am using

FROM turbot/steampipe

Setup prerequisites (as root)

USER root:0 RUN apt-get update -y \ && apt-get install -y git

Install the aws and steampipe plugins for Steampipe (as steampipe user).

USER steampipe:0 RUN steampipe plugin install steampipe aws

A mod may be installed to a working directory

RUN git clone --depth 1 https://github.com/turbot/steampipe-mod-aws-compliance.git /workspace WORKDIR /workspace

[pskrbasu]

Apologies for the late reply @aditivarade. I have been using the same Dockerfile+docker run commands combination, but the export is working for me. So I have some questions for you, which might help me reproduce your issue.

• What system are you using(Mac/Ubuntu/WSL)?
• As you mentioned I can see the output on the console, do you see any steampipe error messages associated with it(Error: no such file)?
• Can you run interactive and exec into the container and run ls /output to check if the myoutput.json is present there?

[aditivarade]

Hi @pskrbasu, to answer your questions

• I am using Ubuntu system
• There is no steampipe error, for your help please have a look at the below snapshots of my terminal
• I do not see the output file(aws_output.json) created inside the container as shown in the below snapshot. I can see only the Dockerfile inside the container which is in my ${PWD}(/tmp/test) directory because of the mount provided in the command.

[pskrbasu]

@aditivarade Thanks for the elaborate answers. I have somewhat reproduced the issue you are facing.

I tried the same on Ubuntu 20.04 and it seems like the export file isn't getting created at all in the container. But unlike your case, I do get an error message from steampipe(Error: open /output/aws_output.json: permission denied).

I will keep digging into why this is happening and would hope to come back with a solution/fix. Stay tuned!

Thanks

[pskrbasu]

Also @aditivarade can you try running docker run -it -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_REGION=$AWS_REGION -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID --name steampipe-compliance --mount type=bind,source="${PWD}",target=/output steampipe-aws-compliance check benchmark.cis_v140_2_1 --share

This should return an error

Error: Not authenticated for Steampipe Cloud.
Please run 'steampipe login' or setup a token.

Can you let us know if you're seeing this error from steampipe?

[aditivarade]

@pskrbasu Thanks for confirming the issue. Tried running the command you shared, it gives an error for "--share" flag

Just FYI I am using "Ubuntu 18.04.1 LTS" system. Will be waiting for the solution.

[pskrbasu]

@aditivarade It seems like you are using an old version of the image(--share is available in our latest releases). I pulled the latest and ran this to verify that I'm using v0.17.1(latest) of steampipe.

docker run -it --rm --name sp steampipe-aws-compliance --version
steampipe version 0.17.1

Can you pull the latest available image(FROM turbot/steampipe:latest) and try docker run -it --rm --name sp steampipe-aws-compliance --version to make sure we're both on the latest version(0.17.1)? Both --share and --export is available in our latest releases.

After verifying that you are now using 0.17.1, can you try:

• export

  docker run
  -it
  --rm
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
  -e AWS_REGION=us-east-1
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
  --name steampipe-compliance
  --mount type=bind,source="${PWD}",target=/output
  steampipe-aws-compliance check benchmark.cis_v140_2_1 --export /output/myoutput.json

  Expectation: still no myoutput.json file in $PWD? and still, no error like this Error: can't open file/ Error: permission denied?

• share

  docker run
  -it
  --rm
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
  -e AWS_REGION=us-east-1
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
  --name steampipe-compliance
  --mount type=bind,source="${PWD}",target=/output
  steampipe-aws-compliance check benchmark.cis_v140_2_1 --share

  Expectation: error Error: Not authenticated for Steampipe Cloud. Please run 'steampipe login' or setup a token.

Let us know what you experience after updating to 0.17.1, these answers can prove to be very helpful for us to pinpoint the issue. Thank you for your patience.

[aditivarade]

@pskrbasu - I was using the latest tag of the image(FROM turbot/steampipe:latest), it seems its actually 0.16.4 version. I tried the latest version(FROM turbot/steampipe:0.17.1) and I'm now getting the same errors that you have mentioned for both export and share.

[pskrbasu]

@aditivarade Thanks for confirming. So I dug into the issue and found out the reason behind it.

I ran docker run ..... --mount type=bind,source="${PWD}",target=/output steampipe-aws-compliance check benchmark.cis_v140_2_1 --export /output/myoutput.json and it returned an Error: open /output/myoutput.json: permission denied error message as we have been getting.

After that, I did an exec into the running container to have a look at the file structures inside and guess what

drwxr-xr-x   5      1000 1000 4096 Nov 10 15:20 output
drwxr-xr-x   1 steampipe root 4096 Nov 11 05:34 workspace

the mounted dir /output is owned by uid 1000 and not by root/steampipe user. I tried to manually create a file inside /output only to see our error reproduced againtouch: cannot touch 'steampipe.json': Permission denied.

So, now steampipe check when run with --export flag, creates a file and writes the output to the file. The problem here is, steampipe is failing to create the file in the mounted dir(/ouptut) and hence getting a permission denied.

This is the issue that we are facing here, unfortunately this is not a bug in our code, but a docker runtime issue.

I'm still looking into ways to overcome this situation in Ubuntu and will update you here if I find anything. Meanwhile, if you find any trick to pass args to docker run to allow permissions to the mounted dir, please let us know.

Thanks

[github-actions[bot]]

'This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

[pskrbasu]

https://github.com/turbot/steampipe-docs/issues/96