search

Ianyliu / blast-radius-fork

Interactive visualizations of Terraform dependency graphs using d3.js
https://28mm.github.io/blast-radius-docs/
MIT License
41 stars 10 forks source link

Docker - TF is already initialized, but it trying to download modules #19

Open lkurzyniec opened 2 years ago

lkurzyniec commented 2 years ago

I have initialized root module. Running with docker I'm getting

❯ docker run --rm -it -p 5000:5000 -v ${pwd}:/data:ro --security-opt apparmor:unconfined --cap-add=SYS_ADMIN ianyliu/blast-radius-fork
Downloading git::https://github.com/.../repo-modules?ref=v1.7.0 for policies.policy_assignment...
╷
│ Error: Failed to download module
│
│ Could not download module "policy_assignment" (modules/policies/encription_policy_assignments.tf) source code from   
│ "git::https://github.com/.../repo-modules?ref=v1.7.0": error downloading 'https://github.com/.../repo-modules?ref=v1.7.0': git must be  
│ available and on the PATH

I'm using PowerShell and running docker on Windows.

Ianyliu commented 2 years ago

That's strange, I tested it on my Windows computer using PowerShell and didn't encounter any error.... @lkurzyniec Could you provide the full log?

Have you tried building the Docker image yourself? If the error still exists then maybe we need to update some things in the Dockerfile (such as installing Git in the container)

Ianyliu commented 2 years ago

@lkurzyniec Also if you managed to get it work, could you please let me know how? Thanks

lkurzyniec commented 2 years ago

My log is reduced, but all entries says that terraform is trying to download modules from external sources (Could not download module "policy_assignment"). So the questions is - why terraform is trying to init the project (during that phase it download the modules) because the project is already initialized, all modules are downloaded and available in .terraform subfolder.

Ianyliu commented 2 years ago

I see. This is actually how it was meant to be be done I guess, because the Docker image that I created (which was built essentially in the same way that the original one was built in) doesn't copy over the .terraform folders, only the .tf and .tfvars files.

This is so that people can run the visualization in a non-Terraform init directory but still expect the same results.

Do you think it would be good to copy the .terraform folder directly? (One problem is that obviously Terraform would still have to be installed, but we don't know what version should be installed)

lkurzyniec commented 2 years ago

The design is correct - it should work the same for init and non-init environments. But in my case, where I'm using modules from a private repository, the only way to solve the issue would be to provide git creds (login/password or token) that terraform would be allowed to download modules from the private repo. That's a security risk.

Maybe the option here would be a flag/parameter to copy also .terraform folder. Of course, with some appropriate info about that flag, like - terraform must be initialized with Linux version (Linux providers downloaded).

Ianyliu commented 2 years ago

Hi, Sorry I've been busy with school, hence the late reply.

I think that idea is great! I've seen other forks before specify Git as something to download in the Dockerfile.. but wasn't sure why... now I know.

I'll try to work on adding a flag, but could you please explain more about what you mean when you say "Linux version (Linux providers downloaded)"?

Ianyliu commented 2 years ago

Sorry @lkurzyniec I just wanted to confirm this is what you're talking about.... ? image

lkurzyniec commented 2 years ago

@Ianyliu no worries about delays. everyone has its own duties ;)

by 'Linux version (Linux providers downloaded)' I mean that - terraform is multiplatform, but you are running blast on alpine, which is linux based, so terraform have to be initialized with linux providers (instead of windows for example).

lkurzyniec commented 2 years ago

when we are invoking this command docker run --rm -it -p 5000:5000 -v ${pwd}:/data:ro --security-opt apparmor:unconfined --cap-add=SYS_ADMIN ianyliu/blast-radius-fork, in the part -v ${pwd}:/data:ro we are creating the volume (shared filesystems) to our terraform source code. later on, I'm getting this error: Could not download module (...) git must be available and on the PATH which means that terraform is trying to init my terraform source code. but it was already initialized by my. so the question is - why terraform inside the docker ignore .terrafrom folder where the configuration, providers and initialized modules are downloaded. as we discussed previously (here: https://github.com/Ianyliu/blast-radius-fork/issues/19#issuecomment-1238821413), it's highly likely that he is ignoring .terraform and trying to init, because we want to work as well on not initialized environment. to solve my problem, I'm suggesting to add one more flag, to skip terraform init and use/consume already existing .terraform folder (which is under ${pwd}/.terraform)

Ianyliu commented 2 years ago

Hi @lkurzyniec Could you please test out this new change I made to the dev branch? It should resolve the problems for remote modules I integrated changes from Jrc356/blast-radius

You can install via: pip install -U git+https://github.com/Ianyliu/blast-radius-fork@dev

If you need me to create a Docker dev branch, please let me know and I will do that

lkurzyniec commented 2 years ago

@Ianyliu please create a Docker image for me. I can work only with docker.

lkurzyniec commented 2 years ago

The newest image gives me an error:

Digest: sha256:e2a00e9108f3716a8d5a5392955954ed062f5fba761d8fbcd2759ea4a87b6df4
Status: Downloaded newer image for ianyliu/blast-radius-fork:latest
exec /bin/docker-entrypoint.sh: no such file or directory
lkurzyniec commented 2 years ago

I have created a repo to test your work. The results of docker run could be found here: https://github.com/lkurzyniec/azure-regions-public/actions/workflows/docker.yml

Ianyliu commented 2 years ago

Okay, I'll have a look later. Thank you so much for your help

Ianyliu commented 1 year ago

Hi @lkurzyniec , I haven't looked at your repo yet, but I rebuilt an image. Could you please test it out and see if it works?

lkurzyniec commented 1 year ago

@Ianyliu the same error - exec /bin/docker-entrypoint.sh: no such file or directory

I added you to my repository as a collaborator. Feel free to run the pipeline (https://github.com/lkurzyniec/azure-regions-public/actions/workflows/docker.yml) to see the results.

Ianyliu commented 1 year ago

Sorry @lkurzyniec I wasn't sure why you shared it with me or how to use GitHub workflows, but now it seems like I have it figured out.

Also I just realized why things weren't working.... it's because I switched from Mac to Windows and the filepath is different on Windows (\ is file separator)...... I can't believe it took me so long to figure out something so simple 😅

Not sure if I need to switch machines though...

Ianyliu commented 1 year ago

@lkurzyniec Can you help me configure a GitHub workflow so that a Docker image is pushed to Docker Hub every time there's a pull request or push to the main branch?

lkurzyniec commented 1 year ago

@Ianyliu To achieve that, simply use the instructions from the action repo: https://github.com/docker/build-push-action/tree/v3 (copy&paste, then adjust). You can also take a quick look on my workflow (https://github.com/lkurzyniec/netcore-boilerplate/blob/master/.github/workflows/docker.yml), but above instructions should be enough. If help need, just give me a shout.

lkurzyniec commented 1 year ago

@Ianyliu finally managed to visualize my terraform. I used different tool. here you can find my current solution: https://github.com/im2nguyen/rover/issues/115#issuecomment-1366607375

Ianyliu commented 1 year ago

@lkurzyniec Sorry for the wait, I was busy with school. This might be late, but recently I got my hands on a Mac and pushed a working Docker image to the Hub.

I indeed have used the tool you mentioned, and I also recommend Pluralith. I do have to admit that they might be better tools than Blast Radius. The only major difference of Blast Radius is the way it visualizes the infrastructure.

I did spend time looking at the workflow, but I was quite confused and later chose to stop working on this project as a whole due to schoolwork.

If you do have time someday to spare, please let me know if my latest image works for you. (I checked the workflow and the app ran successfully but timed out)

Regardless of everything, I hope it went well for you in the end.

lkurzyniec commented 1 year ago

@Ianyliu I notice that you are able to run workflow on your side. that's good.

it looks better. I see that blastradius runs in a container (Running on http://127.0.0.1:5000), but it seems like it didn't receive tf files (Terraform initialized in an empty directory!). my configuration looks ok, the files should be mounted to blastradius container (-v ${pwd}:/data:ro). details: https://github.com/lkurzyniec/azure-regions-public/actions/runs/3925385868/jobs/6710272069