search

Azure / aztfexport

A tool to bring existing Azure resources under Terraform's management
https://azure.github.io/aztfexport/
Mozilla Public License 2.0
1.63k stars 190 forks source link

Error running aztfexport in RHEL8 #461

Open cameronew opened 1 year ago

cameronew commented 1 year ago

Hi,

Having the following issue when using the tool. Terraform init works fine but it seems to have trouble when running a simple export. See error below. The plug does seem to be installed ok.

aztfexport resource-group mytestrg

error message

Microsoft Azure Export for Terraform

initializing provider for the import directories: 2 errors occurred:

plugin

-rwxr-xr-x 1 ewan ewan 194940928 Oct 5 12:55 terraform-provider-azurerm_v3.65.0_x5 [ewan@00f836ff2ebd linux_amd64]$ pwd /home/ewan/test/.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.65.0/linux_amd64

terraform init seems ok

[ewan@00f836ff2ebd test]$ terraform init

Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.

If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. [ewan@00f836ff2ebd test]$

magodo commented 1 year ago

@cameronew This seems to be a network issue when downloading the provider. Did you try to remove the pre-installed provider in your environment, and download the provider again via terraform init?

cameronew commented 1 year ago

@cameronew This seems to be a network issue when downloading the provider. Did you try to remove the pre-installed provider in your environment, and download the provider again via terraform init?

@magodo Yes I deleted the provider and ran terraform init which worked fine then had th log.txt e same issue with azfexport. Attached is the trace log.

magodo commented 1 year ago

Do you have any special settings in your .terraformrc?

cameronew commented 1 year ago

No just the defaults

magodo commented 3 months ago

@cameronew Just want to check if you still have this issue, or I can close it now?

ryanhitch commented 3 months ago

I am seeing the same issue on RHEL9. It looks like /tmp is filling up. My /tmp is 2GB, however aztfexport seems to be repeatedly downloading the same provider into /tmp multiple times which fills it up. Running terraform init in the same directory works fine, so I am not sure what issue aztfexport is having. Any suggestions?

Microsoft Azure Export for Terraform

initializing provider for the import directories: 5 errors occurred:

ryanhitch commented 3 months ago

I am not sure if this is related, but I tried the same thing on a Windows machine and get a step further, however now it appears that aztfexport is only configured to interact with the Azure Commercial Cloud API (management.azure.com). The subscription that I want to access is the Azure US Government Cloud, which uses this API URL management.usgovcloudapi.net. az cli uses the following to set this, and the terraform azurerm provider also supports this with the metadata_host parameter. Is there a way to set it with aztfexport?

az cloud set --name "AzureUSGovernment"
magodo commented 3 months ago

@ryanhitch The first error is due to:

no space left on device

The second ask is now supported by: #546

ryanhitch commented 3 months ago

@magodo Before I run aztfexport 1.9G is available in /tmp. "no space left on device" is caused by aztfexport downloading the azurerm provider repeatedly until there is no space left in /tmp. terraform init in the same directory just downloads the provider once, so I am not sure what aztfexport is doing.

Thanks for #546

magodo commented 3 months ago

@ryanhitch Instead you can configure the terraform to cache the provider to avoid repeated download.

ryanhitch commented 3 months ago

By default it looks like aztfexport downloads 10 copies of the azurerm provider into /tmp when running 'aztfexport rg myResourceGroup' which I am guessing is to export 10 resources at a time. This uses 2.3 GB since the provider is ~228MB right now. It looks like the solution is just to ensure that /tmp has at least 2.3 GB free or to cache the provider as @magodo suggests above.

A default Azure RHEL9 VM image only has a 1GB /tmp partition which is likely why some people have had this issue.

2024-08-19T17:37:58.711Z [INFO] aztfexport: New resource group meta 2024-08-19T17:37:58.712Z [INFO] aztfexport: Init Terraform 2024-08-19T17:37:58.773Z [INFO] aztfexport: Find terraform binary at /usr/bin/terraform 2024-08-19T17:37:58.773Z [INFO] aztfexport: Init provider 2024-08-19T17:37:58.773Z [INFO] aztfexport: Output directory doesn't contain provider setting, create one then 2024-08-19T17:37:58.773Z [INFO] aztfexport: Output directory doesn't contain terraform block, create one then 2024-08-19T17:37:58.773Z [DEBUG] aztfexport: Run "terraform init" for the output directory /home/user/outdir 2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-197078599 2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-643364438 2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-563213887 2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-242333829 2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-3144448852 2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-4214556662 2024-08-19T17:38:01.116Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-1222360874 2024-08-19T17:38:01.116Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-3816848619 2024-08-19T17:38:01.116Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-1915249044 2024-08-19T17:38:01.116Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-2642539073 2024-08-19T17:38:11.071Z [ERROR] aztfexport: aztfexport ends with error: initializing provider for the import directories: 10 errors occurred:

Error: Failed to install provider

Error while installing hashicorp/azurerm v3.99.0: write /tmp/terraform-provider101873945: no space left on device

magodo commented 3 months ago

Another way is to reduce the parallelism to a lower number (defaults to 10).