Add methods to run Terraform Templates using TerraformRunner container APIs

[putmanoj]

• Add run method to run terraform templates in async mode
• Add run_async method to run terraform templates
• Response class to represent TerraformRunner-Stack API response
• ResponseAsync class with stop method to cancel a TerraformRunner-Stack API Job.
• Add spec the test terraform-runner methods

[agrare]

@putmanoj please rebase on master rather than merge master into this branch.

[putmanoj]

@putmanoj please rebase on master rather than merge master into this branch.

ok

[agrare]

@putmanoj you can add this to your ~/.gitconfig

[pull]
    rebase = true

That way git pull upstream master will "just work" (NOTE this is covered in the developer setup guide)

[putmanoj]

~/.gitconfig

ok

[jrafanie]

Ok, I pushed a backup of the current branch here: https://github.com/ManageIQ/manageiq-providers-embedded_terraform/compare/master...jrafanie:manageiq-providers-embedded_terraform:terraform-runner-backup

I'm force pushing minor changes. I've squashed a few "fix" commits. I've marked the comments as resolved. There's still a few remaining questions I'll try to review and resolve tomorrow.

[jrafanie]

@putmanoj I added a few commits to use faraday instead of rest-client. Once you update the docs so I can run the opentofu runner, I'll test it for real. All the tests pass but I'd like to make sure my changes to the various headers such as ssl verify and authorization work correctly. I'll need to check the response logic with real responses but I think it's a good start.

[putmanoj]

@putmanoj I added a few commits to use faraday instead of rest-client. Once you update the docs so I can run the opentofu runner, I'll test it for real. All the tests pass but I'd like to make sure my changes to the various headers such as ssl verify and authorization work correctly. I'll need to check the response logic with real responses but I think it's a good start.

Hi @jrafanie You can now use the terraform-runner repo, to start a dev docker container for your testing, will DM you more details.

[jrafanie]

Ok @putmanoj. I was able to run the simple-template-hcl and run Terraform::Runner.available? and Terraform::Runner.run_async({}, Dir.pwd) in rails console with the TERRAFORM_RUNNER_URL and TERRAFORM_RUNNER_TOKEN env variables set and I get the proper API responses with the faraday code in place. 🎉

[jrafanie]

@putmanoj here's a listing of all my changes from your original PR before I started to now:

git diff origin/terraform-runner-backup putmanoj/terraform-runner

diff --git a/lib/terraform/runner.rb b/lib/terraform/runner.rb
index 172fc0a..ae2cd37 100644
--- a/lib/terraform/runner.rb
+++ b/lib/terraform/runner.rb
@@ -1,5 +1,4 @@
-require 'rest-client'
-require 'timeout'
+require 'faraday'
 require 'tempfile'
 require 'zip'
 require 'base64'
@@ -10,8 +9,8 @@ module Terraform
       def available?
         return @available if defined?(@available)

-        response = terraform_runner_client['api/terraformjobs/count'].get
-        @available = response.code == 200
+        response = terraform_runner_client.get('api/terraformjobs/count')
+        @available = response.status == 200
       rescue
         @available = false
       end
@@ -38,6 +37,10 @@ module Terraform
         Terraform::Runner::ResponseAsync.new(response.stack_id)
       end

+      # To simplify clients who may just call run, we alias it to call
+      # run_async.  If we ever need run_sync, we'll need to revisit this.
+      alias run run_async
+
       # Stop running terraform-runner job by stack_id
       #
       # @param stack_id [String] stack_id from the terraforn-runner job
@@ -47,27 +50,6 @@ module Terraform
         cancel_stack_job(stack_id)
       end

-      # Runs a template, waits until it terraform-runner job completes, via terraform-runner api
-      #
-      # @param input_vars [Hash] Hash with key/value pairs that will be passed as input variables to the
-      #        terraform-runner run
-      # @param template_path [String] Path to the template we will want to run
-      # @param tags [Hash] Hash with key/values pairs that will be passed as tags to the terraform-runner run
-      # @param credentials [Array] List of Authentication object ids to provide to the terraform run
-      # @param env_vars [Hash] Hash with key/value pairs that will be passed as environment variables to the
-      #        terraform-runner run
-      # @return [Terraform::Runner::Response] Response object with final result of terraform run
-      def run(input_vars, template_path, tags: nil, credentials: [], env_vars: {})
-        _log.debug("Run template: #{template_path}")
-        create_stack_job_and_wait_until_completes(
-          template_path,
-          :input_vars  => input_vars,
-          :tags        => tags,
-          :credentials => credentials,
-          :env_vars    => env_vars
-        )
-      end
-
       # Fetch terraform-runner job result/status by stack_id
       #
       # @param stack_id [String] stack_id from the terraforn-runner job
@@ -83,7 +65,7 @@ module Terraform
       private

       def server_url
-        ENV['TERRAFORM_RUNNER_URL'] || 'https://opentofu-runner:27000'
+        ENV.fetch('TERRAFORM_RUNNER_URL', 'https://opentofu-runner:27000')
       end

       def server_token
@@ -91,15 +73,11 @@ module Terraform
       end

       def stack_job_interval_in_secs
-        ENV['TERRAFORM_RUNNER_STACK_JOB_CHECK_INTERVAL'].to_i
-      rescue
-        10
+        ENV.fetch('TERRAFORM_RUNNER_STACK_JOB_CHECK_INTERVAL', 10).to_i
       end

       def stack_job_max_time_in_secs
-        ENV['TERRAFORM_RUNNER_STACK_JOB_MAX_TIME'].to_i
-      rescue
-        120
+        ENV.fetch('TERRAFORM_RUNNER_STACK_JOB_MAX_TIME', 120).to_i
       end

       # Create to paramaters as used by terraform-runner api
@@ -108,33 +86,37 @@ module Terraform
       #        terraform-runner run
       # @return [Array] Array of {:name,:value}
       def convert_to_cam_parameters(vars)
-        parameters = []
-        vars&.each do |key, value|
-          parameters.push(
-            {
-              :name  => key,
-              :value => value
-            }
-          )
+        return [] if vars.nil?
+
+        vars.map do |key, value|
+          {
+            :name  => key,
+            :value => value
+          }
         end
-        parameters
       end

       # create http client for terraform-runner rest-api
       def terraform_runner_client
-        # TODO: verify ssl
-        verify_ssl = false
-
-        RestClient::Resource.new(
-          server_url,
-          :headers    => {:authorization => "Bearer #{server_token}"},
-          :verify_ssl => verify_ssl
-        )
+        @terraform_runner_client ||= begin
+          # TODO: verify ssl
+          verify_ssl = false
+
+          Faraday.new(
+            :url => server_url,
+            :ssl => {:verify => verify_ssl}
+          ) do |builder|
+            builder.request(:authorization, 'Bearer', -> { server_token })
+          end
+        end
       end

       def stack_tenant_id
-        # TODO: fix hardcoded tenant_id
-        'c158d710-d91c-11ed-9fee-d93323035b4e'
+        '00000000-0000-0000-0000-000000000000'.freeze
+      end
+
+      def json_post_arguments(payload)
+        return JSON.generate(payload), "Content-Type" => "application/json".freeze
       end

       # Create TerraformRunner Stack Job
@@ -148,45 +130,32 @@ module Terraform
       )
         _log.info("start stack_job for template: #{template_path}")
         tenant_id = stack_tenant_id
-
-        # Temp Zip File
-        # zip_file_path = Tempfile.new(%w/tmp .zip/).path
-        zip_file_path = Tempfile.new(%w[tmp .zip]).path
-        create_zip_file_from_directory(zip_file_path, template_path)
-        zip_file_hash = Base64.encode64(File.binread(zip_file_path))
+        encoded_zip_file = encoded_zip_from_directory(template_path)

         # TODO: use tags,env_vars
-        payload = JSON.generate(
-          {
-            :cloud_providers => credentials,
-            :name            => name,
-            :tenantId        => tenant_id,
-            :templateZipFile => zip_file_hash,
-            :parameters      => convert_to_cam_parameters(input_vars)
-          }
-        )
+        payload = {
+          :cloud_providers => credentials,
+          :name            => name,
+          :tenantId        => tenant_id,
+          :templateZipFile => encoded_zip_file,
+          :parameters      => convert_to_cam_parameters(input_vars)
+        }
         # _log.debug("Payload:>\n, #{payload}")
-        http_response = terraform_runner_client['api/stack/create'].post(
-          payload, :content_type => 'application/json'
+
+        http_response = terraform_runner_client.post(
+          "api/stack/create",
+          *json_post_arguments(payload)
         )
         _log.debug("==== http_response.body: \n #{http_response.body}")
         _log.info("stack_job for template: #{template_path} running ...")
         Terraform::Runner::Response.parsed_response(http_response)
-      ensure
-        # cleanup temp zip file
-        FileUtils.rm_rf(zip_file_path) if zip_file_path
-        _log.debug("Deleted #{zip_file_path}")
       end

       # Retrieve TerraformRunner Stack Job details
       def retrieve_stack_job(stack_id)
-        payload = JSON.generate(
-          {
-            :stack_id => stack_id
-          }
-        )
-        http_response = terraform_runner_client['api/stack/retrieve'].post(
-          payload, :content_type => 'application/json'
+        http_response = terraform_runner_client.post(
+          "api/stack/retrieve",
+          *json_post_arguments({:stack_id => stack_id})
         )
         _log.info("==== Retrieve Stack Response: \n #{http_response.body}")
         Terraform::Runner::Response.parsed_response(http_response)
@@ -194,99 +163,30 @@ module Terraform

       # Cancel/Stop running TerraformRunner Stack Job
       def cancel_stack_job(stack_id)
-        payload = JSON.generate(
-          {
-            :stack_id => stack_id
-          }
-        )
-        http_response = terraform_runner_client['api/stack/cancel'].post(
-          payload, :content_type => 'application/json'
+        http_response = terraform_runner_client.post(
+          "api/stack/cancel",
+          *json_post_arguments({:stack_id => stack_id})
         )
         _log.info("==== Cancel Stack Response: \n #{http_response.body}")
         Terraform::Runner::Response.parsed_response(http_response)
       end

-      # Wait for TerraformRunner Stack Job to complete
-      def wait_until_completes(stack_id)
-        interval_in_secs = stack_job_interval_in_secs
-        max_time_in_secs = stack_job_max_time_in_secs
-
-        response = nil
-        Timeout.timeout(max_time_in_secs) do
-          _log.debug("Starting wait for terraform-runner/stack/#{stack_id} completes ...")
-          i = 0
-          loop do
-            _log.debug("loop #{i}")
-            i += 1
-
-            response = retrieve_stack_job(stack_id)
-
-            _log.info("status: #{response.status}")
-
-            case response.status
-            when "SUCCESS"
-              _log.debug("Successful! (stack_job/:#{stack_id})")
-              break
-
-            when "FAILED"
-              _log.info("Failed! (stack_job/:#{stack_id} fails!)")
-              _log.info(response.error_message)
-              break
-
-            when nil
-              _log.info("No status! stack_job/:#{stack_id} must have failed, check response ...")
-              _log.info(response.message)
-              break
-            end
-            _log.info("============\n stack_job/:#{stack_id} status=#{response.status} \n============")
-
-            # sleep interval
-            _log.debug("Sleep for #{interval_in_secs} secs")
-            sleep interval_in_secs
-
-            break unless i < 20
-          end
-          _log.debug("loop ends: ran #{i} times")
-        end
-        response
-      end
-
-      # Create TerraformRunner Stack Job, wait until completes
-      def create_stack_job_and_wait_until_completes(
-        template_path,
-        input_vars: [],
-        tags: nil,
-        credentials: [],
-        env_vars: {},
-        name: "stack-#{rand(36**8).to_s(36)}"
-      )
-        _log.info("create_stack_job_and_wait_until_completes for #{template_path}")
-        response = create_stack_job(
-          template_path,
-          :input_vars  => input_vars,
-          :tags        => tags,
-          :credentials => credentials,
-          :env_vars    => env_vars,
-          :name        => name
-        )
-        wait_until_completes(response.stack_id)
-      end
-
-      # create zip from directory
-      def create_zip_file_from_directory(zip_file_path, template_path)
+      # encode zip of a template directory
+      def encoded_zip_from_directory(template_path)
         dir_path = template_path # directory to be zipped
         dir_path = path[0...-1] if dir_path.end_with?('/')

-        _log.debug("Create #{zip_file_path}")
-        Zip::File.open(zip_file_path, Zip::File::CREATE) do |zipfile|
-          Dir.chdir(dir_path)
-          Dir.glob("**/*").reject { |fn| File.directory?(fn) }.each do |file|
-            _log.debug("Adding #{file}")
-            zipfile.add(file.sub("#{dir_path}/", ''), file)
+        Tempfile.create(%w[opentofu-runner-payload .zip]) do |zip_file_path|
+          _log.debug("Create #{zip_file_path}")
+          Zip::File.open(zip_file_path, Zip::File::CREATE) do |zipfile|
+            Dir.chdir(dir_path)
+            Dir.glob("**/*").select { |fn| File.file?(fn) }.each do |file|
+              _log.debug("Adding #{file}")
+              zipfile.add(file.sub("#{dir_path}/", ''), file)
+            end
           end
+          Base64.encode64(File.binread(zip_file_path))
         end
-
-        zip_file_path
       end
     end
   end
diff --git a/lib/terraform/runner/response_async.rb b/lib/terraform/runner/response_async.rb
index 7b62fae..f8b3c9c 100644
--- a/lib/terraform/runner/response_async.rb
+++ b/lib/terraform/runner/response_async.rb
@@ -52,38 +52,15 @@ module Terraform
         @response
       end

-      # Dumps the Terraform::Runner::ResponseAsync into the hash
-      #
-      # @return [Hash] Dumped Terraform::Runner::ResponseAsync object
-      def dump
-        {
-          :stack_id => @stack_id
-        }
-      end
-
-      # Creates the Terraform::Runner::ResponseAsync object from hash data
-      #
-      # @param hash [Hash] Dumped Terraform::Runner::ResponseAsync object
-      #
-      # @return [Terraform::Runner::ResponseAsync] Terraform::Runner::ResponseAsync Object created from hash data
-      def self.load(hash)
-        # Dump dumps a hash and load accepts a hash, so we must expand the hash to kwargs as new expects kwargs
-        new(**hash)
-      end
-
       private

       def completed?(status)
-        # IF NOT SUCCESS,FAILED,CANCELLED
-        if status
-          return (
-            status.start_with?("SUCCESS", "FAILED") ||
-            # @response.status == "ERROR" ||
-            status == "CANCELLED"
-          )
+        case status.to_s.upcase
+        when "SUCCESS", "FAILED", "CANCELLED"
+          true
+        else
+          false
         end
-
-        false
       end
     end
   end
diff --git a/spec/lib/terraform/runner_spec.rb b/spec/lib/terraform/runner_spec.rb
index 2b23008..49ab330 100644
--- a/spec/lib/terraform/runner_spec.rb
+++ b/spec/lib/terraform/runner_spec.rb
@@ -21,89 +21,6 @@ RSpec.describe(Terraform::Runner) do
     end
   end

-  describe ".run hello-world with no input vars ( nil argument )" do
-    let(:input_vars) { nil }
-
-    before do
-      ENV["TERRAFORM_RUNNER_URL"] = "https://1.2.3.4:7000"
-
-      stub_request(:post, "https://1.2.3.4:7000/api/stack/create")
-        .with(:body => hash_including({:parameters => []}))
-        .to_return(
-          :status => 200,
-          :body   => @hello_world_create_response.to_json
-        )
-
-      stub_request(:post, "https://1.2.3.4:7000/api/stack/retrieve")
-        .with(:body => hash_including({:stack_id => @hello_world_retrieve_response['stack_id']}))
-        .to_return(
-          :status => 200,
-          :body   => @hello_world_retrieve_response.to_json
-        )
-    end
-
-    it "runs a hello-world terraform template" do
-      response = Terraform::Runner.run(input_vars, File.join(__dir__, "runner/data/hello-world"))
-
-      expect(response.status).to(eq('SUCCESS'), "terraform-runner failed with:\n#{response.status}")
-      expect(response.message).to(include('greeting = "Hello World"'))
-      expect(response.stack_id).to(eq(@hello_world_retrieve_response['stack_id']))
-      expect(response.action).to(eq('CREATE'))
-      expect(response.stack_name).to(eq(@hello_world_retrieve_response['stack_name']))
-      expect(response.details.keys).to(eq(%w[resources outputs]))
-    end
-  end
-
-  describe ".run hello-world with input_vars" do
-    let(:input_vars) { {:name => 'Mumbai'} }
-
-    def verify_request_and_respond(request)
-      body = JSON.parse(request.body)
-
-      # verify parameters
-      expect(body['parameters'].length).to(eq(1))
-      data = body['parameters'][0]
-      expect(data['name']).to(eq('name'))
-      expect(data['value']).to(eq('Mumbai'))
-
-      # verify other attributes
-      expect(body['name']).not_to(be_empty)
-      expect(body['tenantId']).not_to(be_empty)
-      expect(body['templateZipFile']).not_to(be_empty)
-
-      @hello_world_create_response.to_json
-    end
-
-    before do
-      ENV["TERRAFORM_RUNNER_URL"] = "https://1.2.3.4:7000"
-
-      stub_request(:post, "https://1.2.3.4:7000/api/stack/create")
-        .to_return(->(request) { {:body => verify_request_and_respond(request)} })
-
-      response = @hello_world_retrieve_response.clone
-      response['message'] = response['message'].gsub('World', 'Mumbai')
-      response['details']['outputs'][0]['value'] = response['details']['outputs'][0]['value'].sub('World', 'Mumbai')
-
-      stub_request(:post, "https://1.2.3.4:7000/api/stack/retrieve")
-        .with(:body => hash_including({:stack_id => @hello_world_retrieve_response['stack_id']}))
-        .to_return(
-          :status => 200,
-          :body   => response.to_json
-        )
-    end
-
-    it "runs a hello-world terraform template" do
-      response = Terraform::Runner.run(input_vars, File.join(__dir__, "runner/data/hello-world"))
-
-      expect(response.status).to(eq('SUCCESS'), "terraform-runner failed with:\n#{response.status}")
-      expect(response.message).to(include('greeting = "Hello Mumbai"'))
-      expect(response.stack_id).to(eq(@hello_world_retrieve_response['stack_id']))
-      expect(response.action).to(eq('CREATE'))
-      expect(response.stack_name).to(eq(@hello_world_retrieve_response['stack_name']))
-      expect(response.details.keys).to(eq(%w[resources outputs]))
-    end
-  end
-
   context '.run_async hello-world' do
     describe '.run_async' do
       create_stub = nil
@@ -142,7 +59,10 @@ RSpec.describe(Terraform::Runner) do
         expect(response.stack_name).to(eq(@hello_world_create_response['stack_name']))
         expect(response.message).to(be_nil)
         expect(response.details).to(be_nil)
+      end

+      it "is aliased as run" do
+        expect(Terraform::Runner.method(:run)).to eq(Terraform::Runner.method(:run_async))
       end
     end

[jrafanie]

I'll tackle the ping endpoint and other comments in a followup PR