cloudflare_logpush_job API should contain "dataset" argument

tlozoot commented 4 years ago

Here's the documented API for cloudflare_logpush_job: https://www.terraform.io/docs/providers/cloudflare/r/logpush_job.html

It allows the following arguments:

name
zone_id
destination_conf
ownership_challenge
logpull_options
enable

Cloudflare's API is documented here: https://api.cloudflare.com/#logpush-jobs-create-logpush-job and also contains the "dataset" parameter. This is important for choosing between HTTP requests, Spectrum events, and soon other logs like Firewall and Browser Insight.

There is additional documentation about Logpush jobs here: https://developers.cloudflare.com/logs/logpush/logpush-configuration-api/understanding-logpush-api/

jacobbednarz commented 4 years ago

Cheers @tlozoot 🌮 The dataset attribute will need to be added in cloudflare/cloudflare-go before it can be used here and for who ever picks it up while it's being added, I'd recommend adding the Logpush dataset endpoint itself as well to ensure full coverage.

tlozoot commented 4 years ago

Thanks Jacob! It looks like the LogpushJob struct and API calls do support the "dataset" attribute already in the cloudflare-go library: https://github.com/cloudflare/cloudflare-go/blob/master/logpush.go#L14

Let me know if there's anything else I need to get updated there.

jacobbednarz commented 4 years ago

Ah nice, I totally missed it. That's one less thing to worry about when adding it 😄

prdonahue commented 4 years ago

This is great, looking forward to testing this!

prdonahue commented 4 years ago

Apologies @jacobbednarz but not sure if I should post this error below or open a new ticket?

I'm trying to test the logpush job but running into a problem:

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

sumologic_collector.collector: Refreshing state... [id=108448215]
sumologic_http_source.http_source: Refreshing state... [id=150364538]

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # cloudflare_logpush_job.firewall_events_job will be created
  + resource "cloudflare_logpush_job" "firewall_events_job" {
      + dataset             = "firewall_events"
      + destination_conf    = "sumo://endpoint1.collection.eu.sumologic.com/receiver/v1/http/(redacted)"
      + enabled             = true
      + id                  = (known after apply)
      + logpull_options     = "fields=RayID,Action,ClientIP,ClientRequestHTTPHost,ClientRequestHTTPMethodName,ClientRequestPath,ClientRequestQuery,Datetime,EdgeResponseStatus,RayName&timestamps=rfc3339"
      + name                = "fwevents-logpush-job"
      + ownership_challenge = "(redacted)"
      + zone_id             = "(redacted)"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

$ terraform apply --auto-approve
sumologic_collector.collector: Refreshing state... [id=108448215]
sumologic_http_source.http_source: Refreshing state... [id=150364538]
cloudflare_logpush_job.firewall_events_job: Creating...

Error: error finding logpush job: could not extract Logpush job from resource - invalid identifier (): strconv.Atoi: parsing "": invalid syntax

  on main.tf line 31, in resource "cloudflare_logpush_job" "firewall_events_job":
  31: resource "cloudflare_logpush_job" "firewall_events_job" {

Here's the relevant part of the configuration:

resource "cloudflare_logpush_job" "firewall_events_job" {
  name = "fwevents-logpush-job"
  zone_id = var.cf_zone_id
  enabled = "true"
  dataset = "firewall_events"
  logpull_options = "fields=RayID,Action,ClientIP,ClientRequestHTTPHost,ClientRequestHTTPMethodName,ClientRequestPath,ClientRequestQuery,Datetime,EdgeResponseStatus,RayName&timestamps=rfc3339"
  destination_conf = "sumo://endpoint1.collection.eu.sumologic.com/receiver/v1/http/(redacted)"
  ownership_challenge = "(redacted)"
}

prdonahue commented 4 years ago

Here's with trace output on in case helpful:

cloudflare_logpush_job.firewall_events_job: Creating...
2020/04/20 00:21:04 [DEBUG] cloudflare_logpush_job.firewall_events_job: applying the planned Create change
2020/04/20 00:21:04 [TRACE] GRPCProvider: ApplyResourceChange
2020/04/20 00:21:04 [DEBUG] cloudflare_logpush_job.firewall_events_job: apply errored, but we're indicating that via the Error pointer rather than returning it: error finding logpush job: could not extract Logpush job from resource - invalid identifier (): strconv.Atoi: parsing "": invalid syntax
2020/04/20 00:21:04 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2020/04/20 00:21:04 [TRACE] EvalMaybeTainted: cloudflare_logpush_job.firewall_events_job encountered an error during creation, so it is now marked as tainted
2020/04/20 00:21:04 [TRACE] <root>: eval: *terraform.EvalWriteState
2020/04/20 00:21:04 [TRACE] EvalWriteState: removing state object for cloudflare_logpush_job.firewall_events_job
2020/04/20 00:21:04 [TRACE] <root>: eval: *terraform.EvalApplyProvisioners
2020/04/20 00:21:04 [TRACE] EvalApplyProvisioners: cloudflare_logpush_job.firewall_events_job has no state, so skipping provisioners
2020/04/20 00:21:04 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2020/04/20 00:21:04 [TRACE] EvalMaybeTainted: cloudflare_logpush_job.firewall_events_job encountered an error during creation, so it is now marked as tainted
2020/04/20 00:21:04 [TRACE] <root>: eval: *terraform.EvalWriteState
2020/04/20 00:21:04 [TRACE] EvalWriteState: removing state object for cloudflare_logpush_job.firewall_events_job
2020/04/20 00:21:04 [TRACE] <root>: eval: *terraform.EvalIf
2020/04/20 00:21:04 [TRACE] <root>: eval: *terraform.EvalIf
2020/04/20 00:21:04 [TRACE] <root>: eval: *terraform.EvalWriteDiff
2020/04/20 00:21:04 [TRACE] <root>: eval: *terraform.EvalApplyPost
2020/04/20 00:21:04 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: error finding logpush job: could not extract Logpush job from resource - invalid identifier (): strconv.Atoi: parsing "": invalid syntax
2020/04/20 00:21:04 [ERROR] <root>: eval: *terraform.EvalSequence, err: error finding logpush job: could not extract Logpush job from resource - invalid identifier (): strconv.Atoi: parsing "": invalid syntax
2020/04/20 00:21:04 [TRACE] [walkApply] Exiting eval tree: cloudflare_logpush_job.firewall_events_job
2020/04/20 00:21:04 [TRACE] vertex "cloudflare_logpush_job.firewall_events_job": visit complete
2020/04/20 00:21:04 [TRACE] dag/walk: upstream of "provider.cloudflare (close)" errored, so skipping
2020/04/20 00:21:04 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2020/04/20 00:21:04 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020/04/20 00:21:04 [TRACE] statemgr.Filesystem: not making a backup, because the new snapshot is identical to the old
2020/04/20 00:21:04 [TRACE] statemgr.Filesystem: no state changes since last snapshot
2020/04/20 00:21:04 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate

2020/04/20 00:21:04 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2020/04/20 00:21:04 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
Error: error finding logpush job: could not extract Logpush job from resource - invalid identifier (): strconv.Atoi: parsing "": invalid syntax

  on main.tf line 31, in resource "cloudflare_logpush_job" "firewall_events_job":
  31: resource "cloudflare_logpush_job" "firewall_events_job" {

2020-04-20T00:21:04.713+0100 [DEBUG] plugin: plugin process exited: path=/Users/pdonahue/src/fwe/.terraform/plugins/darwin_amd64/terraform-provider-cloudflare_v2.5.1_x4 pid=49266
2020-04-20T00:21:04.713+0100 [DEBUG] plugin: plugin exited

jacobbednarz commented 4 years ago

It looks like it's failing to find the job ID in the Create 🤔

I'll need to spin one of these up and look at debugging it. I know we've had issues in the past with automating this but not with creating it once you have the ownership_challenge.

jacobbednarz commented 4 years ago

Looking into this, I'm not sure of if something changed since support was added or the implementation wasn't fully vetted at that point as the ownership_challenge is required in the schema but the value can only be created using a separate API call and then read the contents of that file to provide the value to Terraform. Given this, my understanding of how the flow should work is:

Terraform gets the destination_conf set by the user (ownership_challenge should not be required)
Internals then make a call to the GetLogpushOwnershipChallenge cloudflare-go method which returns the file location.
The internals then need to query the service provider (AWS/GCP/Sumo) to get the content of that ownership challenge to put into Terraform to later send with the POST payload.
Once it has that value, it can initiate the Create for the resource.

Based on that, I think we need to work out:

Is that understanding correct?
Is there a way to query the API for the ownership challenge value (not file location)?
If we can't get the value, do we need to bake in the support for querying and downloading AWS/GCP/Sumo validation values?

jacobbednarz commented 4 years ago

Updated the comment as my keyboard got submit happy 😞

prdonahue commented 4 years ago

Hi @jacobbednarz , thanks for looking into this. For the time being I've documented the API challenge calls outside the Terraform flow. I'll speak with @tlozoot about whether we want to bring those in later but for now I'd like to get customers started who already have the challenge response.

Here's the excerpt from my instructions for using log push with Sumo Logic (and note that I've tried replacing the uhh replace() function with a hardcoded variable just to make sure that wasn't the hang up :) —

Create Cloudflare Logpush Job

Before Cloudflare will start start sending logs to your collector, you need to demonstrate the ability to read from it. This check prevents accidental (or intentional) misconfigurations.

Tail the collector

In a new shell window—you should keep the current one with your environment variables set for use with Terraform—we'll start tailing Sumo Logic for events sent from the firewall-events-source HTTP source. The first time that you run livetail you'll need to specify your Sumo Logic Environment, Access ID and Access Key, but these values will be stored for subsequent runs:

$ livetail _source=firewall-events-source
### Welcome to Sumo Logic Live Tail Command Line Interface ###
1 US1
2 US2
3 EU
4 AU
5 DE
6 FED
7 JP
8 CA
Please select Sumo Logic environment: 
See http://help.sumologic.com/Send_Data/Collector_Management_API/Sumo_Logic_Endpoints to choose the correct environment. 3
### Authenticating ###
Please enter your Access ID: <access id>
Please enter your Access Key <access key>
### Starting Live Tail session ###

Request and receive challenge token

Before requesting a challenge token, we need to figure out where Cloudflare should send logs. We do this by asking Terraform for the receiver URL of the recently created HTTP source. (Note that we modify the URL returned slightly as Cloudflare Logs expects "sumo://" rather than "https://".)

$ export SUMO_RECEIVER_URL=$(terraform state show sumologic_http_source.http_source | grep url | awk '{print $3}' | sed -e 's/https:/sumo:/; s/"//g')
$ echo $SUMO_RECEIVER_URL
sumo://endpoint1.collection.eu.sumologic.com/receiver/v1/http/<redacted>

With URL in hand, we can now request the token. Unfortunately the /logpush endpoint doesn't yet support scoped API tokens (working on it!) so you'll need to specify your account's email and global API key:

$ curl -sXPOST -H "Content-Type: application/json" -H "X-Auth-Email: $CF_EMAIL" -H "X-Auth-Key: $TF_VAR_cf_api_key" -d '{"destination_conf":'''"$SUMO_RECEIVER_URL"'''}' https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/logpush/ownership

{"errors":[],"messages":[],"result":{"filename":"ownership-challenge-bb2912e6.txt","message":"","valid":true},"success":true}

Back in the other window where your livetail is running you should see something like this:

{"content":"eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4R0NNIiwidHlwIjoiSldUIn0..WQhkW_EfxVy8p0BQ.oO6YEvfYFMHCTEd6D8MbmyjJqcrASDLRvHFTbZ5yUTMqBf1oniPNzo9Mn3ZzgTdayKg_jk0Gg-mBpdeqNI8LJFtUzzgTGU-aN1-haQlzmHVksEQdqawX7EZu2yiePT5QVk8RUsMrgloa76WANQbKghx1yivTZ3TGj8WquZELgnsiiQSvHqdFjAsiUJ0g73L962rDMJPG91cHuDqgfXWwSUqPsjVk88pmvGEEH4AMdKIol0EOc-7JIAWFBhCqmnv0uAXVOH5uXHHe_YNZ8PNLfYZXkw1xQlVEwH52wRC93ohIxg.pHAeaOGC8ALwLOXqxpXJgQ","filename":"ownership-challenge-bb2912e6.txt"}

Copy the content value from above into an environment variable, as you'll need it in a minute to create the job:

$ export LOGPUSH_CHALLENGE_TOKEN="<content value>"

Create the job using the challenge token

With challenge token in hand, we'll use Terraform to create the job. First we append our Cloudflare configuration to the main.tf

$ cat <<EOF | tee -a main.tf

##################
### CLOUDFLARE ###
##################
provider "cloudflare" {
  version = "~> 2.0"
  email      = var.cf_email
  api_key    = var.cf_api_key
}

resource "cloudflare_logpush_job" "firewall_events_job" {
  enabled = true
  name = "fwevents-logpush-job"
  zone_id = var.cf_zone_id

  dataset = "firewall_events"
  logpull_options = "fields=RayID,Action,ClientIP,ClientRequestHTTPHost,ClientRequestHTTPMethodName,ClientRequestPath,ClientRequestQuery,Datetime,EdgeResponseStatus,RayName&timestamps=rfc3339"
  destination_conf = replace(sumologic_http_source.http_source.url,"https:","sumo:")
  ownership_challenge = "$LOGPUSH_CHALLENGE_TOKEN"
}
EOF

And then we add variables:

$ cat <<EOF | tee -a variables.tf

##################
### CLOUDFLARE ###
##################
variable "cf_email" {
  default = "$CF_EMAIL"
}

variable "cf_zone_id" {
  default = "$CF_ZONE_ID"   
}

# rather than set this here, and potentially commit a secret to our source code repository, we'll set the TF_VAR_cf_api_key environment variable
variable "cf_api_key" {
  default = ""
}
EOF

prdonahue commented 4 years ago

Is there any way to have Terraform dump the underlying API calls it's making (via the golang SDK)?

prdonahue commented 4 years ago

@jacobbednarz good news! @patryk helped me debug just now and I've got it working. Thanks for all your help. Should have a blog post out soon on the new functionality.

jacobbednarz commented 4 years ago

Is there any way to have Terraform dump the underlying API calls it's making (via the golang SDK)?

Just for posterity sake in case someone else stumbles on this, prepending the Terraform command with TF_LOG=DEBUG outputs the HTTP traffic and internal logging.

Awesome to hear! Looks like the Sumo collector is unique here as you can tail the incoming payloads which we can’t unfortunately do for other providers. Looking forward to the blog post and hearing from @tlozoot on whether we can improve this process. Without it, we might only be able to support manual setup (like you’ve mentioned) or import only resources from the UI.

cloudflare / terraform-provider-cloudflare