BigQuery log export

[davidebelloni]

Hi, I'm trying to export some logs to bigquery with the following configuration:

- name: BQ_LOG-bigquery-dataset
  type: gcp-types/bigquery-v2:datasets
  properties:
    access:
    - role: OWNER
      specialGroup: projectOwners
    datasetReference:
      datasetId: BQ_LOG
      projectId: my-project
    description: Dataset BigQuery Logs Export
    friendlyName: Dataset BigQuery Logs Export
    location: EU
- name: log-sink-BQ_LOG-bigquery
  type: gcp-types/logging-v2:projects.sinks
  properties:
    destination: bigquery.googleapis.com/projects/my-project/datasets/$(ref.BQ_LOG-bigquery-dataset.datasetReference.datasetId)
    filter: resource.type="bigquery_resource"
    project: my-project
    sink: log-sink-BQ_LOG-bigquery
    uniqueWriterIdentity: true
- name: BQ_LOG-bigquery-dataset-set-sink-iam
  action: gcp-types/bigquery-v2:bigquery.datasets.patch
  properties:
    accesses:
    - role: OWNER
      specialGroup: projectOwners
    - role: WRITER
      userByEmail: $(ref.log-sink-BQ_LOG-bigquery.writerIdentity)
    datasetId: $(ref.BQ_LOG-bigquery-dataset.datasetReference.datasetId)
    projectId: my-project

What happen is:

• the dataset BQ_LOG is created
• the sink log-sink-BQ_LOG-bigquery is created
• I've no evidence in Stackdriver Logging about bigquery.datasets.patch action
• the dataset BQ_LOG at the end has only the projectOwners:OWNER access
• the DM finish successfully, but as I've said the deployment doesn't make what I want

I've found that the issue is accesses and the right config is access , but no error is fired!

With the correct configuration instead:

• the following error is fired: Email address contains invalid user name
• the translated configuration for BQ_LOG-bigquery-dataset-set-sink-iam resource has userByEmail: serviceAccount:p770900990982-803178@gcp-sa-logging.iam.gserviceaccount.com

I don't know how to take a substring of a referenced value. Do you know how can I achieve the target?

Thanks

[likeulb]

Hi, We will have an internal bug looking into the first issue you mentioned. Seems we currently silently ignore the fields that is not specified in the API discovery doc, for instance, the "accesses" field in your case. For the other issue, we will have a new feature to support some json path functions. When it is available, you can do something like $(ref.BQ_LOG-bigquery-dataset.datasetReference.datasetId.substring(0,10)).

[likeulb]

Hi, For the first issue you mentioned, we will open an internal bug to throw warning in that case. For the second issue, we are already working on the feature. I'll close this issue for now.

Thanks

[LogansUA]

@davidebelloni Have you managed to figure out the solution for this? Because I'm now having similar problems:

1. Dataset is created
2. LogSink is created
3. But sink doesn't have permissions to write in dataset.

@likeulb Maybe you can suggest something since "some json path functions" is still not done (i guess). @ocsig

Thank you

[davidebelloni]

Hi @LogansUA , I've found no solution and I've hardcoded in the jinja file the name of the service account

[davidebelloni]

Hi @likeulb , news about feature to support some json path functions?

Thanks

[ocsig]

Are you getting the same issue with the CFT logsink templates ?

[davidebelloni]

Hi, I cannot use CFT logsink templates because BQ datasets is already active (and in some cases are in a different project)