googleapis / python-bigquery

Apache License 2.0
739 stars 301 forks source link

UnknownJob timestamp properties not being cast #1377

Open yagodorea opened 2 years ago

yagodorea commented 2 years ago

Environment details

Steps to reproduce

  1. Create a service account that doesn't have the bigquery.jobs.listAll permission
  2. Use that service account to fetch a job created by another user (lots of fields will not come, or come redacted, including jobType, so the _AsyncJob will be an UnknownJob, see here)
  3. Try to access the job.started field, it'll break (here) because the timestamp fields aren't being cast to int for UnkownJob instances

Code example

import json
import google_auth_httplib2
import httplib2
from google.cloud import bigquery
from google.oauth2 import service_account
from googleapiclient.discovery import build

f = open("bigquery_service_account.json")
sa = f.read()

scopes = ["https://www.googleapis.com/auth/bigquery.readonly"]
credentials = service_account.Credentials.from_service_account_info(
    json.loads(sa), scopes=scopes
)
# client
client = bigquery.Client(credentials=credentials)

# service
http = httplib2.Http()
authed_http = google_auth_httplib2.AuthorizedHttp(credentials, http=http)
service = build("bigquery", "v2", http=authed_http, cache_discovery=False)

jobs = (
    service.jobs()
    .list(
        allUsers=True,
        projectId="<project_id>",
        maxResults=1,
        projection="full",
    )
    .execute(num_retries=1)
)
jobs = jobs.get("jobs", [])

for job in jobs:
    job = client.job_from_resource(job)
    print(job.started)

Stack trace

Traceback (most recent call last):
  File "bqtest.py", line 38, in <module>
    print(job.started)
  File "[...]lib/python3.8/site-packages/google/cloud/bigquery/job/base.py", line 336, in started
    return _helpers._datetime_from_microseconds(millis * 1000.0)
TypeError: can't multiply sequence by non-int of type 'float'
shollyman commented 1 year ago

TODO: revisit the partial projection issue where the backend redacts some job metadata when parsing jobs.list entries.