Add support for JSON API headers and query string parameters

[mlhamel]

Description

The XML-Api and the Json-API are supporting a large set of parameters as described at https://cloud.google.com/storage/docs/xml-api/reference-headers.

Into those parameters you can find very useful features like x-goog-metageneration which let you control how to deal with versions and also blocking rewriting of blobs in a bucket.

These API used to be available in the AppEngine library for google cloud storage as you can see at https://cloud.google.com/appengine/docs/standard/python/googlecloudstorageclient/functions#open

Proposition

Adds support for extended options on different operations like upload_from_string and delete, something like:

def upload_from_string(self, data, content_type='text/plain', client=None,
                           predefined_acl=None, options=None):

Which would like you pick and sets of options from https://cloud.google.com/storage/docs/xml-api/reference-headers

[tseaver]

Docs for the JSON API headers / query string parameters. Where they differ, we would be following the latter, because we never directly invoke the XML API (Blob.public_url and Blob.generate_signed_url create URLs against the XML API endpoint, but don't invoke them).

[tseaver]

Of the "standard" HTTP headers, I can see that there might be a usecase for supplying:

• Cache-Control (override default 3600 value; only supported for fetching blobs accessible to all anonymous users).
• Range (download partial content)
• If-Match / If-None-Match (check an ETag)

AFAICT, the JSON API only supports one "extension" header which is not covered / superseded by existing google-cloud-storage usage:

• X-HTTP-Method-Override (smuggle PATCH requests through obtuse firewalls)

[tseaver]

Of the the query string parameters supported by the JSON API, we might want to support:

• prettyPrint (remove whitespace from JSON responses)
• quotaUser (enforce per-user quotas from a server-side application even in cases when the user's IP address is unknown)
• userIp (enforce per-user quotas when calling the API from a server-side application.)

The others are ether superseded by the library, or are tied to deprecated authentication schemes / JSON-P.

[tseaver]

With the release of google-cloud-core 1.0.0, the Connection class now exposes an extra_headers property, which can be used to pass whatever additional headers are needed (but not on a per-request basis:

client._connection.extra_headers['X-Goog-Foo'] = 'Bar'

[mlhamel]

Thanks for the updates I'm guessing it means we can now close this issue.

[tseaver]

@mlhamel I don't think the new Connection.extra_headers support is exactly what this issue wanted: it is kind of an "always on" configuration, rather than one which takes effect per-request.

[daniellehanks]

Any update on this? I stumbled upon this needing to specify an If-None-Match header. Basically my use case is my API is a thin layer over GCS (for storing profile images), and I only want to actually read the image data and serve it to the client if the actual content is different from their cache. They will specify an If-None-Match header to my API (as they won't know anything about GCS or generation numbers, kind of the point), so I just need to essentially pass this along to GCS. I don't want to either do the extra round tripping of reading the current file to check the etag and then use the generation number from that in a subsequent request, or not use the client library.

[daniellehanks]

@tseaver Are you still a good resource for this ticket? Is this ticket still a good place to request etag conditional headers support or should I open a new ticket?

[tseaver]

@daniellehanks Adding conditional support for If-Match / If-None-Match should be a separate feature request from this issue -- implementing it would touch tons of places (pretty much every method which already has if_generation_match / if_metageneration_match would ), and It would need design (how does the user ask for the test? Should it use the etag of the resource by default? etc.)

This feature would need to be commissioned by the GCS DPE team.

/cc @frankyn, @tritone, @andrewsg .

[daniellehanks]

Thanks @tseaver. Opened https://github.com/googleapis/python-storage/issues/451.