Minor type variations

[hadley]

When comparing the output of bigrquerystorage to bigrquery, I see a few differences for this query:

SELECT
    '\U0001f603' as unicode,
    datetime,
    TRUE as logicaltrue,
    FALSE as logicalfalse,
    CAST ('Hi' as BYTES) as bytes,
    CAST (datetime as DATE) as date,
    CAST (datetime as TIME) as time,
    CAST (datetime as TIMESTAMP) as timestamp,
    ST_GEOGFROMTEXT('POINT (30 10)') as geography
 FROM (SELECT DATETIME '2000-01-02 03:04:05.67' as datetime)

• bytes is arrow_binary instead of blob_blob. I don't think this is important since they have the same underlying data representation.
• timestamp lacks the UTC timezone — I think this is a bug?
• geography is not parsed. Probably not that important unless you really want to support it here, but I'll need to document.

[meztez]

FMI:

Documented limitations: https://cloud.google.com/bigquery/docs/reference/storage/#arrow_schema_details

• datetime has no timezone, timestamp should have one? If not, bug like mentionned.
• geography are returned as utf8, can be parsed with wk::wkt.

That might have something to do with arrow? There is a test for the geography here: https://github.com/meztez/bigrquerystorage/blob/1daba8945b4e1c37a6984927f05857b33e18191c/tests/testthat/test-integration.R#L176

To myself : Make sure it is a feature, not a bug.

[hadley]

Ah yeah, even if it has no timezone in arrow, no timezone in R means "current timezone", which I don't think is what you want. IMO UTC will give the closest behaviour. (What you really want is a naive time, https://clock.r-lib.org/articles/motivations.html#naive-time, but base R doesn't have that.)

[meztez]

FMI

There is an as_tibble parameter to the bqs_table_download method that calls a parse_postprocess method. There might be something to be done there.

https://github.com/meztez/bigrquerystorage/blob/1daba8945b4e1c37a6984927f05857b33e18191c/R/bqs_download.R#L126

[meztez]

Would you like postprocessing in bigrquerystorage to:

• Set timezone on datetime
• Convert GEOGRAPHY via wk
• Convert bytes to blob_blob

I would convert based on table field metadata.

[hadley]

If you don't mind, I think that would be optimal because it ensures that there's zero cost to switching to the arrow based API.

[meztez]

Well, that was something. Give it a go. #56