Python type of Timestamp and Boolean schema fields

[abhi19gupta]

My Pinot table schema has fields of type Timestamp and Boolean (I believe these types were introduced after Pinot v0.7.1). When querying them using this Python client, the returned values for both of them are of the Python type str. More intuitive Python datatype IMO for them would have been datetime and bool. Can this be supported by this client?

[walterddr]

could you share the schema YAML or table config YAML of your Pinot table? i can quickly try to reproduce

[abhi19gupta]

Something like this:

{
  "schemaName": "my_table",
  "dimensionFieldSpecs": [
    {
      "name": "id",
      "dataType": "STRING"
    },
    {
      "name": "is_approved",
      "dataType": "BOOLEAN"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "createdAt",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:TIMESTAMP",
      "granularity": "1:MILLISECONDS"
    }
  ],
  "primaryKeyColumns": [
    "id"
  ]
}

[walterddr]

thanks I will try to reproduce and create a fix

[abhi19gupta]

Thanks @walterddr. Any estimate on the timeline? It will help me plan better.

[walterddr]

it seems a bit tricky. had to restructure the data type system in order for it to be reusable in both sqlalchemy and regular connection. will try to get this in by next week

[abhi19gupta]

@walterddr In the PR, I see datetime.strptime being used. It might be way faster (50x) to use ciso8601.parse_datetime instead. Could you consider that? It will be essential when fetching thousands of records.

[walterddr]

@xiangfu0 do we always return datetime column in ISO8601 format (with the 2 extra double quote)? if so we can consider blindly remove the head/tail and use the faster conversion ^

[abhi19gupta]

Bumping ^ @walterddr @xiangfu0