Open mudasirmirza opened 2 months ago
@mudasirmirza Hello! I'm here to assist you with any issues you're facing. Let's work together to resolve the problem.
The error 'str' object has no attribute 'tzinfo'
occurs because the code expects a datetime object with timezone information but encounters a string instead. To resolve this, you need to ensure that datetime objects in your data have the tzinfo
attribute set correctly.
Here are the steps to fix this issue:
Preprocess Data to Ensure Timezone Information: Convert string representations of datetime to actual datetime objects with timezone information before they are processed. You can modify the data preprocessing step as follows:
from datetime import datetime
import pytz
def preprocess_data(data):
for row in data:
for i, value in enumerate(row):
if isinstance(value, str):
try:
# Attempt to parse the string as a datetime object
dt = datetime.fromisoformat(value)
# Add timezone information if missing
if dt.tzinfo is None:
dt = dt.replace(tzinfo=pytz.UTC)
row[i] = dt
except ValueError:
# If parsing fails, keep the original string
pass
return data
# Example usage
data = [
("2023-10-01T12:00:00", "some other value"),
("2023-10-02T13:00:00+00:00", "another value"),
]
preprocessed_data = preprocess_data(data)
Ensure convert_dttm
Method Handles Datetime Objects Correctly:
The convert_dttm
method in the ElasticSearchEngineSpec
class should convert datetime objects to the appropriate string format for Elasticsearch queries. Here is the relevant implementation:
class ElasticSearchEngineSpec(BaseEngineSpec): # pylint: disable=abstract-method
engine = "elasticsearch"
engine_name = "ElasticSearch (SQL API)"
time_groupby_inline = True
allows_joins = False
allows_subqueries = True
allows_sql_comments = False
@classmethod
def convert_dttm(
cls, target_type: str, dttm: datetime, db_extra: Optional[dict[str, Any]] = None
) -> Optional[str]:
db_extra = db_extra or {}
sqla_type = cls.get_sqla_column_type(target_type)
if isinstance(sqla_type, types.DateTime):
es_version = db_extra.get("version")
supports_dttm_parse = False
try:
if es_version:
supports_dttm_parse = Version(es_version) >= Version("7.8")
except Exception as ex: # pylint: disable=broad-except
logger.error("Unexpected error while convert es_version", exc_info=True)
logger.exception(ex)
if supports_dttm_parse:
datetime_formatted = dttm.isoformat(sep=" ", timespec="seconds")
return (
f"""DATETIME_PARSE('{datetime_formatted}', 'yyyy-MM-dd HH:mm:ss')"""
)
return f"""CAST('{dttm.isoformat(timespec="seconds")}' AS DATETIME)"""
return None
By ensuring that your datetime objects have the tzinfo
attribute set and are correctly formatted for Elasticsearch queries, you can avoid the 'str' object has no attribute 'tzinfo'
error when querying an Elasticsearch data source in Apache Superset [1][2].
To continue talking to Dosu, mention @dosu.
Just to add to it. If I run the same query directly selecting "ElasticSearch" as the data source and selecting the "Index"
SELECT
customer_id as CustomerID,
customer_name as CustomerName
FROM
"customer_availability"
This query returns proper results.
Hello, were you able to solve the above issue? I am also running into the same problem when trying to get elasticsearch table data using superset meta db. The query works fine when using "ElasticSearch" as a datasource. Let me know if you were able to solve it and how?
Hi @hardikjain178 ,
I was unable to resolve this and because of this I just dropped working on this.
Bug description
I have added 2 data source. One ElaticSearch (named as es) and one MySQL (named as mysql). I have enabled the "querying across database" feature using the feature FLAG as described in "https://superset.apache.org/docs/configuration/databases/#querying-across-databases" and named the meta db as "superset_meta"
Now when I go to sqllab and select "superset_meta" and then a simple query like below
I get below error
And in the docker-compose logs, I see below
In the logs, I can clearly see that it fetched the data but for some reason it is trying to find
tzinfo
in the fetchedjson
from the elasticsearch data source.Also in the logs, when it stared to process this query, it made a call like below
And after this I see the log showing the fetched columns.
I am not sure why is it still trying to look for
tzinfo
.Please let me know if any more information is required from my side related to this issue.
How to reproduce the bug
ENABLE_SUPERSET_META_DB
insuperset_config.py
elasticsearch-dbapi
indocker/requirements-local.txt
docker compose -f docker-compose-image-tag.yml up
superset
meta dbScreenshots/recordings
No response
Superset version
master / latest-dev
Python version
3.9
Node version
18 or greater
Browser
Chrome
Additional context
No response
Checklist