Should date.to_pandas() return datetime64?

NickCrews commented 10 months ago

What happened?

This is what currently happens on main, per a change I instigated in https://github.com/ibis-project/ibis/issues/7299:

import ibis

ibis.options.interactive = True

d = ibis.memtable({"date": ["2024-01-01", "2024-01-02"]}).cast({"date": "date"}).date
s = d.to_pandas()
print(type(s[0]))
# <class 'datetime.date'>
s
# 0    2024-01-01
# 1    2024-01-02
# Name: date, dtype: object

Perhaps this is what is intended, but this annoyingly breaking for me. I was making plots with altair as alt.Chart(table.to_pandas()) using tables with a date column. Before, the pandas DF had type datetime64. This is serializable to JSON for altair/vega, and worked. Now, the datetime.date objects are not serializable to JSON, and I get an error. I think the old behavior was better, since .to_pandas() implies to me that we are going to pandas-land, and so we should use as canonical-to-pandas-as-possible dtypes.

It seems like you really thought about the semantics of this already per some comments in that issue and linked PR, but curious if there would be a problem with these semantics:

DateScalar.to_pandas() -> pd.Timestamp
DateColumn.to_pandas() -> pd.Series[datetime64]
DateScalar.execute() -> pd.Timestamp (because by definition this uses pandas)
DateColumn.execute() -> pd.Series[datetime64] (same)
repr(Date) -> "YYYY-MM-DD" (even if under the hood this uses .to_pandas() and we have pd.Timestamps in intermediate steps)

What version of ibis are you using?

main

What backend(s) are you using, if any?

NA

Relevant log output

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

hottwaj commented 4 months ago

Just came across this too and wanted to add a vote for getting it fixed :)

mfatihaktas commented 4 months ago

@hottwaj Thanks for chiming in. The fix is implemented in this PR. It is waiting for the merge per this comment.

ibis-project / ibis