Open jday1 opened 8 months ago
cc @phofl
take
pandas does not support NumPy fixed length strings. Does this exhibit the same issue when you use the object dtype?
Using 'object' dtype on the array creation works correctly. However pandas 2.2.1 brought a change to how 'S' dtype arrays were handled during Index creation, no longer automatically converting them to 'object' dtype. Shouldn't this functionality still be implemented?
Is that behavior documented somewhere? if not then no...I'd say its not worth doing too much here. We have never offered support for fixed length NumPy strings. object
has been the historically supported data type, but we also now have "string", "string[pyarrow]" and "string[pyarrow_numpy]` and maybe even a NumPy native string in the future. I don't see fixed length string support as something worth additionally investing in
it should convert automatically to object, you should never end up with numpy fixed length strings
In the commit previously mentioned in the issue description, the if statement:
elif isinstance(values, ABCSeries):
return values._values
was added to this function in the file pandas/core/common.py
def asarray_tuplesafe(values: Iterable, dtype: NpDtype | None = None) -> ArrayLike:
if not (isinstance(values, (list, tuple)) or hasattr(values, "__array__")):
values = list(values)
elif isinstance(values, ABCIndex):
return values._values
elif isinstance(values, ABCSeries):
return values._values
if isinstance(values, list) and dtype in [np.object_, object]:
return construct_1d_object_array_from_listlike(values)
try:
with warnings.catch_warnings():
# Can remove warning filter once NumPy 1.24 is min version
if not np_version_gte1p24:
warnings.simplefilter("ignore", np.VisibleDeprecationWarning)
result = np.asarray(values, dtype=dtype)
except ValueError:
# Using try/except since it's more performant than checking is_list_like
# over each element
# error: Argument 1 to "construct_1d_object_array_from_listlike"
# has incompatible type "Iterable[Any]"; expected "Sized"
return construct_1d_object_array_from_listlike(values) # type: ignore[arg-type]
if issubclass(result.dtype.type, str):
result = np.asarray(values, dtype=object)
if result.ndim == 2:
# Avoid building an array of arrays:
values = [tuple(x) for x in values]
result = construct_1d_object_array_from_listlike(values)
return result
It seems that with the addition of that if statement, any variable values that is of the type series will result in a true statement and return values._values without checking its content , this way not creating an array with the dtype object as it used to. Would something like this be an acceptable solution?.
elif isinstance(values, ABCSeries) and values._values.dtype.kind != "S"
return values._values
Pandas version checks
[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pandas.
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
The issue shown in the provided code is that when attempting to set the index of the Pandas DataFrame using
df.set_index("fruit", inplace=True)
, aNotImplementedError
is raised with the message|S6
.The reason for this error is that the dtype
|S6
is not supported as an index type in Pandas. When you create a NumPy array with a dtype of fixed-length strings using np.array() and attempt to set it as an index in a DataFrame, Pandas tries to convert it to a suitable index type, but|S6
is not recognized as a valid option for the index.This is applicable to all
S
dtypes. The offending logic was introduced in pandas 2.2.1 by these lines of this commit: https://github.com/pandas-dev/pandas/commit/b6fb90574631c84f19f2dbdc68c26d6ce97446b4#diff-fb8a9c322624b0777f3ff7e3ef8320d746b15a2a0b80b7cab3dfbe2e12e06daaR239-R240The code works as expected in pandas 2.2.0.
Expected Behavior
What happens when using 2.2.0:
Prints:
Installed Versions