search

cubewise-code / tm1py

TM1py is a Python package that wraps the TM1 REST API in a simple to use library.
http://tm1py.readthedocs.io/en/latest/
MIT License
190 stars 109 forks source link

Element named "None" gets interpreted as 'nan' when using "execute_mdx_dataframe(..., use_blob=True)" #1151

Open bdunleavy22 opened 1 month ago

bdunleavy22 commented 1 month ago

Describe the bug When an execute_mdx_dataframe is run with use_blob=True, any element named "None" will be interpreted as NaN, instead of a string "None". This behavior is a result of lines 551/556 in 'build_dataframe_from_csv' in "Utils.py", df = pd.read_csv(StringIO(raw_csv), sep=sep, na_values=["", None], keep_default_na=False, **kwargs). If that line is changed to df = pd.read_csv(StringIO(raw_csv), sep=sep, na_values=[""], keep_default_na=False, **kwargs) (removing None from na_values), the function then behaves as expected. I don't know why, but it seems like pandas is treating None as "None"?

To Reproduce

  1. Have a cube where a dimension has an element named "None".
  2. Run the function execute_mdx_dataframe, with use_blob=True and the "None" element is included in the MDXView
  3. The element will be returned as NaN in the DataFrame.
  4. (This might also be true for cell values that are "None", I have not tested that)

Expected behavior Elements named "None" would be in the DataFrame as "None" instead of "NaN".

Version TM1py 2.0.4 TM1 Server Version: 11.8.02500.3

MariusWirtz commented 1 month ago

Good catch. Thank you for raising this, @bennybooo22.

We must not substitute None element names with nan in element columns. I will look into this