fix: handle null values in data

[alespour]

Closes #621

Proposed Changes

Handles data with missing values when querying to data frames. The query functions query_data_frame... have new optional parameter use_extension_dtypes.

def query_data_frame(self, query: str, org=None, data_frame_index: List[str] = None, params: dict = None,
                     use_extension_dtypes: bool = False):
    ...

def query_data_frame_stream(self, query: str, org=None, data_frame_index: List[str] = None, params: dict = None,
                     use_extension_dtypes: bool = False):
    ...

• when set to True, missing values are represented as pandas.NA and dtype of columns containing <NA> is of corresponding nullable extension dtypes from pandas package (ie. Int64, Float64,Boolean etc). Missing value can be checked using pandas.isna() function.
• when False (default), missing values are represented as None, and dtype of columns with missing values is either 'object' or 'float64' when type of values is numeric. This is a standard conversion behavior of data frames, see
  • https://note.nkmk.me/en/python-pandas-nan-none-na/
  • https://pandas.pydata.org/docs/user_guide/missing_data.html

Example output (with data from #621):

use_extension_dtypes=True

<bound method NDFrame.head of     result  table                    _start                     _stop                            _time _measurement  test_double  test_long
0  _result      0 2023-12-15 13:19:54+00:00 2023-12-15 13:19:57+00:00 2023-12-15 13:19:55.372000+00:00         test          4.0       <NA>
1  _result      0 2023-12-15 13:19:54+00:00 2023-12-15 13:19:57+00:00        2023-12-15 13:19:56+00:00         test         <NA>          1>

use_extension_dtypes=False

<bound method NDFrame.head of     result  table                    _start                     _stop                            _time _measurement  test_double  test_long
0  _result      0 2023-12-15 13:19:54+00:00 2023-12-15 13:19:57+00:00 2023-12-15 13:19:55.372000+00:00         test          4.0        NaN
1  _result      0 2023-12-15 13:19:54+00:00 2023-12-15 13:19:57+00:00        2023-12-15 13:19:56+00:00         test          NaN        1.0>

Note: the conversion of numeric values to extension dtypes works properly with pandas>=2.0, so in Python 3.7 environment, where the latest available pandas is 1.3.5, dtype of columns with NA values is 'object' ie. same as without the use extension types. For Python 3.8+, pandas 2.x is available.

Checklist

• [x] CHANGELOG.md updated
• [x] Rebased/mergeable
• [x] A test has been added if appropriate
• [x] pytest tests completes successfully
• [x] Commit messages are conventional

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (27777d1) 90.19% compared to head (17ab3b1) 90.40%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #636 +/- ## ========================================== + Coverage 90.19% 90.40% +0.21% ========================================== Files 39 39 Lines 3467 3503 +36 ========================================== + Hits 3127 3167 +40 + Misses 340 336 -4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.