Fix nan case for ingest_dataframe

istranical commented 3 months ago

🚀 🚀 Pull Request

Impact

[x] Bug fix (non-breaking change which fixes expected existing functionality)
[ ] Enhancement/New feature (adds functionality without impacting existing logic)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description

-- Replaces nans in the dataframe so ingestion does not fail for cases when there are strings and nans in the same column. In general, since Deep Lake doesn't support nan explicitly, it's best to replace these values with None in all cases, even if it could work for arrays in some cases.

Things to be aware of

Things to worry about

-- How much is the slowdown for replacing nans in large dataframes

Additional Context

CLAassistant commented 3 months ago

All committers have signed the CLA.

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.96%. Comparing base (5004257) to head (70a84a5).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #2800 +/- ## ========================================== - Coverage 84.50% 83.96% -0.55% ========================================== Files 235 235 Lines 26527 26528 +1 ========================================== - Hits 22417 22273 -144 - Misses 4110 4255 +145 ``` | [Flag](https://app.codecov.io/gh/activeloopai/deeplake/pull/2800/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=activeloopai) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/activeloopai/deeplake/pull/2800/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=activeloopai) | `83.96% <100.00%> (-0.55%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=activeloopai#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.