deduplicate bug in Spark in case of a null column.

[clintf1982]

Describe the bug

Same bug as https://github.com/dbt-labs/dbt-utils/issues/713 however for Spark When there is a null column(Unrelated to the partition by or order by columns), Spark doesn't return the expected rows in the deduplicate default function.

Steps to reproduce

*** This is python code where the deduplication code was copied to in order to reproduce the bug.

  sql = """
  with relation as (
  select * from ( values (1, 3, null), (1, 2, null), (1, 1, null)) 
  ), row_numbered as (
      select
          _inner.*,
          row_number() over (
              partition by col1
              order by col2
          ) as rn
      from relation as _inner
  )

  select
      distinct data.*
  from relation as data
  natural join row_numbered
  where row_numbered.rn = 1
  """
  df = session.sql(sql)
  df.show()

Expected results

+----+----+----+ |col1|col2|col3| +----+----+----+ | 1| 1| null | +----+----+----+

Actual results

+----+----+----+ |col1|col2|col3| +----+----+----+ +----+----+----+

Screenshots and log output

System information

The contents of your packages.yml file:

Which database are you using dbt with?

• [ ] postgres
• [ ] redshift
• [ ] bigquery
• [ ] snowflake
• [x] other (specify: Spark)

The output of dbt --version:

Core:
  - installed: 1.6.0-b8
  - latest:    1.5.3    - Ahead of latest version!

Plugins:

Additional context

Are you interested in contributing the fix?

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions[bot]]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.