test_append_behavour:test should only have 2 fields from df2
{b'col_a': b'324', b'col_c': b'423'}
Please note, when key collision happens and SaveMode.Append is set, the former row is replaced with a new one.
According to the docs, the row of df1 should be replace by df2 in append mode, because they share the same key.
However, the col_a from df1 is still there after append, that means not entire row is replaced. We only replace the field if there is any key collision.
Hi @tsekityam ,
the SaveMode.Append uses hmset command internally, so it may not completely overwrite the row if the scheme of new dataframe is different. You are right, the documentation is not accurate.
What did I do
What did I see
test_append_behavour:test
now has 3 fieldsWhat did I expect
test_append_behavour:test
should only have 2 fields fromdf2
According to the docs, the row of
df1
should be replace bydf2
in append mode, because they share the same key.However, the
col_a
fromdf1
is still there after append, that means not entire row is replaced. We only replace the field if there is any key collision.