data loader: ignore old pd idx from multiple files

twitter / communitynotes

Documentation and source code powering Twitter's Community Notes

https://twitter.github.io/communitynotes

Apache License 2.0

1.42k stars 196 forks source link

data loader: ignore old pd idx from multiple files #180

Closed avalanchesiqi closed 9 months ago

avalanchesiqi commented 10 months ago

We should pass the option ignore_index=True when concatenating multiple pandas dataframes. It is usually ok, but if one wants to query the merged dataframe by calling the row index, having conflicted indexes is not desirable.

I add a usage comment about how to concatenate multiple rating files in main.py.