fhamborg / news-please

news-please - an integrated web crawler and information extractor for news that just works
Apache License 2.0
1.99k stars 414 forks source link

Add unique constraint on column `url` on table `CurrentVersions` in Postgres pipeline #268

Closed anteverse closed 1 week ago

anteverse commented 2 weeks ago

Hello, we've noticed performances issues when running lots of urls with the Postgres pipeline.

We would like to add a unique constraint on url column since:

anteverse commented 1 week ago

@fhamborg Any opinion on that? I feel like we should never have duplicate urls in CurrentVersions table

fhamborg commented 1 week ago

thanks for the pr!

anteverse commented 1 week ago

Thanks for the quick merge!