search

jim256 / ebay

0 stars 0 forks source link

Question: database reads #25

Closed jim256 closed 4 years ago

jim256 commented 4 years ago

Since I constantly have multiple writes, updates, and reads to the db from 5-6 different processes, I want to make sure to minimize the chance for deadlocks as much as possible. I recently found that one of my original spiders was locking the table each time it read from it to to see if a listing already existed. Does the eBay spider lock the table for reads?

james-carpenter commented 4 years ago

The eBay spider is written not to check whether a record exists. Instead it attempts the insert, and if that fails because of a unique constraint, it falls back to an update instead. This is more scalable and minimizes the duration of a transaction that has the opportunity for database locks. Also, all queries are qualified to the source so other sources performing updates are not impacted.

jim256 commented 4 years ago

Thanks for the clarification.