search

scrapinghub / frontera

A scalable frontier for web crawlers
BSD 3-Clause "New" or "Revised" License
1.29k stars 216 forks source link

Fix sqlalchemy queue component partition_id #353

Open jpbalarini opened 5 years ago

jpbalarini commented 5 years ago

I found that the partition id was always set to 0 when disabling the QUEUE_HOSTNAME_PARTITIONING setting, using a sqlalchemy backend. The problem was that the schedule method on the Queue component was always using the hostname as the partitioning key (independent of the QUEUE_HOSTNAME_PARTITIONING setting).

Also added a quick description of the QUEUE_HOSTNAME_PARTITIONING setting for the documentation (since it was missing).

Thanks!

icapurro commented 5 years ago

Thanks for this fix @jpbalarini !

jpbalarini commented 5 years ago

@sibiryakov what do you think? Thanks!

jpbalarini commented 5 years ago

@sibiryakov what do you think of these changes? Adding them will fix the case when someone disables hostname partitioning (my case)