search

dedupeio / dedupe-examples

:id: Examples for using the dedupe library
MIT License
404 stars 216 forks source link

Execution got Stuck at finding the Threshold #95

Closed KCSekhar-DE closed 4 years ago

KCSekhar-DE commented 5 years ago

Hi

I'm trying to find the duplicates in a somewhat large .csv file with around 100K records. I have successfully completed the training part. After that when i try to compute the clustering threshold and call the actual match method, i don't know what's going on whether execution got stuck or still running even 20 hours.. Is this normal for a file of this size or do i need to do anything else? Please Suggest..

Here i attached a screenshot where i'm having a problem.

Capture

Any help is much appreciated. Thanks, Sekhar

KCSekhar-DE commented 5 years ago

Hello Guys,

Can you please respond to this..

Thanks, Sekhar

fgregg commented 4 years ago

threshold has been deprecated

reloaded001 commented 4 years ago

Hi,

I have had the same issue. I am using Gazetteer and that I believe uses the previous version?

reloaded001 commented 4 years ago

I am using the Postgres verison and this is where it is stuck at:


Indexing canonical records
INFO:dedupe.canopy_index:Removing stop word  b
INFO:dedupe.canopy_index:Removing stop word  s
INFO:dedupe.canopy_index:Removing stop word ab
INFO:dedupe.canopy_index:Removing stop word ar
INFO:dedupe.canopy_index:Removing stop word d
INFO:dedupe.canopy_index:Removing stop word id
INFO:dedupe.canopy_index:Removing stop word n
INFO:dedupe.canopy_index:Removing stop word sa
INFO:dedupe.canopy_index:Removing stop word us
INFO:dedupe.canopy_index:Removing stop word ha
INFO:dedupe.canopy_index:Removing stop word ri
INFO:dedupe.canopy_index:Removing stop word  k
INFO:dedupe.canopy_index:Removing stop word ee
INFO:dedupe.canopy_index:Removing stop word ka
INFO:dedupe.canopy_index:Removing stop word  a
INFO:dedupe.canopy_index:Removing stop word as
INFO:dedupe.canopy_index:Removing stop word ed
INFO:dedupe.canopy_index:Removing stop word l
INFO:dedupe.canopy_index:Removing stop word za
INFO:dedupe.canopy_index:Removing stop word  u
INFO:dedupe.canopy_index:Removing stop word a
INFO:dedupe.canopy_index:Removing stop word mo
INFO:dedupe.canopy_index:Removing stop word s
INFO:dedupe.canopy_index:Removing stop word ra
INFO:dedupe.canopy_index:Removing stop word  j
INFO:dedupe.canopy_index:Removing stop word at
INFO:dedupe.canopy_index:Removing stop word hm
INFO:dedupe.canopy_index:Removing stop word ja
INFO:dedupe.canopy_index:Removing stop word wa
INFO:dedupe.canopy_index:Removing stop word di
INFO:dedupe.canopy_index:Removing stop word iq
INFO:dedupe.canopy_index:Removing stop word si
INFO:dedupe.canopy_index:Removing stop word hi
INFO:dedupe.canopy_index:Removing stop word kh
INFO:dedupe.canopy_index:Removing stop word sh
INFO:dedupe.canopy_index:Removing stop word am
INFO:dedupe.canopy_index:Removing stop word li
INFO:dedupe.canopy_index:Removing stop word mm
INFO:dedupe.canopy_index:Removing stop word uh
INFO:dedupe.canopy_index:Removing stop word an
INFO:dedupe.canopy_index:Removing stop word af
INFO:dedupe.canopy_index:Removing stop word ul
INFO:dedupe.canopy_index:Removing stop word i
INFO:dedupe.canopy_index:Removing stop word he
INFO:dedupe.canopy_index:Removing stop word  t
INFO:dedupe.canopy_index:Removing stop word ta
INFO:dedupe.canopy_index:Removing stop word Dr
INFO:dedupe.canopy_index:Removing stop word  M
INFO:dedupe.canopy_index:Removing stop word ea
INFO:dedupe.canopy_index:Removing stop word  r
INFO:dedupe.canopy_index:Removing stop word na
INFO:dedupe.canopy_index:Removing stop word oo
INFO:dedupe.canopy_index:Removing stop word  n
INFO:dedupe.canopy_index:Removing stop word mi
INFO:dedupe.canopy_index:Removing stop word er
INFO:dedupe.canopy_index:Removing stop word ia
INFO:dedupe.canopy_index:Removing stop word Mr
INFO:dedupe.canopy_index:Removing stop word Ms
INFO:dedupe.canopy_index:Removing stop word muhammad
INFO:dedupe.canopy_index:Removing stop word khan
INFO:dedupe.canopy_index:Removing stop word ali
INFO:dedupe.canopy_index:Removing stop word Dr
INFO:dedupe.canopy_index:Removing stop word Dear
INFO:dedupe.canopy_index:Removing stop word Mrs
INFO:dedupe.blocking:10000, 9.0271942 seconds
INFO:dedupe.blocking:20000, 17.7935462 seconds
INFO:dedupe.blocking:30000, 27.7324642 seconds
INFO:dedupe.blocking:40000, 38.6244052 seconds
INFO:dedupe.blocking:50000, 48.6582412 seconds
INFO:dedupe.blocking:60000, 57.5101272 seconds
INFO:dedupe.blocking:70000, 67.4521752 seconds
INFO:dedupe.blocking:80000, 76.7485662 seconds
INFO:dedupe.blocking:90000, 85.6015492 seconds

Calculating threshold
INFO:dedupe.blocking:10000, 6.9244312 seconds
INFO:dedupe.blocking:20000, 13.5718022 seconds
INFO:dedupe.blocking:30000, 18.9501042 seconds
INFO:dedupe.blocking:40000, 23.1933072 seconds
INFO:dedupe.blocking:50000, 27.3323532 seconds
INFO:dedupe.blocking:60000, 30.5420322 seconds
INFO:dedupe.blocking:70000, 32.4935432 seconds
INFO:dedupe.blocking:80000, 34.2263532 seconds
INFO:dedupe.blocking:90000, 36.1096462 seconds
INFO:dedupe.blocking:100000, 38.9538842 seconds
INFO:dedupe.blocking:110000, 40.8626792 seconds
INFO:dedupe.blocking:120000, 42.5689852 seconds
INFO:dedupe.blocking:130000, 44.4924132 seconds
INFO:dedupe.blocking:140000, 47.0227152 seconds
INFO:dedupe.blocking:150000, 50.8561182 seconds
INFO:dedupe.blocking:160000, 56.6791892 seconds
INFO:dedupe.blocking:170000, 62.2290912 seconds
INFO:dedupe.blocking:180000, 67.6016512 seconds
INFO:dedupe.blocking:190000, 73.0131112 seconds
INFO:dedupe.blocking:200000, 78.1848122 seconds
INFO:dedupe.blocking:210000, 83.1902312 seconds
INFO:dedupe.blocking:220000, 87.7322112 seconds
INFO:dedupe.blocking:230000, 92.2850642 seconds
INFO:dedupe.blocking:240000, 96.6137802 seconds
INFO:dedupe.blocking:250000, 101.0468212 seconds
INFO:dedupe.blocking:260000, 105.4295142 seconds
INFO:dedupe.blocking:270000, 110.0581972 seconds
INFO:dedupe.blocking:280000, 115.1203402 seconds
INFO:dedupe.blocking:290000, 121.8007752 seconds
INFO:dedupe.blocking:300000, 127.1561152 seconds
INFO:dedupe.blocking:310000, 132.8270122 seconds
INFO:dedupe.blocking:320000, 137.7794102 seconds
INFO:dedupe.blocking:330000, 143.1801442 seconds
INFO:dedupe.blocking:340000, 148.9719372 seconds
INFO:dedupe.blocking:350000, 154.7131142 seconds
INFO:dedupe.blocking:360000, 157.2674552 seconds
INFO:dedupe.blocking:370000, 158.8572472 seconds
INFO:dedupe.blocking:380000, 160.3582952 seconds
INFO:dedupe.blocking:390000, 161.8701412 seconds
INFO:dedupe.blocking:400000, 165.0027082 seconds
INFO:dedupe.blocking:410000, 168.1594672 seconds
INFO:dedupe.blocking:420000, 171.0168992 seconds
INFO:dedupe.blocking:430000, 173.6541212 seconds
INFO:dedupe.blocking:440000, 176.1871382 seconds
INFO:dedupe.blocking:450000, 178.7551532 seconds
INFO:dedupe.blocking:460000, 182.5600752 seconds
INFO:dedupe.blocking:470000, 187.1910012 seconds
INFO:dedupe.blocking:480000, 192.0024852 seconds
INFO:dedupe.blocking:490000, 197.7354372 seconds
INFO:dedupe.blocking:500000, 201.5290922 seconds
INFO:dedupe.blocking:510000, 205.5159912 seconds
INFO:dedupe.blocking:520000, 210.0301002 seconds
INFO:dedupe.blocking:530000, 213.2896602 seconds
INFO:dedupe.blocking:540000, 217.5416862 seconds
INFO:dedupe.blocking:550000, 221.1548092 seconds
INFO:dedupe.blocking:560000, 225.2612652 seconds
INFO:dedupe.blocking:570000, 228.9188592 seconds
INFO:dedupe.blocking:580000, 232.4007402 seconds
INFO:dedupe.blocking:590000, 236.1616962 seconds
INFO:dedupe.blocking:600000, 239.5791542 seconds
INFO:dedupe.blocking:610000, 243.5738902 seconds
INFO:dedupe.blocking:620000, 249.7598802 seconds
INFO:dedupe.blocking:630000, 255.3383132 seconds
INFO:dedupe.blocking:640000, 258.4197382 seconds
INFO:dedupe.blocking:650000, 262.0669912 seconds
INFO:dedupe.blocking:660000, 267.7002742 seconds
INFO:dedupe.blocking:670000, 273.4454842 seconds
INFO:dedupe.blocking:680000, 279.2386922 seconds
INFO:dedupe.blocking:690000, 282.0541532 seconds
INFO:dedupe.blocking:700000, 287.6151962 seconds
INFO:dedupe.blocking:710000, 293.3538142 seconds
INFO:dedupe.blocking:720000, 298.2577752 seconds
INFO:dedupe.blocking:730000, 302.3843062 seconds
INFO:dedupe.blocking:740000, 306.6532852 seconds
INFO:dedupe.blocking:750000, 312.1278012 seconds
INFO:dedupe.blocking:760000, 317.1442192 seconds
INFO:dedupe.blocking:770000, 320.6875962 seconds
INFO:dedupe.blocking:780000, 324.0742942 seconds
INFO:dedupe.blocking:790000, 326.2218542 seconds
INFO:dedupe.blocking:800000, 331.4391002 seconds
INFO:dedupe.blocking:810000, 335.7521712 seconds
INFO:dedupe.blocking:820000, 340.0127082 seconds
INFO:dedupe.blocking:830000, 344.7962272 seconds
INFO:dedupe.blocking:840000, 350.1022512 seconds
INFO:dedupe.blocking:850000, 355.8178622 seconds
INFO:dedupe.blocking:860000, 360.4865822 seconds
INFO:dedupe.blocking:870000, 364.7321682 seconds
INFO:dedupe.blocking:880000, 369.0402952 seconds
INFO:dedupe.blocking:890000, 372.8638202 seconds
INFO:dedupe.blocking:900000, 376.8215982 seconds
INFO:dedupe.blocking:910000, 381.5600202 seconds
INFO:dedupe.blocking:920000, 386.0418362 seconds
INFO:dedupe.blocking:930000, 389.2117992 seconds
INFO:dedupe.blocking:940000, 392.0786062 seconds
INFO:dedupe.blocking:950000, 395.4065142 seconds
INFO:dedupe.blocking:960000, 398.1694702 seconds
INFO:dedupe.blocking:970000, 401.3737132 seconds
INFO:dedupe.blocking:980000, 404.6730702 seconds
INFO:dedupe.blocking:990000, 406.2671642 seconds
INFO:dedupe.blocking:1000000, 408.0282722 seconds
INFO:dedupe.blocking:1010000, 411.2678412 seconds
INFO:dedupe.blocking:1020000, 413.9649342 seconds
INFO:dedupe.blocking:1030000, 434.4980772 seconds
INFO:dedupe.blocking:1040000, 438.2868422 seconds
INFO:dedupe.blocking:1050000, 441.9858742 seconds
INFO:dedupe.blocking:1060000, 446.4614782 seconds
INFO:dedupe.blocking:1070000, 450.8633412 seconds
INFO:dedupe.blocking:1080000, 457.8175262 seconds
INFO:dedupe.blocking:1090000, 464.1444542 seconds
INFO:dedupe.blocking:1100000, 470.4915052 seconds
INFO:dedupe.blocking:1110000, 475.6536162 seconds
INFO:dedupe.blocking:1120000, 481.2985292 seconds
INFO:dedupe.blocking:1130000, 486.5487192 seconds
INFO:dedupe.blocking:1140000, 491.2621332 seconds
INFO:dedupe.blocking:1150000, 496.4464982 seconds
INFO:dedupe.blocking:1160000, 501.7356642 seconds
INFO:dedupe.blocking:1170000, 507.1197092 seconds
INFO:dedupe.blocking:1180000, 512.2594682 seconds
INFO:dedupe.blocking:1190000, 516.4970872 seconds
INFO:dedupe.blocking:1200000, 522.1960732 seconds
INFO:dedupe.blocking:1210000, 530.4908882 seconds
INFO:dedupe.blocking:1220000, 537.8464432 seconds
INFO:dedupe.blocking:1230000, 543.8597942 seconds
INFO:dedupe.blocking:1240000, 547.2332302 seconds
INFO:dedupe.blocking:1250000, 551.1092292 seconds
INFO:dedupe.blocking:1260000, 555.3259522 seconds
INFO:dedupe.blocking:1270000, 559.1861852 seconds
INFO:dedupe.blocking:1280000, 563.2216132 seconds
INFO:dedupe.blocking:1290000, 567.5269212 seconds
INFO:dedupe.blocking:1300000, 572.4622092 seconds
INFO:dedupe.blocking:1310000, 577.1467332 seconds
INFO:dedupe.blocking:1320000, 580.9615662 seconds
INFO:dedupe.blocking:1330000, 584.4497222 seconds
INFO:dedupe.blocking:1340000, 587.4754462 seconds
INFO:dedupe.blocking:1350000, 590.7758922 seconds
INFO:dedupe.blocking:1360000, 594.3111892 seconds
INFO:dedupe.blocking:1370000, 598.1856242 seconds
INFO:dedupe.blocking:1380000, 602.4337902 seconds
INFO:dedupe.blocking:1390000, 607.7458662 seconds
INFO:dedupe.blocking:1400000, 611.0156562 seconds
INFO:dedupe.blocking:1410000, 613.9616272 seconds
INFO:dedupe.blocking:1420000, 620.5273172 seconds
INFO:dedupe.blocking:1430000, 627.9955522 seconds
INFO:dedupe.blocking:1440000, 635.5040112 seconds
INFO:dedupe.blocking:1450000, 642.1905662 seconds
INFO:dedupe.blocking:1460000, 648.7694922 seconds
INFO:dedupe.blocking:1470000, 654.8877882 seconds
INFO:dedupe.blocking:1480000, 682.2895212 seconds
WARNING:dedupe.backport:Dedupe does not currently support multiprocessing on Windows
INFO:dedupe.api:Maximum expected recall and precision
INFO:dedupe.api:recall: 0.724
INFO:dedupe.api:precision: 0.532
INFO:dedupe.api:With threshold: 0.281

Matching messy records
INFO:dedupe.blocking:10000, 7.8709082 seconds
INFO:dedupe.blocking:20000, 13.4321382 seconds
INFO:dedupe.blocking:30000, 18.7797442 seconds
INFO:dedupe.blocking:40000, 24.2510142 seconds
INFO:dedupe.blocking:50000, 29.5264042 seconds
INFO:dedupe.blocking:60000, 33.1379392 seconds
INFO:dedupe.blocking:70000, 35.2007022 seconds
INFO:dedupe.blocking:80000, 36.9202822 seconds
INFO:dedupe.blocking:90000, 38.8237252 seconds
INFO:dedupe.blocking:100000, 40.7261622 seconds
INFO:dedupe.blocking:110000, 42.6701212 seconds
INFO:dedupe.blocking:120000, 44.3606532 seconds
INFO:dedupe.blocking:130000, 46.2159692 seconds
INFO:dedupe.blocking:140000, 49.1121632 seconds
INFO:dedupe.blocking:150000, 53.2538112 seconds
INFO:dedupe.blocking:160000, 60.4661152 seconds
INFO:dedupe.blocking:170000, 67.2778642 seconds
INFO:dedupe.blocking:180000, 73.6361242 seconds
INFO:dedupe.blocking:190000, 80.2449022 seconds
INFO:dedupe.blocking:200000, 87.0532062 seconds
INFO:dedupe.blocking:210000, 95.0590052 seconds
INFO:dedupe.blocking:220000, 101.5438152 seconds
INFO:dedupe.blocking:230000, 108.5141542 seconds
INFO:dedupe.blocking:240000, 115.1324742 seconds
INFO:dedupe.blocking:250000, 121.5100822 seconds
INFO:dedupe.blocking:260000, 127.1068162 seconds
INFO:dedupe.blocking:270000, 133.3162322 seconds
INFO:dedupe.blocking:280000, 139.0805372 seconds
INFO:dedupe.blocking:290000, 144.8649092 seconds
INFO:dedupe.blocking:300000, 150.8455812 seconds
INFO:dedupe.blocking:310000, 157.1281442 seconds
INFO:dedupe.blocking:320000, 162.4728292 seconds
INFO:dedupe.blocking:330000, 168.3232902 seconds
INFO:dedupe.blocking:340000, 174.5672992 seconds
INFO:dedupe.blocking:350000, 180.6697582 seconds
INFO:dedupe.blocking:360000, 183.3207382 seconds
INFO:dedupe.blocking:370000, 184.9963402 seconds
INFO:dedupe.blocking:380000, 186.6922042 seconds
INFO:dedupe.blocking:390000, 188.4882502 seconds
INFO:dedupe.blocking:400000, 192.0352942 seconds
INFO:dedupe.blocking:410000, 196.4640042 seconds
INFO:dedupe.blocking:420000, 200.3756672 seconds
INFO:dedupe.blocking:430000, 203.5970312 seconds
INFO:dedupe.blocking:440000, 206.6204132 seconds
INFO:dedupe.blocking:450000, 209.6790472 seconds
INFO:dedupe.blocking:460000, 214.1764572 seconds
INFO:dedupe.blocking:470000, 219.3674642 seconds
INFO:dedupe.blocking:480000, 224.6278232 seconds
INFO:dedupe.blocking:490000, 230.8908742 seconds
INFO:dedupe.blocking:500000, 235.0349692 seconds
INFO:dedupe.blocking:510000, 239.7076002 seconds
INFO:dedupe.blocking:520000, 244.9303302 seconds
INFO:dedupe.blocking:530000, 248.7779062 seconds
INFO:dedupe.blocking:540000, 253.4292612 seconds
INFO:dedupe.blocking:550000, 257.6719492 seconds
INFO:dedupe.blocking:560000, 261.8226212 seconds
INFO:dedupe.blocking:570000, 265.9076292 seconds
INFO:dedupe.blocking:580000, 270.0852012 seconds
INFO:dedupe.blocking:590000, 274.3342192 seconds
INFO:dedupe.blocking:600000, 278.3451932 seconds
INFO:dedupe.blocking:610000, 283.0454912 seconds
INFO:dedupe.blocking:620000, 288.9097862 seconds
INFO:dedupe.blocking:630000, 295.3240582 seconds
INFO:dedupe.blocking:640000, 298.7416032 seconds
INFO:dedupe.blocking:650000, 302.8786612 seconds
INFO:dedupe.blocking:660000, 308.9905672 seconds
INFO:dedupe.blocking:670000, 317.2533352 seconds
INFO:dedupe.blocking:680000, 324.4388532 seconds
INFO:dedupe.blocking:690000, 327.5854242 seconds
INFO:dedupe.blocking:700000, 333.7813662 seconds
INFO:dedupe.blocking:710000, 340.4150512 seconds
INFO:dedupe.blocking:720000, 346.9384202 seconds
INFO:dedupe.blocking:730000, 352.0812012 seconds
INFO:dedupe.blocking:740000, 357.4439822 seconds
INFO:dedupe.blocking:750000, 364.5146912 seconds
INFO:dedupe.blocking:760000, 370.3431772 seconds
INFO:dedupe.blocking:770000, 374.7052442 seconds
INFO:dedupe.blocking:780000, 378.8440352 seconds
INFO:dedupe.blocking:790000, 381.4762042 seconds
INFO:dedupe.blocking:800000, 387.7562082 seconds
INFO:dedupe.blocking:810000, 392.9177762 seconds
INFO:dedupe.blocking:820000, 398.0973962 seconds
INFO:dedupe.blocking:830000, 403.9359342 seconds
INFO:dedupe.blocking:840000, 410.6431172 seconds
INFO:dedupe.blocking:850000, 417.5976012 seconds
INFO:dedupe.blocking:860000, 423.1830052 seconds
INFO:dedupe.blocking:870000, 428.0491572 seconds
INFO:dedupe.blocking:880000, 433.3516442 seconds
INFO:dedupe.blocking:890000, 437.9899032 seconds
INFO:dedupe.blocking:900000, 442.8309402 seconds
INFO:dedupe.blocking:910000, 448.6472592 seconds
INFO:dedupe.blocking:920000, 454.0946682 seconds
INFO:dedupe.blocking:930000, 457.9243942 seconds
INFO:dedupe.blocking:940000, 461.9029512 seconds
INFO:dedupe.blocking:950000, 465.9984052 seconds
INFO:dedupe.blocking:960000, 469.4141892 seconds
INFO:dedupe.blocking:970000, 473.7421442 seconds
INFO:dedupe.blocking:980000, 477.4075812 seconds
INFO:dedupe.blocking:990000, 479.4579932 seconds
INFO:dedupe.blocking:1000000, 481.6435812 seconds
INFO:dedupe.blocking:1010000, 484.9307262 seconds
INFO:dedupe.blocking:1020000, 487.9485532 seconds
INFO:dedupe.blocking:1030000, 492.8688682 seconds
INFO:dedupe.blocking:1040000, 497.5001292 seconds
INFO:dedupe.blocking:1050000, 501.9063912 seconds
INFO:dedupe.blocking:1060000, 507.0508702 seconds
INFO:dedupe.blocking:1070000, 511.8384522 seconds
INFO:dedupe.blocking:1080000, 518.9712352 seconds
INFO:dedupe.blocking:1090000, 526.1780242 seconds
INFO:dedupe.blocking:1100000, 531.2930502 seconds
INFO:dedupe.blocking:1110000, 536.6526362 seconds
INFO:dedupe.blocking:1120000, 542.4947332 seconds
INFO:dedupe.blocking:1130000, 547.9720092 seconds
INFO:dedupe.blocking:1140000, 553.4121722 seconds
INFO:dedupe.blocking:1150000, 562.3748552 seconds
INFO:dedupe.blocking:1160000, 586.8946162 seconds
INFO:dedupe.blocking:1170000, 599.4329082 seconds
INFO:dedupe.blocking:1180000, 609.5948672 seconds
INFO:dedupe.blocking:1190000, 619.5129082 seconds
INFO:dedupe.blocking:1200000, 630.1360432 seconds
INFO:dedupe.blocking:1210000, 645.9481412 seconds
INFO:dedupe.blocking:1220000, 659.3619532 seconds
INFO:dedupe.blocking:1230000, 678.1534542 seconds
INFO:dedupe.blocking:1240000, 685.3737312 seconds
INFO:dedupe.blocking:1250000, 693.0808252 seconds
INFO:dedupe.blocking:1260000, 700.3726952 seconds
INFO:dedupe.blocking:1270000, 708.2382862 seconds
INFO:dedupe.blocking:1280000, 716.5473182 seconds
INFO:dedupe.blocking:1290000, 724.8316072 seconds
INFO:dedupe.blocking:1300000, 732.7601362 seconds
INFO:dedupe.blocking:1310000, 741.5343892 seconds
INFO:dedupe.blocking:1320000, 749.6605602 seconds
INFO:dedupe.blocking:1330000, 756.2051332 seconds
INFO:dedupe.blocking:1340000, 762.4327952 seconds
INFO:dedupe.blocking:1350000, 769.1101752 seconds
INFO:dedupe.blocking:1360000, 777.3280412 seconds
INFO:dedupe.blocking:1370000, 785.7166692 seconds
INFO:dedupe.blocking:1380000, 795.5288042 seconds
INFO:dedupe.blocking:1390000, 807.8985402 seconds
INFO:dedupe.blocking:1400000, 814.3149352 seconds
INFO:dedupe.blocking:1410000, 821.1073662 seconds
INFO:dedupe.blocking:1420000, 833.7249562 seconds
INFO:dedupe.blocking:1430000, 847.4266372 seconds
INFO:dedupe.blocking:1440000, 861.4583152 seconds
INFO:dedupe.blocking:1450000, 873.0963422 seconds
INFO:dedupe.blocking:1460000, 886.7989452 seconds
INFO:dedupe.blocking:1470000, 896.6543402 seconds
INFO:dedupe.blocking:1480000, 908.1049692 seconds