Mortgage Fix - Githubissues

Not all mortgages are being downloaded.

~~Add feature to parse all, but order based on those that have been not parsed for the longest amount of time~~ Not MVP, not adding this feature.

[ ] Set filter to pull down all mortgage results again before closing this ticket

Revisit:

0930152010- no mortgage 0936126003 - no mortgage 0935200076 - saved 0930153003 - no mortgage 0930153012- no mortgage 0532300009 - no mortgage 0923100020- no mortgage 0531200004- no mortgage 0525202001 - no mortgage 0527127017 - no mortgage 0533200003 - no mortgage 1330451007 - no mortgage 0519200025 - no mortgage

Reached a situation where most recent item was returning false on all searches. Suspect this is because the token is expiring. Code this and find a slution

@walinchus Assigning this ticket to you -- if you agree that it looks like we've fixed this bug, would you mind closing this ticket?

Sure. It is running now and it looks like it is catching most of them although every once in a while it's not so I will update you when it finishes.

Awesome, sounds good! I'll be here.

And yeah, in testing, I noticed every once in a while I'd encounter a property that didn't have a mortgage associated with it. It's possible some of them are like that!

Also, a piece of context I forgot to mention: typically the token expires after about an hour or so. So once we make it past that, we'll be getting more than before.

El sáb., 12 de ene. de 2019 a la(s) 18:06, Lucia Walinchus ( notifications@github.com) escribió:

Sure. It is running now and it looks like it is catching most of them although every once in a while it's not so I will update you when it finishes.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ludstuen90/property_project_ohio/issues/112#issuecomment-453780968, or mute the thread https://github.com/notifications/unsubscribe-auth/AGgHK4YfswgLSEeNrKdsQC6spMSuxiEUks5vCk46gaJpZM4ZyEf1 .

-- Lukas Udstuen http://www.LukasUdstuen.com/ http://www.lukasudstuen.com/

Lukas.Udstuen@gmail.com @LukasUdstuen Cell: +54 9 11 2387-0032

Script stopped execution with the following error:

(env) Daddy-Mac-418:property_project_ohio Lucia$ python manage.py warren-mort
No mortgage identified on account number:  2102064
Found mortgage date of $ 12656.79  date  2003-08-27 03:37:55  on account number:  3509168
Found mortgage date of $ 55000.0  date  2015-01-29 10:39:47  on account number:  200191
Found mortgage date of $ 172000.0  date  2018-07-24 02:45:11  on account number:  506419
Found mortgage date of $ 116000.0  date  2009-08-05 03:16:03  on account number:  2632510
Found mortgage date of $ 25319.0  date  2017-09-22 11:22:03  on account number:  820620
Found mortgage date of $ 428004.85  date  2016-12-14 11:59:53  on account number:  2141329
Found mortgage date of $ 80000.0  date  2012-02-16 11:02:52  on account number:  1405519
No mortgage identified on account number:  2631899
Found mortgage date of $ 192000.0  date  2014-11-21 03:46:58  on account number:  7623305
Found mortgage date of $ 163725.0  date  2013-04-04 08:55:11  on account number:  6150879
No mortgage identified on account number:  2100231
Found mortgage date of $ 159579.0  date  2016-11-04 10:59:58  on account number:  7632339
Found mortgage date of $ 239000.0  date  2016-05-23 01:24:53  on account number:  2630387
Found mortgage date of $ 400000.0  date  2018-11-13 12:42:49  on account number:  931276
Found mortgage date of $ 157590.0  date  2012-04-17 01:37:23  on account number:  7700016
Found mortgage date of $ 146500.0  date  2012-11-06 02:33:30  on account number:  1132941
Found mortgage date of $ 304330.0  date  2018-05-03 11:46:32  on account number:  5333078
Found mortgage date of $ 79100.0  date  2014-08-21 01:43:47  on account number:  5306451
Found mortgage date of $ 215000.0  date  2015-04-06 04:38:47  on account number:  1126454
Found mortgage date of $ 88000.0  date  2012-09-24 09:22:16  on account number:  1820153
Found mortgage date of $ 158332.0  date  2018-11-29 03:04:26  on account number:  1619608
No mortgage identified on account number:  5316065
Found mortgage date of $ 35000.0  date  2014-05-22 01:28:00  on account number:  613395
Found mortgage date of $ 410000.0  date  2013-02-07 09:43:04  on account number:  5331338
Found mortgage date of $ 174775.0  date  2018-08-20 10:52:02  on account number:  516511
Found mortgage date of $ 140200.0  date  2013-07-10 02:17:16  on account number:  910414
Found mortgage date of $ 145000.0  date  2011-10-06 10:46:28  on account number:  1000331
Found mortgage date of $ 125000.0  date  2016-10-11 08:18:43  on account number:  1114484
Found mortgage date of $ 135600.0  date  2011-06-20 12:34:40  on account number:  2631813
Found mortgage date of $ 34500.0  date  2014-07-28 01:58:25  on account number:  942692
Found mortgage date of $ 365700.0  date  2018-08-14 12:52:42  on account number:  1136740
Found mortgage date of $ 372000.0  date  2018-01-04 10:47:21  on account number:  941419
Found mortgage date of $ 40000.0  date  2018-11-16 11:04:56  on account number:  536407
Found mortgage date of $ 361570.0  date  2017-10-24 02:36:23  on account number:  1202801
Found mortgage date of $ 1440000.0  date  2017-05-12 03:53:01  on account number:  6115691
Found mortgage date of $ 4350000.0  date  2012-02-08 03:04:08  on account number:  5500125
Found mortgage date of $ 332000.0  date  2016-03-08 11:04:24  on account number:  1129577
Found mortgage date of $ 90000.0  date  2013-05-07 03:01:31  on account number:  550122
Found mortgage date of $ 50000.0  date  2018-12-19 07:51:14  on account number:  5701507
Found mortgage date of $ 128000.0  date  2007-07-31 08:36:22  on account number:  801003
Found mortgage date of $ 70000.0  date  2018-08-15 03:59:40  on account number:  613905
Found mortgage date of $ 150000.0  date  2014-06-18 10:27:53  on account number:  6154199
No mortgage identified on account number:  823959
Found mortgage date of $ 249000.0  date  2016-06-21 10:15:05  on account number:  2804328
Found mortgage date of $ 181649.0  date  2016-06-08 09:21:21  on account number:  4818040
Found mortgage date of $ 36800.0  date  2018-06-01 08:08:41  on account number:  2127628
Found mortgage date of $ 96000.0  date  2005-10-31 03:54:25  on account number:  200409
Found mortgage date of $ 2240000.0  date  2015-06-25 02:29:10  on account number:  1817159
Found mortgage date of $ 104000.0  date  2007-01-31 03:06:45  on account number:  8101841
Found mortgage date of $ 144400.0  date  2015-11-04 03:12:16  on account number:  2631745
Found mortgage date of $ 166920.0  date  2012-08-23 08:34:32  on account number:  641957
Found mortgage date of $ 160000.0  date  2001-08-22 02:24:13  on account number:  2117061
Found mortgage date of $ 76000.0  date  2000-08-24 12:08:36  on account number:  1408674
No mortgage identified on account number:  4814584
Found mortgage date of $ 150000.0  date  2018-09-11 12:19:28  on account number:  1620169
Found mortgage date of $ 92350.0  date  2012-12-19 10:24:25  on account number:  822903
Found mortgage date of $ 179350.0  date  2013-08-06 02:06:26  on account number:  543195
Found mortgage date of $ 127050.0  date  2010-12-13 12:06:32  on account number:  822033
Found mortgage date of $ 167500.0  date  2012-10-03 02:48:00  on account number:  941482
Found mortgage date of $ 73000.0  date  2016-04-07 10:48:04  on account number:  1135038
Found mortgage date of $ 256405.0  date  2018-04-10 09:06:21  on account number:  550959
Found mortgage date of $ 21200.0  date  2004-04-12 10:44:30  on account number:  1415085
Found mortgage date of $ 85521.0  date  2018-05-16 11:48:31  on account number:  3900142
No mortgage identified on account number:  6125034
No mortgage identified on account number:  508993
Found mortgage date of $ 124375.0  date  2004-10-21 10:55:11  on account number:  2104326
Found mortgage date of $ 202000.0  date  2012-12-19 10:21:46  on account number:  823696
Found mortgage date of $ 130500.0  date  2013-11-22 02:50:14  on account number:  529842
Found mortgage date of $ 25000.0  date  2014-05-22 01:28:00  on account number:  1129032
Found mortgage date of $ 53000.0  date  2009-08-03 01:50:48  on account number:  1120107
Found mortgage date of $ 172265.0  date  2017-08-09 10:43:32  on account number:  6152740
Found mortgage date of $ 102000.0  date  2013-06-14 12:11:51  on account number:  4813022
Found mortgage date of $ 0.0  date  1998-02-02 04:30:43  on account number:  7002721
No mortgage identified on account number:  2630353
Found mortgage date of $ 62525.0  date  2010-05-11 01:19:07  on account number:  900024
Found mortgage date of $ 104000.0  date  2012-10-24 03:51:26  on account number:  8102791
Found mortgage date of $ 19500.0  date  2015-06-29 11:02:25  on account number:  825775
Found mortgage date of $ 359575.0  date  2017-10-05 01:54:45  on account number:  5333019
Found mortgage date of $ 81850.0  date  2011-11-04 01:01:59  on account number:  8106185
Found mortgage date of $ 83500.0  date  2008-02-21 09:43:41  on account number:  5003504
Found mortgage date of $ 96000.0  date  1999-03-02 10:48:54  on account number:  2104997
Found mortgage date of $ 270675.0  date  2017-09-15 01:16:01  on account number:  1126734
Found mortgage date of $ 168000.0  date  2012-04-25 08:40:18  on account number:  5323371
Found mortgage date of $ 200000.0  date  2006-03-14 01:20:36  on account number:  1128871
Found mortgage date of $ 221906.0  date  2017-07-19 11:30:07  on account number:  5010165
Found mortgage date of $ 93800.0  date  2009-05-18 01:10:18  on account number:  824819
Found mortgage date of $ 50000.0  date  2005-03-17 09:56:00  on account number:  551035
Found mortgage date of $ 248900.0  date  2004-12-21 12:38:32  on account number:  6152020
Found mortgage date of $ 40000.0  date  2016-05-13 10:16:25  on account number:  8011508
Found mortgage date of $ 168000.0  date  2018-07-05 11:19:24  on account number:  1126606
Found mortgage date of $ 33300.0  date  2014-03-13 01:30:00  on account number:  1131076
Found mortgage date of $ 50000.0  date  2016-01-07 12:08:43  on account number:  1611518
Found mortgage date of $ 123717.0  date  2017-01-26 12:17:36  on account number:  6118399
Found mortgage date of $ 261600.0  date  2018-11-13 02:06:16  on account number:  414914
No mortgage identified on account number:  411923
Found mortgage date of $ 77080.0  date  2011-04-20 10:14:54  on account number:  1419706
Found mortgage date of $ 70677.0  date  2017-12-22 09:21:04  on account number:  5001919
Found mortgage date of $ 117600.0  date  2003-05-23 09:48:14  on account number:  539236
No mortgage identified on account number:  7407743
Found mortgage date of $ 10180000.0  date  1999-05-27 10:41:00  on account number:  6150272
Found mortgage date of $ 10792.0  date  2018-08-24 08:10:17  on account number:  2631832
No mortgage identified on account number:  2625032
No mortgage identified on account number:  2635212
Found mortgage date of $ 0.0  date  1992-07-15 07:01:00  on account number:  4815807
Found mortgage date of $ 113000.0  date  2012-06-07 02:07:42  on account number:  825480
Found mortgage date of $ 756000.0  date  2017-05-25 02:07:16  on account number:  5802100
Found mortgage date of $ 88511.0  date  2008-07-02 11:03:24  on account number:  1405268
Found mortgage date of $ 110000.0  date  2016-01-21 11:44:58  on account number:  824708
Found mortgage date of $ 200000.0  date  2010-05-19 03:24:36  on account number:  1128680
Found mortgage date of $ 166000.0  date  2013-02-14 11:23:50  on account number:  2608367
No mortgage identified on account number:  2304541
Found mortgage date of $ 110000.0  date  2017-11-13 10:26:22  on account number:  143818
Found mortgage date of $ 175000.0  date  2015-02-17 01:52:12  on account number:  820368
Found mortgage date of $ 22000.0  date  2017-12-20 08:21:20  on account number:  142360
Found mortgage date of $ 250000.0  date  2007-11-05 01:50:22  on account number:  3525660
Found mortgage date of $ 250400.0  date  2018-07-06 11:07:07  on account number:  1131718
No mortgage identified on account number:  3518680
Found mortgage date of $ 52000.0  date  2012-07-09 02:18:52  on account number:  641085
Found mortgage date of $ 102044.0  date  2016-11-22 10:14:14  on account number:  5332611
Found mortgage date of $ 326376.0  date  2017-10-05 01:35:09  on account number:  144535
Found mortgage date of $ 93500.0  date  2017-03-16 12:11:28  on account number:  2624931
Found mortgage date of $ 76000.0  date  2014-10-02 12:10:30  on account number:  1005104
Found mortgage date of $ 270750.0  date  2016-06-23 10:06:22  on account number:  1128581
Found mortgage date of $ 360000000.0  date  2018-10-25 11:59:06  on account number:  552631
No mortgage identified on account number:  2637190
Found mortgage date of $ 91575.0  date  2013-03-19 09:14:25  on account number:  1814745
Found mortgage date of $ 86400.0  date  2013-10-10 12:46:42  on account number:  1802364
Found mortgage date of $ 125793.0  date  2011-09-07 09:37:59  on account number:  528080
Found mortgage date of $ 125457.0  date  2016-08-26 03:31:43  on account number:  6128084
Found mortgage date of $ 188000.0  date  2014-05-16 02:31:00  on account number:  6103791
Found mortgage date of $ 11500.0  date  2014-11-10 10:05:39  on account number:  6112421
No mortgage identified on account number:  6150370
Found mortgage date of $ 85000.0  date  2001-07-11 10:19:00  on account number:  5330320
Found mortgage date of $ 20000.0  date  2005-10-26 09:38:05  on account number:  2111314
Found mortgage date of $ 233000.0  date  2017-06-21 01:01:07  on account number:  2633499
Found mortgage date of $ 216500.0  date  2011-04-21 12:06:36  on account number:  822965
Found mortgage date of $ 55000000.0  date  2014-10-24 02:56:24  on account number:  5332704
No mortgage identified on account number:  1129265
No mortgage identified on account number:  551988
Found mortgage date of $ 15000.0  date  2004-03-30 02:32:49  on account number:  3201422
Found mortgage date of $ 153725.0  date  2016-09-13 12:01:52  on account number:  5010037
Found mortgage date of $ 160000.0  date  2017-11-30 09:11:51  on account number:  2605414
Found mortgage date of $ 1360000.0  date  2018-07-05 10:56:02  on account number:  1131764
No mortgage identified on account number:  3301176
Found mortgage date of $ 15000.0  date  2013-06-11 09:11:18  on account number:  8103372
Found mortgage date of $ 250750.0  date  2004-03-08 09:34:40  on account number:  1131531
No mortgage identified on account number:  2806460
Found mortgage date of $ 131869.0  date  2010-11-19 10:20:14  on account number:  5332537
No mortgage identified on account number:  1005060
Found mortgage date of $ 291240.0  date  2016-07-06 02:08:59  on account number:  1130344
No mortgage identified on account number:  2610949
Found mortgage date of $ 195000.0  date  2016-12-08 10:38:04  on account number:  821588
No mortgage identified on account number:  806251
Found mortgage date of $ 113600.0  date  2014-04-04 02:10:00  on account number:  3202925
Found mortgage date of $ 158650.0  date  2016-07-19 09:46:34  on account number:  1140973
Found mortgage date of $ 90072.0  date  2011-04-08 11:04:50  on account number:  1202200
Found mortgage date of $ 101600.0  date  2013-03-11 03:07:26  on account number:  1134287
Found mortgage date of $ 250000.0  date  2018-05-03 10:13:44  on account number:  6609988
Found mortgage date of $ 118600.0  date  2014-07-28 11:00:53  on account number:  641919
No mortgage identified on account number:  505781
Found mortgage date of $ 50000.0  date  2016-01-14 11:18:52  on account number:  550861
Found mortgage date of $ 159120.0  date  2008-12-12 12:13:29  on account number:  1135695
Found mortgage date of $ 52000.0  date  2016-11-21 03:04:30  on account number:  143746
Found mortgage date of $ 31339.0  date  2003-03-21 10:01:08  on account number:  2102293
Found mortgage date of $ 200000.0  date  2012-07-18 12:43:15  on account number:  5331602
Found mortgage date of $ 98600.0  date  2013-09-19 01:02:11  on account number:  1126519
Found mortgage date of $ 239200.0  date  2018-07-19 11:41:32  on account number:  2633250
Found mortgage date of $ 35000.0  date  2007-03-29 01:51:27  on account number:  1615386
Found mortgage date of $ 120000.0  date  2017-03-24 10:50:34  on account number:  4815335
Found mortgage date of $ 199800.0  date  2016-05-12 01:55:54  on account number:  6305938
Found mortgage date of $ 35000.0  date  2006-05-12 01:48:19  on account number:  5311136
Found mortgage date of $ 50000.0  date  2005-06-30 02:30:59  on account number:  7623194
No mortgage identified on account number:  5304288
No mortgage identified on account number:  6150429
Found mortgage date of $ 126000.0  date  2009-03-04 09:25:12  on account number:  5314623
Found mortgage date of $ 200000.0  date  2016-02-05 01:42:08  on account number:  1128259
Found mortgage date of $ 35000.0  date  2013-07-15 01:36:34  on account number:  6124259
Found mortgage date of $ 221357.0  date  2018-08-22 10:27:58  on account number:  1005023
Found mortgage date of $ 0.0  date  1996-05-17 07:01:00  on account number:  1118285
Found mortgage date of $ 25400.0  date  2006-04-04 08:54:58  on account number:  6807747
Found mortgage date of $ 135000.0  date  2015-05-20 11:02:05  on account number:  4504470
Found mortgage date of $ 45000.0  date  2014-10-15 01:08:15  on account number:  1128619
No mortgage identified on account number:  2122642
Found mortgage date of $ 60200.0  date  2018-11-30 12:49:25  on account number:  942061
Found mortgage date of $ 130790.0  date  2011-11-08 09:38:24  on account number:  3511812
Found mortgage date of $ 67800.0  date  2009-06-03 09:04:29  on account number:  2624559
Found mortgage date of $ 220000.0  date  2016-03-15 11:00:02  on account number:  4700197
Found mortgage date of $ 0.0  date  1981-04-15 07:01:00  on account number:  310239
Found mortgage date of $ 50000.0  date  2017-11-14 11:29:32  on account number:  551787
Found mortgage date of $ 171000.0  date  2016-03-07 11:12:26  on account number:  528331
Found mortgage date of $ 2240000.0  date  2015-06-25 02:29:10  on account number:  1815555
Found mortgage date of $ 32500.0  date  2005-12-29 10:08:28  on account number:  103462
Found mortgage date of $ 81750.0  date  2010-10-08 02:59:30  on account number:  1130299
Found mortgage date of $ 139200.0  date  2014-01-03 03:49:00  on account number:  815837
Found mortgage date of $ 54014.0  date  2010-04-02 11:36:59  on account number:  7615671
Found mortgage date of $ 12000.0  date  2006-08-16 01:19:32  on account number:  1130504
Found mortgage date of $ 276000.0  date  2017-01-10 01:21:36  on account number:  5332058
No mortgage identified on account number:  5332973
Found mortgage date of $ 117727.0  date  2010-01-11 08:42:00  on account number:  8100772
Found mortgage date of $ 417000.0  date  2015-06-12 09:44:08  on account number:  3502643
No mortgage identified on account number:  3515303
Found mortgage date of $ 159600.0  date  2018-05-25 01:06:14  on account number:  2635515
Found mortgage date of $ 76095.0  date  2018-06-05 02:27:45  on account number:  2108879
Found mortgage date of $ 44000.0  date  2011-04-01 03:39:41  on account number:  8105286
No mortgage identified on account number:  2635631
Found mortgage date of $ 158083.0  date  2019-01-11 01:22:19  on account number:  606712
Found mortgage date of $ 351670.0  date  2016-12-13 01:04:36  on account number:  1202543
No mortgage identified on account number:  1408305
Found mortgage date of $ 65000.0  date  2007-02-23 11:08:19  on account number:  213373
Found mortgage date of $ 141000.0  date  2008-05-28 10:19:32  on account number:  6147186
Found mortgage date of $ 153609.0  date  2012-07-26 01:38:11  on account number:  8100071
Found mortgage date of $ 115028.0  date  2014-04-10 02:33:00  on account number:  6106056
Found mortgage date of $ 80000.0  date  2011-03-23 10:37:11  on account number:  1800345
Found mortgage date of $ 233910.0  date  2018-08-10 10:49:32  on account number:  2634957
No mortgage identified on account number:  2805960
No mortgage identified on account number:  501425
Found mortgage date of $ 271000.0  date  2017-06-29 02:47:33  on account number:  1128206
Found mortgage date of $ 116961.0  date  2017-05-23 11:01:35  on account number:  2128501
Found mortgage date of $ 506170.0  date  2017-08-18 12:05:22  on account number:  641903
Found mortgage date of $ 55000.0  date  2018-03-07 01:47:49  on account number:  6153103
Found mortgage date of $ 112000.0  date  2013-08-14 11:29:57  on account number:  2116812
Found mortgage date of $ 258000.0  date  2016-09-09 12:56:40  on account number:  7616708
Found mortgage date of $ 25000.0  date  2018-08-27 03:41:24  on account number:  1439677
Found mortgage date of $ 625000.0  date  2011-03-02 11:43:47  on account number:  2808031
No mortgage identified on account number:  942701
Found mortgage date of $ 322700.0  date  2003-10-16 08:55:09  on account number:  6306268
Found mortgage date of $ 223250.0  date  2018-05-31 12:08:59  on account number:  7005207
Found mortgage date of $ 35000.0  date  2016-11-15 08:02:52  on account number:  1127834
No mortgage identified on account number:  940706
Found mortgage date of $ 100000.0  date  2017-12-20 12:01:06  on account number:  1140132
No mortgage identified on account number:  3901068
Found mortgage date of $ 58500.0  date  2016-03-24 11:26:38  on account number:  2123843
No mortgage identified on account number:  1129324
Found mortgage date of $ 84000.0  date  2004-03-02 10:21:42  on account number:  601079
Found mortgage date of $ 552830000.0  date  2015-07-08 01:01:55  on account number:  2633825
Found mortgage date of $ 70700.0  date  2004-04-12 09:28:51  on account number:  930261
Found mortgage date of $ 62000.0  date  2001-04-23 03:09:51  on account number:  7300727
Found mortgage date of $ 44969.0  date  2014-09-05 01:50:24  on account number:  642877
Found mortgage date of $ 147831.0  date  2009-07-01 02:18:30  on account number:  6128645
Found mortgage date of $ 209000.0  date  2009-03-09 09:34:59  on account number:  643072
Found mortgage date of $ 164737.0  date  2018-06-14 02:11:46  on account number:  1400029
Found mortgage date of $ 50000.0  date  2018-05-02 10:31:27  on account number:  1126840
No mortgage identified on account number:  1146327
Found mortgage date of $ 249000.0  date  2012-03-06 09:51:00  on account number:  6609490
Found mortgage date of $ 365000.0  date  2017-12-19 08:18:12  on account number:  915645
Found mortgage date of $ 208000.0  date  2017-04-12 02:41:44  on account number:  1440454
Found mortgage date of $ 232000.0  date  2017-03-29 11:34:39  on account number:  551165
Found mortgage date of $ 182000.0  date  2013-08-29 12:39:28  on account number:  2634320
Found mortgage date of $ 80000.0  date  2006-07-05 10:37:00  on account number:  825336
Found mortgage date of $ 180000.0  date  2015-09-15 11:35:55  on account number:  816663
Found mortgage date of $ 94600.0  date  2000-06-06 10:30:03  on account number:  126675
Found mortgage date of $ 75000.0  date  2006-09-29 01:11:30  on account number:  2605228
No mortgage identified on account number:  200921
Found mortgage date of $ 0.0  date  1996-12-17 07:01:00  on account number:  8102261
Found mortgage date of $ 50898.32  date  2012-08-02 03:44:55  on account number:  1820015
Found mortgage date of $ 348305.0  date  2017-01-09 12:38:08  on account number:  3301311
Found mortgage date of $ 22600.0  date  2016-07-05 01:34:38  on account number:  1402285
Found mortgage date of $ 4350000.0  date  2012-02-08 03:04:08  on account number:  5501458
Found mortgage date of $ 141391.0  date  2013-09-05 02:25:08  on account number:  822305
Found mortgage date of $ 71000.0  date  2012-11-27 09:05:43  on account number:  814997
Found mortgage date of $ 140000.0  date  2017-03-02 01:38:11  on account number:  5010043
Found mortgage date of $ 185000.0  date  2012-07-19 11:58:00  on account number:  6152650
Found mortgage date of $ 115000.0  date  2013-02-05 02:53:00  on account number:  2632592
No mortgage identified on account number:  8011613
Found mortgage date of $ 33150.0  date  2006-07-05 01:50:38  on account number:  8100764
Found mortgage date of $ 35000.0  date  2013-11-06 02:59:12  on account number:  106356
Found mortgage date of $ 123780.0  date  2012-02-08 09:48:26  on account number:  2630667
Found mortgage date of $ 20000.0  date  2011-11-04 02:58:23  on account number:  2627329
Found mortgage date of $ 228816.0  date  2016-03-02 02:56:40  on account number:  2636026
Found mortgage date of $ 185743.0  date  2016-02-19 11:34:33  on account number:  552275
Found mortgage date of $ 100000.0  date  2015-03-11 11:50:34  on account number:  552060
Found mortgage date of $ 50000.0  date  2012-10-09 02:21:14  on account number:  2630201
No mortgage identified on account number:  1128297
Found mortgage date of $ 220000.0  date  2010-03-10 02:09:36  on account number:  821608
Found mortgage date of $ 153000.0  date  2010-03-10 02:09:35  on account number:  5332340
Found mortgage date of $ 401000.0  date  2015-11-17 10:12:52  on account number:  1134864
No mortgage identified on account number:  5005621
Found mortgage date of $ 98250.0  date  2016-12-21 10:02:36  on account number:  823331
No mortgage identified on account number:  2603772
Found mortgage date of $ 55800.0  date  2014-08-21 01:43:43  on account number:  2138735
Found mortgage date of $ 61812.0  date  2004-04-29 01:52:52  on account number:  824019
Found mortgage date of $ 95000.0  date  2005-11-04 12:40:46  on account number:  551992
No mortgage identified on account number:  2142279
Found mortgage date of $ 90000.0  date  2009-07-21 11:02:39  on account number:  2631332
Found mortgage date of $ 295200.0  date  2016-12-07 10:49:04  on account number:  942246
Found mortgage date of $ 159820.0  date  2017-10-17 10:49:55  on account number:  2632354
Found mortgage date of $ 270000.0  date  2015-05-04 02:00:27  on account number:  5332422
Found mortgage date of $ 240000.0  date  2015-09-10 01:32:44  on account number:  3301074
Found mortgage date of $ 145350.0  date  2005-06-06 11:39:17  on account number:  1132541
Found mortgage date of $ 224000.0  date  2016-04-07 12:56:11  on account number:  2630085
Found mortgage date of $ 60000.0  date  1999-05-11 08:44:06  on account number:  1809695
No mortgage identified on account number:  6135293
No mortgage identified on account number:  806994
Found mortgage date of $ 45000.0  date  2018-07-02 02:28:18  on account number:  7613806
Found mortgage date of $ 240000.0  date  2014-08-25 02:36:02  on account number:  141595
No mortgage identified on account number:  6807381
Found mortgage date of $ 236328.0  date  2011-12-29 01:00:38  on account number:  825788
Found mortgage date of $ 385400.0  date  2016-09-06 10:15:39  on account number:  1136382
Found mortgage date of $ 112000.0  date  2018-08-16 01:08:17  on account number:  6126049
Found mortgage date of $ 70729.0  date  2005-08-18 12:59:00  on account number:  1702095
Found mortgage date of $ 329000.0  date  2016-02-03 12:14:35  on account number:  2105764
Found mortgage date of $ 265020.0  date  2016-02-22 11:43:17  on account number:  8507611
No mortgage identified on account number:  2103575
Found mortgage date of $ 180000.0  date  2017-12-22 10:27:54  on account number:  633205
Found mortgage date of $ 9900.0  date  2017-08-24 08:07:43  on account number:  548693
Found mortgage date of $ 157102.0  date  2015-06-25 01:26:33  on account number:  2634372
Found mortgage date of $ 99415.0  date  2017-04-19 10:12:36  on account number:  6152994
Found mortgage date of $ 160000.0  date  2003-05-29 08:54:44  on account number:  1502751
Found mortgage date of $ 488000.0  date  2012-03-01 10:38:52  on account number:  941748
No mortgage identified on account number:  6806091
Found mortgage date of $ 367650.0  date  2012-11-14 12:16:09  on account number:  1128710
Found mortgage date of $ 75000.0  date  2006-07-24 11:37:44  on account number:  5304652
Found mortgage date of $ 157592.0  date  2017-11-22 08:31:58  on account number:  2630659
Found mortgage date of $ 50000.0  date  2010-09-17 12:11:49  on account number:  1128785
Found mortgage date of $ 50927.0  date  2014-06-04 12:26:21  on account number:  903431
Found mortgage date of $ 15000.0  date  2016-05-17 07:38:24  on account number:  6101739
Found mortgage date of $ 24000.0  date  2017-01-09 10:00:52  on account number:  2700212
No mortgage identified on account number:  2603527
Found mortgage date of $ 253000.0  date  2018-03-22 08:18:51  on account number:  5322685
Found mortgage date of $ 52000.0  date  2015-11-23 08:11:46  on account number:  1101447
No mortgage identified on account number:  1400193
Found mortgage date of $ 120700.0  date  2012-03-15 11:01:42  on account number:  2138463
No mortgage identified on account number:  2629895
Found mortgage date of $ 92000.0  date  2012-10-10 01:18:53  on account number:  1500511
Found mortgage date of $ 90500.0  date  2012-09-19 02:33:19  on account number:  819123
No mortgage identified on account number:  4817648
Found mortgage date of $ 72000.0  date  2013-03-21 03:00:54  on account number:  8107653
No mortgage identified on account number:  7621809
Found mortgage date of $ 153835.0  date  2010-08-24 01:43:16  on account number:  2630943
No mortgage identified on account number:  2620952
Found mortgage date of $ 88000.0  date  2003-10-27 08:37:42  on account number:  1004964
Found mortgage date of $ 36766.5  date  2005-12-05 03:01:43  on account number:  8011265
Found mortgage date of $ 45500.0  date  2019-01-10 02:33:52  on account number:  4400771
Found mortgage date of $ 18948.0  date  2005-05-06 03:55:41  on account number:  2635674
Found mortgage date of $ 376000.0  date  2017-09-18 02:32:39  on account number:  143679
No mortgage identified on account number:  2123681
Found mortgage date of $ 102150.0  date  2016-12-22 11:29:03  on account number:  2115018
Found mortgage date of $ 110000.0  date  2016-09-07 11:06:02  on account number:  6152155
Found mortgage date of $ 160000.0  date  2005-10-25 03:10:10  on account number:  523763
Found mortgage date of $ 181600.0  date  2017-09-18 10:54:20  on account number:  813567
Found mortgage date of $ 25000.0  date  2012-04-27 12:53:05  on account number:  2633730
Found mortgage date of $ 100000.0  date  2017-06-19 10:12:27  on account number:  915609
Found mortgage date of $ 358500.0  date  2012-01-05 12:04:11  on account number:  533602
Found mortgage date of $ 50000.0  date  2015-01-29 12:03:36  on account number:  904984
Found mortgage date of $ 12400.0  date  2013-08-20 01:03:44  on account number:  1132697
No mortgage identified on account number:  1134442
Found mortgage date of $ 171200.0  date  2018-01-18 11:12:57  on account number:  3518345
Found mortgage date of $ 8750000.0  date  2015-01-12 09:32:34  on account number:  825633
No mortgage identified on account number:  507059
Found mortgage date of $ 50000.0  date  2018-04-02 09:30:06  on account number:  1116347
Found mortgage date of $ 192000.0  date  2017-08-22 09:23:05  on account number:  928844
Found mortgage date of $ 333400.0  date  2016-10-20 02:28:49  on account number:  2503157
Found mortgage date of $ 210000.0  date  2008-12-29 10:36:38  on account number:  545821
Found mortgage date of $ 199192.0  date  2017-08-28 02:14:57  on account number:  6151373
Found mortgage date of $ 149572.0  date  2012-11-14 09:32:48  on account number:  940119
No mortgage identified on account number:  935166
Found mortgage date of $ 122187.0  date  2003-06-24 01:54:33  on account number:  1132588
Found mortgage date of $ 123500.0  date  2003-06-25 08:34:43  on account number:  940457
Found mortgage date of $ 123000.0  date  2012-08-16 01:03:09  on account number:  2633432
Found mortgage date of $ 199879.86  date  2013-04-02 12:39:20  on account number:  5331672
Found mortgage date of $ 264000.0  date  2014-12-16 08:27:46  on account number:  3103110
Found mortgage date of $ 300000.0  date  2014-12-02 03:20:45  on account number:  821866
No mortgage identified on account number:  824519
No mortgage identified on account number:  1118251
Found mortgage date of $ 166500.0  date  2017-05-03 10:40:06  on account number:  2637318
Found mortgage date of $ 20000.0  date  2009-04-29 11:39:29  on account number:  5329175
No mortgage identified on account number:  3203301
No mortgage identified on account number:  2433986
Found mortgage date of $ 137750.0  date  2015-12-23 10:50:10  on account number:  113611
Found mortgage date of $ 101559.0  date  2012-01-03 03:33:42  on account number:  904321
Found mortgage date of $ 90000.0  date  2011-11-21 01:42:59  on account number:  1127326
Found mortgage date of $ 132905.0  date  2014-12-08 03:52:00  on account number:  1620079
Found mortgage date of $ 43893.0  date  2018-02-02 08:11:31  on account number:  2634221
Found mortgage date of $ 100000.0  date  2014-03-27 12:27:00  on account number:  5330423
Found mortgage date of $ 109200.0  date  2014-01-06 03:38:00  on account number:  917737
Found mortgage date of $ 131442.0  date  2010-09-07 09:38:05  on account number:  1109251
Found mortgage date of $ 141100.0  date  2011-06-13 11:27:06  on account number:  820323
Found mortgage date of $ 25000.0  date  2006-04-13 08:50:01  on account number:  1133409
Found mortgage date of $ 564508.93  date  2003-07-18 09:06:30  on account number:  6152498
Found mortgage date of $ 50000.0  date  2014-05-13 03:36:00  on account number:  2637410
Found mortgage date of $ 144750.0  date  2012-10-16 02:05:43  on account number:  103039
Found mortgage date of $ 45000.0  date  2007-08-01 02:05:44  on account number:  5001528
Found mortgage date of $ 0.0  date  1997-07-10 07:01:00  on account number:  608669
No mortgage identified on account number:  300128
Found mortgage date of $ 116000.0  date  2010-12-13 02:35:55  on account number:  204021
Found mortgage date of $ 216200.0  date  2016-08-31 12:42:48  on account number:  551077
Found mortgage date of $ 67500.0  date  2010-10-05 12:38:18  on account number:  4801172
Found mortgage date of $ 20000.0  date  2011-11-18 12:17:32  on account number:  6125832
No mortgage identified on account number:  625059
No mortgage identified on account number:  7628048
Found mortgage date of $ 235000.0  date  2017-06-27 12:34:55  on account number:  7631570
Found mortgage date of $ 143569.0  date  2018-04-11 03:23:48  on account number:  814717
No mortgage identified on account number:  6129803
Found mortgage date of $ 251643.0  date  2004-12-13 02:34:40  on account number:  552057
Found mortgage date of $ 195500.0  date  2016-12-05 12:26:34  on account number:  2900793
Found mortgage date of $ 36000.0  date  2006-03-01 03:17:25  on account number:  4807146
Found mortgage date of $ 333800.0  date  2010-05-03 12:53:24  on account number:  5331712
Found mortgage date of $ 2240000.0  date  2015-06-25 02:29:10  on account number:  1817019
Found mortgage date of $ 249900.0  date  2018-09-06 01:28:12  on account number:  3301307
Found mortgage date of $ 154700.0  date  2015-05-04 03:48:57  on account number:  1441361
Found mortgage date of $ 85000.0  date  2013-04-16 02:34:50  on account number:  6150772
Found mortgage date of $ 106232.0  date  2004-08-18 02:33:24  on account number:  2120631
Found mortgage date of $ 162000.0  date  2014-11-10 11:29:41  on account number:  2139275
Found mortgage date of $ 193000.0  date  2016-02-17 09:59:13  on account number:  2633095
Found mortgage date of $ 162306.0  date  2008-05-21 10:17:13  on account number:  900613
Found mortgage date of $ 25707.41  date  2015-03-16 01:10:18  on account number:  3508412
Found mortgage date of $ 5000000.0  date  2011-08-19 01:36:54  on account number:  2434271
No mortgage identified on account number:  1424424
Found mortgage date of $ 211894.0  date  2009-12-30 10:44:23  on account number:  5330135
Found mortgage date of $ 76000.0  date  2010-03-31 12:34:56  on account number:  642973
Found mortgage date of $ 136500.0  date  2014-03-07 01:14:00  on account number:  906387
Found mortgage date of $ 116100.0  date  2015-08-17 03:28:08  on account number:  6151644
Found mortgage date of $ 0.0  date  1989-10-06 07:01:00  on account number:  1114026
Found mortgage date of $ 213218.0  date  2017-12-19 11:24:34  on account number:  2636878
Found mortgage date of $ 92500.0  date  2004-08-04 03:31:00  on account number:  5330513
Found mortgage date of $ 372000.0  date  2018-05-21 01:32:16  on account number:  141142
Found mortgage date of $ 158000.0  date  2001-11-28 11:13:00  on account number:  2622483
Found mortgage date of $ 25000.0  date  2014-07-03 01:27:10  on account number:  822775
Found mortgage date of $ 285000.0  date  2016-12-13 09:33:42  on account number:  311367
Found mortgage date of $ 387264.0  date  2018-06-29 02:03:05  on account number:  1136832
Found mortgage date of $ 199492.0  date  2015-11-18 12:49:05  on account number:  6153131
Found mortgage date of $ 296244.0  date  2016-10-03 11:19:34  on account number:  2139236
Found mortgage date of $ 99040.0  date  2012-06-20 12:49:42  on account number:  1400924
Found mortgage date of $ 73641.0  date  2015-10-02 10:45:53  on account number:  2101556
No mortgage identified on account number:  5330187
Found mortgage date of $ 172600.0  date  2007-11-02 12:17:39  on account number:  6100619
Found mortgage date of $ 97500.0  date  2017-08-30 12:25:10  on account number:  6121136
Found mortgage date of $ 313500.0  date  2017-07-12 03:36:03  on account number:  142226
No mortgage identified on account number:  1130882
Found mortgage date of $ 18000.0  date  2003-12-11 03:22:40  on account number:  1410601
Found mortgage date of $ 232000.0  date  2018-07-03 10:16:14  on account number:  1134407
Found mortgage date of $ 192679.0  date  2015-03-20 10:24:52  on account number:  6811074
Found mortgage date of $ 269500.0  date  2018-12-05 08:42:48  on account number:  6128581
Found mortgage date of $ 78900.0  date  2014-12-15 11:28:25  on account number:  2634123
Found mortgage date of $ 84550.0  date  2016-09-13 10:35:10  on account number:  3201619
Found mortgage date of $ 400000.0  date  2014-12-04 02:53:05  on account number:  642778
Found mortgage date of $ 223250.0  date  2015-06-09 08:09:46  on account number:  914355
Found mortgage date of $ 12593.15  date  2014-02-10 10:56:00  on account number:  6129277
Found mortgage date of $ 49000.0  date  2006-12-19 03:33:22  on account number:  4900553
Found mortgage date of $ 60000.0  date  2016-04-06 01:21:18  on account number:  6135137
Found mortgage date of $ 43000.0  date  2008-02-13 01:26:35  on account number:  136387
Found mortgage date of $ 269535.0  date  2009-08-11 11:33:16  on account number:  3511677
Found mortgage date of $ 217600.0  date  2012-03-28 12:09:57  on account number:  3101827
Found mortgage date of $ 252000.0  date  2016-08-19 01:30:29  on account number:  552166
Found mortgage date of $ 266387.0  date  2012-03-12 01:23:56  on account number:  3103130
No mortgage identified on account number:  2107392
Found mortgage date of $ 168036.0  date  2014-12-10 10:25:34  on account number:  2635087
Found mortgage date of $ 63175.69  date  2012-03-06 11:15:23  on account number:  821190
Found mortgage date of $ 30590.0  date  2018-12-14 07:58:39  on account number:  2632942
Found mortgage date of $ 199410.0  date  2016-08-08 02:22:15  on account number:  826029
Found mortgage date of $ 117826.0  date  2010-05-14 02:07:59  on account number:  6111033
Found mortgage date of $ 311400.0  date  2016-09-22 12:51:11  on account number:  822319
Found mortgage date of $ 83000.0  date  2017-01-05 12:35:37  on account number:  1133514
Found mortgage date of $ 139650.0  date  2014-06-30 12:51:18  on account number:  922269
Found mortgage date of $ 138000.0  date  2010-07-27 08:58:09  on account number:  943376
Found mortgage date of $ 100000.0  date  2018-12-26 03:23:17  on account number:  1437950
No mortgage identified on account number:  2615339
Found mortgage date of $ 94400.0  date  2018-09-28 08:04:27  on account number:  2618621
No mortgage identified on account number:  2635020
No mortgage identified on account number:  541729
No mortgage identified on account number:  1800027
Found mortgage date of $ 289000.0  date  2013-03-29 02:02:47  on account number:  1133816
Found mortgage date of $ 63822.0  date  2010-04-13 01:12:59  on account number:  6130909
No mortgage identified on account number:  603659
Found mortgage date of $ 75000.0  date  2015-02-18 02:51:42  on account number:  1135781
Found mortgage date of $ 104000.0  date  2014-07-24 11:42:24  on account number:  508381
Found mortgage date of $ 41899.0  date  2018-06-14 12:55:13  on account number:  2503164
Found mortgage date of $ 218000.0  date  2015-02-17 09:36:11  on account number:  824831
Found mortgage date of $ 73750.0  date  2010-03-22 01:26:30  on account number:  822475
No mortgage identified on account number:  2632261
No mortgage identified on account number:  2808408
Found mortgage date of $ 20500.0  date  2014-03-26 09:03:00  on account number:  5002818
Found mortgage date of $ 28500.0  date  2012-12-10 01:19:21  on account number:  118516
Found mortgage date of $ 82400.0  date  2012-11-07 02:27:47  on account number:  1202255
Found mortgage date of $ 552830000.0  date  2015-07-08 01:01:55  on account number:  2635827
No mortgage identified on account number:  5200806
No mortgage identified on account number:  1133043
Found mortgage date of $ 204300.0  date  2015-11-10 02:30:11  on account number:  814202
Found mortgage date of $ 400000.0  date  2015-04-29 09:38:35  on account number:  7606982
No mortgage identified on account number:  1146947
No mortgage identified on account number:  1418467
Found mortgage date of $ 254333.0  date  2016-03-10 11:08:22  on account number:  6153445
Found mortgage date of $ 278986.0  date  2018-03-15 02:13:23  on account number:  901971
Found mortgage date of $ 26000.0  date  2015-05-08 11:21:16  on account number:  6106846
No mortgage identified on account number:  2635191
Found mortgage date of $ 240000.0  date  2018-03-29 03:01:57  on account number:  3102033
No mortgage identified on account number:  1434241
Found mortgage date of $ 80000.0  date  2009-10-05 03:10:57  on account number:  2631360
Found mortgage date of $ 816640.0  date  2013-09-06 02:52:16  on account number:  1620143
Found mortgage date of $ 35140.0  date  2016-02-19 02:44:42  on account number:  905879
Found mortgage date of $ 92000.0  date  2009-09-09 11:16:42  on account number:  2636494
Found mortgage date of $ 169150.0  date  2004-08-24 01:39:38  on account number:  820379
Found mortgage date of $ 117090.0  date  2016-04-20 11:53:39  on account number:  6400949
Found mortgage date of $ 136870.0  date  2017-08-21 10:32:02  on account number:  1127479
Found mortgage date of $ 107250.0  date  2016-01-26 12:53:57  on account number:  823640
Found mortgage date of $ 4350000.0  date  2012-02-08 03:04:08  on account number:  5501431
Found mortgage date of $ 171830.0  date  2018-10-25 11:48:29  on account number:  1617850
Found mortgage date of $ 25000.0  date  2016-10-28 03:31:41  on account number:  3601935
Found mortgage date of $ 56000.0  date  2003-12-04 12:26:17  on account number:  1714861
No mortgage identified on account number:  1002937
Found mortgage date of $ 465000.0  date  2009-03-26 09:06:27  on account number:  1132849
Found mortgage date of $ 250000.0  date  2013-03-05 11:06:18  on account number:  1131142
Found mortgage date of $ 289338.0  date  2018-01-05 11:43:51  on account number:  6811173
Found mortgage date of $ 200000.0  date  2017-12-04 10:55:17  on account number:  824017
Found mortgage date of $ 108700.0  date  2012-10-18 10:13:33  on account number:  6152951
Found mortgage date of $ 212500.0  date  2015-01-05 01:54:04  on account number:  5332921
Found mortgage date of $ 2240000.0  date  2015-06-25 02:29:10  on account number:  1816721
Found mortgage date of $ 100000.0  date  2009-09-22 01:51:08  on account number:  1129813
Found mortgage date of $ 122000.0  date  2014-07-14 03:19:48  on account number:  660088
No mortgage identified on account number:  940603
Found mortgage date of $ 40000.0  date  2017-06-19 09:39:14  on account number:  1134631
Found mortgage date of $ 30000.0  date  2013-06-11 11:02:57  on account number:  941854
No mortgage identified on account number:  511188
Found mortgage date of $ 330000.0  date  2018-08-20 11:14:48  on account number:  6150423
Found mortgage date of $ 191916.0  date  2009-03-31 03:51:37  on account number:  643922
Found mortgage date of $ 925000.0  date  2016-08-08 12:54:38  on account number:  2605589
Found mortgage date of $ 167400.0  date  2016-10-11 01:28:21  on account number:  3205054
Found mortgage date of $ 272500.0  date  2016-11-04 02:09:43  on account number:  2637048
Found mortgage date of $ 100000.0  date  2017-09-26 08:40:26  on account number:  1113674
Found mortgage date of $ 142000.0  date  2011-12-15 10:48:00  on account number:  2632475
Found mortgage date of $ 35000.0  date  2013-03-20 02:46:39  on account number:  1133840
Found mortgage date of $ 65600.0  date  2017-12-07 03:17:07  on account number:  6152415
Found mortgage date of $ 192000.0  date  2017-02-03 10:21:21  on account number:  2635594
Found mortgage date of $ 100000.0  date  2002-10-14 10:37:30  on account number:  2138286
Found mortgage date of $ 246000.0  date  2017-02-16 01:16:41  on account number:  6305822
Found mortgage date of $ 413250.0  date  2015-05-01 09:19:50  on account number:  1126978
Found mortgage date of $ 186423.0  date  2016-07-19 01:12:17  on account number:  8113793
Found mortgage date of $ 127300.0  date  2013-11-27 08:38:58  on account number:  2636935
No mortgage identified on account number:  4302842
Found mortgage date of $ 418431.0  date  2018-04-16 01:07:17  on account number:  2638071
Found mortgage date of $ 134139.25  date  2014-12-09 11:18:54  on account number:  820934
Found mortgage date of $ 39300.0  date  2012-11-07 10:17:15  on account number:  6152517
Found mortgage date of $ 280000.0  date  2016-04-27 09:48:15  on account number:  1820825
Found mortgage date of $ 192700.0  date  2011-09-26 01:02:49  on account number:  2634969
Found mortgage date of $ 218500.0  date  2018-08-29 11:47:43  on account number:  1440616
Found mortgage date of $ 185250.0  date  2015-05-27 01:44:07  on account number:  7610033
Found mortgage date of $ 63000.0  date  1999-08-03 11:47:00  on account number:  2123045
Found mortgage date of $ 100000.0  date  2018-08-22 08:48:23  on account number:  2628635
Found mortgage date of $ 178800.0  date  2010-02-17 01:11:49  on account number:  821793
Found mortgage date of $ 324000.0  date  2008-10-20 03:42:37  on account number:  4800435
Found mortgage date of $ 329600.0  date  2016-01-15 10:04:23  on account number:  6609430
Found mortgage date of $ 20000.0  date  2003-05-02 01:56:58  on account number:  2621908
Found mortgage date of $ 324500.0  date  2017-06-21 04:08:30  on account number:  7626231
Found mortgage date of $ 109250.0  date  2015-10-01 11:10:58  on account number:  940904
Found mortgage date of $ 160000.0  date  2012-02-02 01:23:48  on account number:  941143
No mortgage identified on account number:  942518
Found mortgage date of $ 201600.0  date  2018-01-09 09:52:55  on account number:  550853
Found mortgage date of $ 44100.0  date  2004-01-29 12:00:39  on account number:  3701328
Found mortgage date of $ 153200.0  date  2004-07-19 10:38:42  on account number:  6151786
Found mortgage date of $ 92000.0  date  2012-05-02 03:48:14  on account number:  5325633
Found mortgage date of $ 61050.0  date  2018-11-26 10:41:28  on account number:  2148919
Found mortgage date of $ 242588.0  date  2018-09-13 01:13:03  on account number:  820845
No mortgage identified on account number:  635895
Found mortgage date of $ 170600.0  date  2015-05-11 11:06:56  on account number:  534196
Found mortgage date of $ 140500.0  date  2011-07-19 12:46:07  on account number:  5010069
Found mortgage date of $ 73934.79  date  2012-10-03 03:17:19  on account number:  2635494
No mortgage identified on account number:  933112
No mortgage identified on account number:  8503411
Found mortgage date of $ 60000.0  date  2011-02-03 12:27:47  on account number:  507563
No mortgage identified on account number:  1133585
Found mortgage date of $ 25000.0  date  2018-11-13 04:01:02  on account number:  941483
Found mortgage date of $ 240000.0  date  2009-10-09 03:24:37  on account number:  8510042
Found mortgage date of $ 133200.0  date  2015-02-10 03:13:13  on account number:  4814363
Found mortgage date of $ 102000.0  date  2015-10-07 03:05:26  on account number:  6130143
Found mortgage date of $ 327787.0  date  2018-01-16 12:43:22  on account number:  1202793
Found mortgage date of $ 125000.0  date  2005-09-12 08:41:44  on account number:  1803795
No mortgage identified on account number:  3301145
Found mortgage date of $ 239800.0  date  2016-05-24 02:29:29  on account number:  3102041
Found mortgage date of $ 55000.0  date  2018-08-27 08:59:13  on account number:  2630719
Found mortgage date of $ 170000.0  date  2007-11-27 10:03:15  on account number:  2620472
Found mortgage date of $ 72000.0  date  2006-12-14 12:45:00  on account number:  511081
Found mortgage date of $ 50000.0  date  2017-08-11 02:01:50  on account number:  1127546
Found mortgage date of $ 93000.0  date  2015-10-20 10:14:23  on account number:  820944
Found mortgage date of $ 25000.0  date  2008-11-07 01:23:20  on account number:  940728
No mortgage identified on account number:  144256
Found mortgage date of $ 265500.0  date  2014-03-05 02:21:00  on account number:  5330562
Found mortgage date of $ 42000.0  date  2018-10-22 09:19:02  on account number:  7400096
No mortgage identified on account number:  2618486
Found mortgage date of $ 18000.0  date  2010-12-29 09:17:07  on account number:  1438166
Found mortgage date of $ 125252.0  date  2018-10-10 08:41:03  on account number:  3501078
Found mortgage date of $ 320000.0  date  2017-09-21 01:15:36  on account number:  1127392
Found mortgage date of $ 0.0  date  1991-05-20 07:01:00  on account number:  6141226
Found mortgage date of $ 47500.0  date  2005-03-30 11:25:10  on account number:  1802241
Found mortgage date of $ 840000.0  date  2017-03-23 08:07:04  on account number:  2808165
Found mortgage date of $ 128300.0  date  2013-01-15 08:46:24  on account number:  5309557
Found mortgage date of $ 111200.0  date  2013-03-19 02:43:47  on account number:  2608189
Found mortgage date of $ 46000.0  date  2016-09-01 08:56:34  on account number:  2123436
No mortgage identified on account number:  2132087
Found mortgage date of $ 105000.0  date  2018-04-16 03:06:04  on account number:  534137
No mortgage identified on account number:  826171
Found mortgage date of $ 20000.0  date  2013-06-18 10:05:32  on account number:  5327083
Found mortgage date of $ 104044.0  date  2013-01-23 08:38:46  on account number:  1610341
No mortgage identified on account number:  7620659
Found mortgage date of $ 106080.0  date  2007-01-04 02:42:25  on account number:  2636936
Found mortgage date of $ 57800.0  date  2011-10-12 10:45:52  on account number:  550118
Found mortgage date of $ 1350000.0  date  2006-03-29 02:37:00  on account number:  119059
Found mortgage date of $ 176000.0  date  2018-06-14 01:42:34  on account number:  5331439
Found mortgage date of $ 151851.0  date  2018-06-14 02:17:17  on account number:  6100465
Found mortgage date of $ 129000.0  date  2003-01-14 03:05:10  on account number:  1129707
Found mortgage date of $ 114824.0  date  2015-06-08 12:23:04  on account number:  1128046
Found mortgage date of $ 150000.0  date  2017-08-29 08:09:56  on account number:  822808
Found mortgage date of $ 70210.33  date  2003-08-04 11:21:48  on account number:  600129
Found mortgage date of $ 186300.0  date  2009-11-13 08:52:52  on account number:  8011796
Found mortgage date of $ 30000.0  date  2018-03-07 02:31:39  on account number:  3511995
No mortgage identified on account number:  2808061
Found mortgage date of $ 288000.0  date  2018-03-27 12:14:05  on account number:  550412
Found mortgage date of $ 166250.0  date  2014-07-22 09:13:52  on account number:  6300308
Found mortgage date of $ 38000.0  date  2017-11-02 02:43:27  on account number:  914444
Found mortgage date of $ 331500.0  date  2014-02-10 10:35:00  on account number:  2808037
Found mortgage date of $ 457500.0  date  2014-06-02 11:21:00  on account number:  2128659
No mortgage identified on account number:  6810756
Found mortgage date of $ 513315800.0  date  2014-12-19 01:23:23  on account number:  4700163
No mortgage identified on account number:  5006872
Found mortgage date of $ 126000.0  date  2001-03-30 12:27:46  on account number:  3702847
Found mortgage date of $ 600000.0  date  2018-02-01 02:16:52  on account number:  2633048
Found mortgage date of $ 152290.0  date  2017-07-03 09:03:29  on account number:  506991
Found mortgage date of $ 135000.0  date  2013-06-20 02:03:16  on account number:  2301072
Found mortgage date of $ 43000.0  date  2018-11-13 08:40:03  on account number:  5200431
Found mortgage date of $ 184400.0  date  2010-12-27 10:56:37  on account number:  8510000
Found mortgage date of $ 186868.0  date  2018-10-24 03:31:26  on account number:  7609272
No mortgage identified on account number:  2607344
Found mortgage date of $ 316000.0  date  2016-06-27 03:04:16  on account number:  3701867
Found mortgage date of $ 199759.0  date  2010-09-03 02:18:33  on account number:  3505111
Found mortgage date of $ 146000.0  date  2014-08-25 02:35:59  on account number:  943593
Found mortgage date of $ 238687.0  date  2015-03-03 01:21:15  on account number:  643200
No mortgage identified on account number:  921599
Found mortgage date of $ 200000.0  date  2003-09-09 01:46:24  on account number:  2701111
Found mortgage date of $ 116000.0  date  2016-07-13 08:02:13  on account number:  822413
Found mortgage date of $ 150750.0  date  2016-08-03 01:29:48  on account number:  8501087
Found mortgage date of $ 190176.0  date  2017-04-14 10:45:04  on account number:  2636883
Found mortgage date of $ 258300.0  date  2017-01-17 12:13:02  on account number:  822197
Found mortgage date of $ 245000.0  date  2016-03-28 01:04:11  on account number:  6152858
Found mortgage date of $ 15000.0  date  2013-09-09 12:11:47  on account number:  940922
No mortgage identified on account number:  1128128
Found mortgage date of $ 195750.0  date  2003-05-02 03:19:07  on account number:  1407074
Found mortgage date of $ 137450.0  date  2012-09-28 01:39:48  on account number:  540927
Found mortgage date of $ 191330.0  date  2017-11-07 11:49:52  on account number:  553251
Found mortgage date of $ 30000.0  date  2018-11-26 02:42:46  on account number:  7106289
Found mortgage date of $ 152559.26  date  2004-02-13 10:36:57  on account number:  603985
Found mortgage date of $ 135200.0  date  2012-08-08 03:43:21  on account number:  528943
Found mortgage date of $ 320000.0  date  2015-12-28 12:22:02  on account number:  2808311
Found mortgage date of $ 264609.0  date  2013-03-22 03:32:38  on account number:  136328
Found mortgage date of $ 135500.0  date  2017-08-31 10:27:04  on account number:  2637592
Found mortgage date of $ 56000.0  date  2015-07-02 02:52:03  on account number:  550007
No mortgage identified on account number:  2134691
No mortgage identified on account number:  2109808
No mortgage identified on account number:  2603268
Found mortgage date of $ 20000.0  date  2003-09-12 02:17:29  on account number:  6150897
No mortgage identified on account number:  2114615
No mortgage identified on account number:  909351
Found mortgage date of $ 100500.0  date  2003-07-29 08:33:30  on account number:  540870
Found mortgage date of $ 4350000.0  date  2012-02-08 03:04:08  on account number:  5500605
Found mortgage date of $ 50000.0  date  2014-07-29 10:50:01  on account number:  2633872
Found mortgage date of $ 48000.0  date  2016-10-27 11:35:54  on account number:  5330683
Found mortgage date of $ 70200.0  date  2012-12-10 01:19:22  on account number:  940535
No mortgage identified on account number:  4700201
No mortgage identified on account number:  823359
No mortgage identified on account number:  513318
Found mortgage date of $ 184820.0  date  2013-03-12 09:20:37  on account number:  5800242
Found mortgage date of $ 159600.0  date  2004-11-19 09:41:35  on account number:  940665
Found mortgage date of $ 178683.0  date  2008-10-01 09:04:07  on account number:  550794
Found mortgage date of $ 130500.0  date  2014-06-24 01:49:34  on account number:  817724
Found mortgage date of $ 33384.0  date  2013-06-13 02:50:01  on account number:  1806904
Found mortgage date of $ 122700.0  date  2017-02-08 11:33:33  on account number:  1134916
Found mortgage date of $ 31000.0  date  2018-08-30 08:10:58  on account number:  1131273
Found mortgage date of $ 250000.0  date  2015-12-09 02:19:22  on account number:  1134844
Found mortgage date of $ 117170.0  date  2017-07-03 08:02:53  on account number:  412709
Found mortgage date of $ 0.0  date  1993-11-26 07:01:00  on account number:  904511
No mortgage identified on account number:  641518
Found mortgage date of $ 50000.0  date  2018-08-20 08:11:52  on account number:  6153607
Found mortgage date of $ 100000.0  date  2014-01-07 01:04:00  on account number:  1612115
Found mortgage date of $ 209000.0  date  2017-01-12 12:04:48  on account number:  812765
Found mortgage date of $ 485000.0  date  2016-05-25 11:18:46  on account number:  824886
Found mortgage date of $ 275000.0  date  2014-02-11 01:58:00  on account number:  1130691
Found mortgage date of $ 110500.0  date  2002-04-18 10:11:51  on account number:  6105254
Found mortgage date of $ 528418000.0  date  2015-02-09 04:17:51  on account number:  2632769
Found mortgage date of $ 85300.0  date  2015-12-07 09:26:35  on account number:  1134151
Found mortgage date of $ 168540.0  date  2016-01-12 01:10:01  on account number:  2635480
No mortgage identified on account number:  821533
Found mortgage date of $ 174000.0  date  2014-07-11 01:18:25  on account number:  2637717
No mortgage identified on account number:  101303
Found mortgage date of $ 313500.0  date  2018-07-18 10:05:46  on account number:  7210159
Found mortgage date of $ 100000.0  date  2011-06-20 02:50:07  on account number:  2802376
Found mortgage date of $ 117826.0  date  2015-11-12 01:14:00  on account number:  4505012
Found mortgage date of $ 15000.0  date  2016-11-29 08:48:47  on account number:  1127803
Found mortgage date of $ 75000.0  date  2014-04-28 01:52:00  on account number:  6300154
Found mortgage date of $ 185000.0  date  2009-03-19 10:46:32  on account number:  627329
Found mortgage date of $ 336000.0  date  2013-06-28 03:18:18  on account number:  550274
No mortgage identified on account number:  2632547
Found mortgage date of $ 71250.0  date  2004-07-09 10:17:47  on account number:  538043
Found mortgage date of $ 79632.0  date  2018-09-14 03:44:38  on account number:  2101921
Found mortgage date of $ 216500.0  date  2012-08-08 10:28:55  on account number:  550993
Found mortgage date of $ 109483.0  date  2015-05-15 03:49:54  on account number:  5327733
Found mortgage date of $ 229600.0  date  2015-10-30 01:01:53  on account number:  551346
Found mortgage date of $ 166920.0  date  2016-10-11 02:05:15  on account number:  2138775
Found mortgage date of $ 134000.0  date  2017-12-01 11:23:33  on account number:  4800699
No mortgage identified on account number:  1142747
Found mortgage date of $ 349600.0  date  2018-05-03 11:16:39  on account number:  1202675
Found mortgage date of $ 244921.0  date  2018-09-28 12:05:13  on account number:  820370
No mortgage identified on account number:  5329264
Found mortgage date of $ 120000.0  date  2013-08-21 03:54:42  on account number:  6305512
Found mortgage date of $ 124140.0  date  2015-05-11 11:15:55  on account number:  4804783
Found mortgage date of $ 254137.0  date  2018-01-04 08:25:51  on account number:  140120
Found mortgage date of $ 66200000.0  date  2017-08-31 02:51:56  on account number:  807184
No mortgage identified on account number:  5310059
Found mortgage date of $ 50000.0  date  2018-04-18 11:27:57  on account number:  823835
Found mortgage date of $ 34000.0  date  2018-11-13 02:17:11  on account number:  924946
Found mortgage date of $ 159750.0  date  2009-07-01 10:41:16  on account number:  5323169
Found mortgage date of $ 530000.0  date  2017-04-06 02:11:15  on account number:  902764
Found mortgage date of $ 17000.0  date  2017-05-11 02:06:50  on account number:  803031
No mortgage identified on account number:  820115
Found mortgage date of $ 181808.0  date  2017-11-20 01:23:25  on account number:  2300902
Found mortgage date of $ 150000.0  date  2017-09-22 11:34:37  on account number:  642255
No mortgage identified on account number:  6150127
Found mortgage date of $ 108000.0  date  2017-04-27 02:30:04  on account number:  1820857
Found mortgage date of $ 0.0  date  1985-12-17 07:01:00  on account number:  1406515
Found mortgage date of $ 248600.0  date  2015-06-26 11:10:45  on account number:  1202450
Found mortgage date of $ 76607.0  date  2013-03-08 01:48:54  on account number:  940336
Found mortgage date of $ 66000.0  date  2009-02-26 01:06:23  on account number:  1146653
Found mortgage date of $ 128500.0  date  2013-07-30 12:31:22  on account number:  3400016
Found mortgage date of $ 0.0  date  2017-03-02 01:13:08  on account number:  1202477
Found mortgage date of $ 52500.0  date  2002-04-25 08:58:22  on account number:  1606999
Found mortgage date of $ 131600.0  date  2018-09-05 01:32:51  on account number:  2139095
Found mortgage date of $ 112000.0  date  2000-10-25 12:18:38  on account number:  529303
Found mortgage date of $ 132000.0  date  2003-09-18 10:31:00  on account number:  1127375
Found mortgage date of $ 232000.0  date  2015-01-06 11:29:38  on account number:  822923
Found mortgage date of $ 296000.0  date  2016-09-27 11:14:47  on account number:  1132635
No mortgage identified on account number:  5005701
No mortgage identified on account number:  3517802
No mortgage identified on account number:  1130896
Found mortgage date of $ 45000.0  date  2008-01-03 03:05:27  on account number:  941835
Found mortgage date of $ 300000.0  date  2018-01-29 02:37:17  on account number:  2638069
Found mortgage date of $ 112677.0  date  2017-11-29 08:42:45  on account number:  2633674
Found mortgage date of $ 341600.0  date  2018-06-28 03:28:56  on account number:  537217
No mortgage identified on account number:  6148298
Found mortgage date of $ 95076.0  date  2012-11-20 08:44:48  on account number:  1135728
Found mortgage date of $ 50000.0  date  2018-09-13 02:30:18  on account number:  1103296
Found mortgage date of $ 600000.0  date  2018-02-08 01:37:00  on account number:  929492
Found mortgage date of $ 0.0  date  2001-05-07 03:08:42  on account number:  3800601
Found mortgage date of $ 145125.0  date  2018-02-20 10:10:00  on account number:  2636483
Found mortgage date of $ 28217.0  date  2009-05-07 11:38:51  on account number:  2128918
Found mortgage date of $ 94000.0  date  2013-08-06 02:09:59  on account number:  3521877
Found mortgage date of $ 170800.0  date  2016-02-11 01:13:31  on account number:  551142
No mortgage identified on account number:  823468
No mortgage identified on account number:  2606305
No mortgage identified on account number:  2634655
Found mortgage date of $ 2425.0  date  2013-06-26 02:47:31  on account number:  1102605
Found mortgage date of $ 154180.0  date  2012-03-23 02:50:22  on account number:  6151548
Found mortgage date of $ 118200.0  date  2013-01-07 08:47:30  on account number:  941523
Traceback (most recent call last):
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 321, in begin
    self.headers = self.msg = parse_headers(self.fp)
  File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 208, in parse_headers
    raise LineTooLong("header line")
http.client.LineTooLong: got more than 65536 bytes when reading header line

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 321, in begin
    self.headers = self.msg = parse_headers(self.fp)
  File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 208, in parse_headers
    raise LineTooLong("header line")
urllib3.exceptions.ProtocolError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute
    output = self.handle(*args, **options)
  File "/Users/Lucia/Code/Blue/property_project_ohio/propertyrecords/management/commands/warren-mort.py", line 10, in handle
    a.download_mortgage_info()
  File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 159, in download_mortgage_info
    self.identify_most_recent_mortage_for_each()
  File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 124, in identify_most_recent_mortage_for_each
    recorder_data = self.download_list_of_recorder_data_items(trimmed_parcel_number)
  File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 73, in download_list_of_recorder_data_items
    response = requests.request("POST", self.WARREN_INITIAL_SEARCH, data=payload, headers=self.HEADERS)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',))

It's clear to me from the error message neither what information is being returned in the headers that's causing a connection to be too long, nor if it's a one-time or persistent error.

Added an exception to catch the ConnectionError if it happens in this data request, log the parameters, and log the output. As of 14:07 ART, script running with no exception yet.

Added fix for : https://github.com/ludstuen90/property_project_ohio/issues/111 within branch 112-mortgage2

As of now, scraper has been running for 2 hrs 20 minutes without throwing this exception.

As per email, pushed updates that will allow us to continue scraping today, even without fully testing solution.

@walinchus Assigning this back to you, let me know if we're able to parse mortgage successfully!

We are getting closer! But looks like we are having another issue. Here is the error: [Several mortgages downloaded.]

Found mortgage date of $ 0.0 date 1993-08-12 07:01:00 on account number: 2608154 No mortgage identified on account number: 1129895 No mortgage identified on account number: 1200364 Found mortgage date of $ 299900.0 date 2016-04-05 03:51:28 on account number: 2637773 No mortgage identified on account number: 124389 Found mortgage date of $ 183998.0 date 2013-06-13 02:40:03 on account number: 2635745 Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect conn = self._new_conn() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x11769a748>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line utility.execute() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv self.execute(*args, *cmd_options) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute output = self.handle(args, options) File "/Users/Lucia/Code/Blue/property_project_ohio/propertyrecords/management/commands/warren-mort.py", line 10, in handle a.download_mortgage_info() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 172, in download_mortgage_info self.identify_most_recent_mortage_for_each() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 151, in identify_most_recent_mortage_for_each self.download_mortgage_detail({prop_to_parse: most_recent_item}) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 121, in download_mortgage_detail mortgage_amount = self.retrieve_document_details(mortgage_info['Id']) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 103, in retrieve_document_details response = requests.request("POST", self.WARREN_DOCUMENT_DETAIL, data=payload, headers=self.HEADERS) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11769a748>: Failed to establish a new connection: [Errno 60] Operation timed out',)) (env) Daddy-Mac-418:property_project_ohio Lucia$

That means the internet connection died! I'd just run again

More I think about it, it seems like we will need a "last scanned" field to not lose our place

I can make tonight

El lun., 14 de ene. de 2019 11:28, Lucia Walinchus notifications@github.com escribió:

We are getting closer! But looks like we are having another issue. Here is the error: [Several mortgages downloaded.]

Found mortgage date of $ 0.0 date 1993-08-12 07:01:00 on account number: 2608154 No mortgage identified on account number: 1129895 No mortgage identified on account number: 1200364 Found mortgage date of $ 299900.0 date 2016-04-05 03:51:28 on account number: 2637773 No mortgage identified on account number: 124389 Found mortgage date of $ 183998.0 date 2013-06-13 02:40:03 on account number: 2635745 Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect conn = self._new_conn() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x11769a748>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=' oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11769a748>: Failed to establish a new connection: [Errno 60] Operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/ init.py", line 381, in execute_from_command_line utility.execute() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/ init.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv self.execute(*args, *cmd_options) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute output = self.handle(args, options) File "/Users/Lucia/Code/Blue/property_project_ohio/propertyrecords/management/commands/warren-mort.py", line 10, in handle a.download_mortgage_info() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 172, in download_mortgage_info self.identify_most_recent_mortage_for_each() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 151, in identify_most_recent_mortage_for_each self.download_mortgage_detail({prop_to_parse: most_recent_item}) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 121, in download_mortgage_detail mortgage_amount = self.retrieve_document_details(mortgage_info['Id']) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 103, in retrieve_document_details response = requests.request("POST", self.WARREN_DOCUMENT_DETAIL, data=payload, headers=self.HEADERS) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host=' oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11769a748>: Failed to establish a new connection: [Errno 60] Operation timed out',)) (env) Daddy-Mac-418:property_project_ohio Lucia$

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ludstuen90/property_project_ohio/issues/112#issuecomment-454023181, or mute the thread https://github.com/notifications/unsubscribe-auth/AGgHK0JxiTzbZz6GBjX1UUQlClsFDIvcks5vDJQmgaJpZM4ZyEf1 .

Ah ok! Thanks.

Lucia Walinchus Executive Director

Phone: 646-397-7761 Email: Lucia@EyeOnOhio.com Twitter: @SoSaysLucia Want to get investigative news first? Sign up for our free newsletter http://EyeOnOhio.com! (Homepage bottom left)

On Mon, Jan 14, 2019 at 9:59 AM Lukas Udstuen notifications@github.com wrote:

That means the internet connection died! I'd just run again

More I think about it, it seems like we will need a "last scanned" field to not lose our place

I can make tonight

El lun., 14 de ene. de 2019 11:28, Lucia Walinchus < notifications@github.com> escribió:

We are getting closer! But looks like we are having another issue. Here is the error: [Several mortgages downloaded.]

Found mortgage date of $ 0.0 date 1993-08-12 07:01:00 on account number: 2608154 No mortgage identified on account number: 1129895 No mortgage identified on account number: 1200364 Found mortgage date of $ 299900.0 date 2016-04-05 03:51:28 on account number: 2637773 No mortgage identified on account number: 124389 Found mortgage date of $ 183998.0 date 2013-06-13 02:40:03 on account number: 2635745 Traceback (most recent call last): File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect conn = self._new_conn() File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x11769a748>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=' oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11769a748>: Failed to establish a new connection: [Errno 60] Operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/ init.py", line 381, in execute_from_command_line utility.execute() File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/ init.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv self.execute(*args, **cmd_options) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute output = self.handle(*args, **options) File

"/Users/Lucia/Code/Blue/property_project_ohio/propertyrecords/management/commands/warren-mort.py", line 10, in handle a.download_mortgage_info() File

"/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 172, in download_mortgage_info self.identify_most_recent_mortage_for_each() File

"/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 151, in identify_most_recent_mortage_for_each self.download_mortgage_detail({prop_to_parse: most_recent_item}) File

"/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 121, in download_mortgage_detail mortgage_amount = self.retrieve_document_details(mortgage_info['Id']) File

"/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 103, in retrieve_document_details response = requests.request("POST", self.WARREN_DOCUMENT_DETAIL, data=payload, headers=self.HEADERS) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File

"/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host=' oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11769a748>: Failed to establish a new connection: [Errno 60] Operation timed out',)) (env) Daddy-Mac-418:property_project_ohio Lucia$

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub < https://github.com/ludstuen90/property_project_ohio/issues/112#issuecomment-454023181 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AGgHK0JxiTzbZz6GBjX1UUQlClsFDIvcks5vDJQmgaJpZM4ZyEf1

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ludstuen90/property_project_ohio/issues/112#issuecomment-454033838, or mute the thread https://github.com/notifications/unsubscribe-auth/AYwbZe6UOUfvpYAOoxg3hwzhelandoQAks5vDJtegaJpZM4ZyEf1 .

Scraper stopped again last night, looks like a new connection error. I pulled in updates, verified that I was in the same folder as requirements.txt, updated the Django and restarted again:

Found mortgage date of $ 62614.0 date 2018-07-25 11:40:00 on account number: 641472 No mortgage identified on account number: 1805746 Found mortgage date of $ 30000.0 date 2015-11-23 03:02:35 on account number: 515002 Found mortgage date of $ 121500.0 date 2002-10-16 09:44:17 on account number: 2406035 Found mortgage date of $ 85500.0 date 2002-10-08 11:35:19 on account number: 1130512 No mortgage identified on account number: 6144187 Found mortgage date of $ 85000.0 date 2016-02-09 11:55:26 on account number: 1500589 Found mortgage date of $ 46000.0 date 2018-11-27 02:04:13 on account number: 2128586 Found mortgage date of $ 200000.0 date 2012-10-03 11:31:55 on account number: 5303206 Found mortgage date of $ 150000.0 date 2015-02-10 11:38:29 on account number: 2630043 Found mortgage date of $ 25000.0 date 2012-04-19 01:18:00 on account number: 1142666 Found mortgage date of $ 60000.0 date 2013-12-06 11:00:17 on account number: 4000498 Found mortgage date of $ 119000.0 date 2014-11-21 09:33:28 on account number: 2633763 Found mortgage date of $ 130000.0 date 2003-09-19 02:38:00 on account number: 3000680 Found mortgage date of $ 4350000.0 date 2012-02-08 03:04:08 on account number: 5500974 Found mortgage date of $ 119350.0 date 2009-10-06 08:53:20 on account number: 2119986 Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect conn = self._new_conn() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x11b56c278>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line utility.execute() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv self.execute(*args, *cmd_options) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute output = self.handle(args, options) File "/Users/Lucia/Code/Blue/property_project_ohio/propertyrecords/management/commands/warren-mort.py", line 10, in handle a.download_mortgage_info() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 172, in download_mortgage_info self.identify_most_recent_mortage_for_each() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 151, in identify_most_recent_mortage_for_each self.download_mortgage_detail({prop_to_parse: most_recent_item}) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 121, in download_mortgage_detail mortgage_amount = self.retrieve_document_details(mortgage_info['Id']) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 103, in retrieve_document_details response = requests.request("POST", self.WARREN_DOCUMENT_DETAIL, data=payload, headers=self.HEADERS) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11b56c278>: Failed to establish a new connection: [Errno 60] Operation timed out',)) (env) Daddy-Mac-418:property_project_ohio Lucia$ it pull origin master bash: it: command not found (env) Daddy-Mac-418:property_project_ohio Lucia$ git pull origin master remote: Enumerating objects: 34, done. remote: Counting objects: 100% (34/34), done. remote: Compressing objects: 100% (10/10), done. remote: Total 38 (delta 25), reused 31 (delta 24), pack-reused 4 Unpacking objects: 100% (38/38), done. From https://github.com/ludstuen90/property_project_ohio

branch master -> FETCH_HEAD e908345..24988ba master -> origin/master Updating e908345..24988ba Fast-forward propertyrecords/test_data/{cuyahoga_mort_deed_data.p => cuyahoga_mort_deed_data1.p} | Bin propertyrecords/test_data/cuyahoga_mort_deed_data2.p | Bin 0 -> 12592 bytes propertyrecords/tests.py | 27 ++++++++++++++++++--------- propertyrecords/utils.py | 46 +++++++++++++++++++++++++--------------------- requirements.txt | 2 +- scrapyohio/scrapyohio/spiders/cuyahoga-real.py | 52 ++++++++++++++++++++++++++++++++-------------------- scrapyohio/scrapyohio/spiders/cuyahoga.py | 2 -- 7 files changed, 76 insertions(+), 53 deletions(-) rename propertyrecords/test_data/{cuyahoga_mort_deed_data.p => cuyahoga_mort_deed_data1.p} (100%) create mode 100644 propertyrecords/test_data/cuyahoga_mort_deed_data2.p (env) Daddy-Mac-418:property_project_ohio Lucia$ cd scrapyohio (env) Daddy-Mac-418:scrapyohio Lucia$ ls scrapy.cfg scrapyohio (env) Daddy-Mac-418:scrapyohio Lucia$ cd .. (env) Daddy-Mac-418:property_project_ohio Lucia$ ls README.md env manage.py ohio propertyrecords requirements.txt scrapyohio (env) Daddy-Mac-418:property_project_ohio Lucia$ pip install -r requirements.txt Requirement already satisfied: asn1crypto==0.24.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.24.0) Requirement already satisfied: atomicwrites==1.2.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (1.2.1) Requirement already satisfied: attrs==18.2.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (18.2.0) Requirement already satisfied: Automat==0.7.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (0.7.0) Requirement already satisfied: beautifulsoup4==4.6.3 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (4.6.3) Requirement already satisfied: certifi==2018.10.15 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 6)) (2018.10.15) Requirement already satisfied: cffi==1.11.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 7)) (1.11.5) Requirement already satisfied: chardet==3.0.4 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 8)) (3.0.4) Requirement already satisfied: constantly==15.1.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 9)) (15.1.0) Requirement already satisfied: cryptography==2.4.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 10)) (2.4.1) Requirement already satisfied: cssselect==1.0.3 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 11)) (1.0.3) Collecting Django==2.1.5 (from -r requirements.txt (line 12)) Downloading https://files.pythonhosted.org/packages/36/50/078a42b4e9bedb94efd3e0278c0eb71650ed9672cdc91bd5542953bec17f/Django-2.1.5-py3-none-any.whl (7.3MB) 100% |████████████████████████████████| 7.3MB 1.4MB/s Requirement already satisfied: django-environ==0.4.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 13)) (0.4.5) Requirement already satisfied: hyperlink==18.0.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 14)) (18.0.0) Requirement already satisfied: idna==2.7 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 15)) (2.7) Requirement already satisfied: incremental==17.5.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 16)) (17.5.0) Requirement already satisfied: lxml==4.2.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 17)) (4.2.5) Requirement already satisfied: more-itertools==4.3.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 18)) (4.3.0) Requirement already satisfied: parsel==1.5.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 19)) (1.5.1) Requirement already satisfied: pluggy==0.8.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 20)) (0.8.0) Requirement already satisfied: psycopg2==2.7.6.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 21)) (2.7.6.1) Requirement already satisfied: py==1.7.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 22)) (1.7.0) Requirement already satisfied: pyasn1==0.4.4 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 23)) (0.4.4) Requirement already satisfied: pyasn1-modules==0.2.2 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 24)) (0.2.2) Requirement already satisfied: pycparser==2.19 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 25)) (2.19) Requirement already satisfied: PyDispatcher==2.0.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 26)) (2.0.5) Requirement already satisfied: PyHamcrest==1.9.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 27)) (1.9.0) Requirement already satisfied: pyOpenSSL==18.0.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 28)) (18.0.0) Requirement already satisfied: pytest==3.10.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 29)) (3.10.1) Requirement already satisfied: pytz==2018.7 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 30)) (2018.7) Requirement already satisfied: queuelib==1.5.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 31)) (1.5.0) Requirement already satisfied: requests==2.20.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 32)) (2.20.1) Requirement already satisfied: Scrapy==1.5.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 33)) (1.5.1) Requirement already satisfied: scrapy-djangoitem==1.1.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 34)) (1.1.1) Requirement already satisfied: service-identity==17.0.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 35)) (17.0.0) Requirement already satisfied: six==1.11.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 36)) (1.11.0) Requirement already satisfied: Twisted==18.9.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 37)) (18.9.0) Requirement already satisfied: urllib3==1.24.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 38)) (1.24.1) Requirement already satisfied: w3lib==1.19.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 39)) (1.19.0) Requirement already satisfied: zope.interface==4.6.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 40)) (4.6.0) Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from PyHamcrest==1.9.0->-r requirements.txt (line 27)) (40.6.2) Installing collected packages: Django Found existing installation: Django 2.1.3 Uninstalling Django-2.1.3: Successfully uninstalled Django-2.1.3 Successfully installed Django-2.1.5 (env) Daddy-Mac-418:property_project_ohio Lucia$ python manage.py warren-mort Found mortgage date of $ 113913.0 date 2013-02-08 02:46:18 on account number: 2143135 Found mortgage date of $ 204300.0 date 2016-04-18 10:03:02 on account number: 823656 Found mortgage date of $ 276454.0 date 2012-09-06 02:36:38 on account number: 7609949 Found mortgage date of $ 80000.0 date 2007-11-21 11:50:46 on account number: 2806428 No mortgage identified on account number: 2140659 No mortgage identified on account number: 2633267 No mortgage identified on account number: 551775 Found mortgage date of $ 111102.69 date 2013-07-30 12:10:41 on account number: 3513815 Found mortgage date of $ 250000.0 date 2005-11-09 12:22:40 on account number: 635879 ...

Ah dang, OK! I can code in the "start where you left off" feature tonight - could also take a Crack at something that handles connection errors!

El mié., 16 de ene. de 2019 08:45, Lucia Walinchus notifications@github.com escribió:

Scraper stopped again last night, looks like a new connection error. I pulled in updates, verified that I was in the same folder as requirements.txt, updated the Django and restarted again:

Found mortgage date of $ 62614.0 date 2018-07-25 11:40:00 on account number: 641472 No mortgage identified on account number: 1805746 Found mortgage date of $ 30000.0 date 2015-11-23 03:02:35 on account number: 515002 Found mortgage date of $ 121500.0 date 2002-10-16 09:44:17 on account number: 2406035 Found mortgage date of $ 85500.0 date 2002-10-08 11:35:19 on account number: 1130512 No mortgage identified on account number: 6144187 Found mortgage date of $ 85000.0 date 2016-02-09 11:55:26 on account number: 1500589 Found mortgage date of $ 46000.0 date 2018-11-27 02:04:13 on account number: 2128586 Found mortgage date of $ 200000.0 date 2012-10-03 11:31:55 on account number: 5303206 Found mortgage date of $ 150000.0 date 2015-02-10 11:38:29 on account number: 2630043 Found mortgage date of $ 25000.0 date 2012-04-19 01:18:00 on account number: 1142666 Found mortgage date of $ 60000.0 date 2013-12-06 11:00:17 on account number: 4000498 Found mortgage date of $ 119000.0 date 2014-11-21 09:33:28 on account number: 2633763 Found mortgage date of $ 130000.0 date 2003-09-19 02:38:00 on account number: 3000680 Found mortgage date of $ 4350000.0 date 2012-02-08 03:04:08 on account number: 5500974 Found mortgage date of $ 119350.0 date 2009-10-06 08:53:20 on account number: 2119986 Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect conn = self._new_conn() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x11b56c278>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=' oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11b56c278>: Failed to establish a new connection: [Errno 60] Operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/ init.py", line 381, in execute_from_command_line utility.execute() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/ init.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv self.execute(*args, *cmd_options) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute output = self.handle(args, options) File "/Users/Lucia/Code/Blue/property_project_ohio/propertyrecords/management/commands/warren-mort.py", line 10, in handle a.download_mortgage_info() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 172, in download_mortgage_info self.identify_most_recent_mortage_for_each() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 151, in identify_most_recent_mortage_for_each self.download_mortgage_detail({prop_to_parse: most_recent_item}) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 121, in download_mortgage_detail mortgage_amount = self.retrieve_document_details(mortgage_info['Id']) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 103, in retrieve_document_details response = requests.request("POST", self.WARREN_DOCUMENT_DETAIL, data=payload, headers=self.HEADERS) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host=' oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11b56c278>: Failed to establish a new connection: [Errno 60] Operation timed out',)) (env) Daddy-Mac-418:property_project_ohio Lucia$ it pull origin master bash: it: command not found (env) Daddy-Mac-418:property_project_ohio Lucia$ git pull origin master remote: Enumerating objects: 34, done. remote: Counting objects: 100% (34/34), done. remote: Compressing objects: 100% (10/10), done. remote: Total 38 (delta 25), reused 31 (delta 24), pack-reused 4 Unpacking objects: 100% (38/38), done. From https://github.com/ludstuen90/property_project_ohio

branch master -> FETCH_HEAD e908345 https://github.com/ludstuen90/property_project_ohio/commit/e90834524cfb7f54c484158c58ada584160aca85 ..24988ba https://github.com/ludstuen90/property_project_ohio/commit/24988ba01c16aef0ff3073b550bf8e97fe5099ec master -> origin/master Updating e908345 https://github.com/ludstuen90/property_project_ohio/commit/e90834524cfb7f54c484158c58ada584160aca85 ..24988ba https://github.com/ludstuen90/property_project_ohio/commit/24988ba01c16aef0ff3073b550bf8e97fe5099ec Fast-forward propertyrecords/test_data/{cuyahoga_mort_deed_data.p => cuyahoga_mort_deed_data1.p} | Bin propertyrecords/test_data/cuyahoga_mort_deed_data2.p | Bin 0 -> 12592 bytes propertyrecords/tests.py | 27 ++++++++++++++++++--------- propertyrecords/utils.py | 46 +++++++++++++++++++++++++--------------------- requirements.txt | 2 +- scrapyohio/scrapyohio/spiders/cuyahoga-real.py | 52 ++++++++++++++++++++++++++++++++-------------------- scrapyohio/scrapyohio/spiders/cuyahoga.py | 2 -- 7 files changed, 76 insertions(+), 53 deletions(-) rename propertyrecords/test_data/{cuyahoga_mort_deed_data.p => cuyahoga_mort_deed_data1.p} (100%) create mode 100644 propertyrecords/test_data/cuyahoga_mort_deed_data2.p (env) Daddy-Mac-418:property_project_ohio Lucia$ cd scrapyohio (env) Daddy-Mac-418:scrapyohio Lucia$ ls scrapy.cfg scrapyohio (env) Daddy-Mac-418:scrapyohio Lucia$ cd .. (env) Daddy-Mac-418:property_project_ohio Lucia$ ls README.md env manage.py ohio propertyrecords requirements.txt scrapyohio (env) Daddy-Mac-418:property_project_ohio Lucia$ pip install -r requirements.txt Requirement already satisfied: asn1crypto==0.24.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.24.0) Requirement already satisfied: atomicwrites==1.2.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (1.2.1) Requirement already satisfied: attrs==18.2.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (18.2.0) Requirement already satisfied: Automat==0.7.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (0.7.0) Requirement already satisfied: beautifulsoup4==4.6.3 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (4.6.3) Requirement already satisfied: certifi==2018.10.15 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 6)) (2018.10.15) Requirement already satisfied: cffi==1.11.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 7)) (1.11.5) Requirement already satisfied: chardet==3.0.4 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 8)) (3.0.4) Requirement already satisfied: constantly==15.1.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 9)) (15.1.0) Requirement already satisfied: cryptography==2.4.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 10)) (2.4.1) Requirement already satisfied: cssselect==1.0.3 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 11)) (1.0.3) Collecting Django==2.1.5 (from -r requirements.txt (line 12)) Downloading https://files.pythonhosted.org/packages/36/50/078a42b4e9bedb94efd3e0278c0eb71650ed9672cdc91bd5542953bec17f/Django-2.1.5-py3-none-any.whl (7.3MB) 100% |████████████████████████████████| 7.3MB 1.4MB/s Requirement already satisfied: django-environ==0.4.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 13)) (0.4.5) Requirement already satisfied: hyperlink==18.0.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 14)) (18.0.0) Requirement already satisfied: idna==2.7 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 15)) (2.7) Requirement already satisfied: incremental==17.5.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 16)) (17.5.0) Requirement already satisfied: lxml==4.2.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 17)) (4.2.5) Requirement already satisfied: more-itertools==4.3.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 18)) (4.3.0) Requirement already satisfied: parsel==1.5.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 19)) (1.5.1) Requirement already satisfied: pluggy==0.8.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 20)) (0.8.0) Requirement already satisfied: psycopg2==2.7.6.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 21)) (2.7.6.1) Requirement already satisfied: py==1.7.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 22)) (1.7.0) Requirement already satisfied: pyasn1==0.4.4 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 23)) (0.4.4) Requirement already satisfied: pyasn1-modules==0.2.2 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 24)) (0.2.2) Requirement already satisfied: pycparser==2.19 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 25)) (2.19) Requirement already satisfied: PyDispatcher==2.0.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 26)) (2.0.5) Requirement already satisfied: PyHamcrest==1.9.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 27)) (1.9.0) Requirement already satisfied: pyOpenSSL==18.0.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 28)) (18.0.0) Requirement already satisfied: pytest==3.10.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 29)) (3.10.1) Requirement already satisfied: pytz==2018.7 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 30)) (2018.7) Requirement already satisfied: queuelib==1.5.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 31)) (1.5.0) Requirement already satisfied: requests==2.20.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 32)) (2.20.1) Requirement already satisfied: Scrapy==1.5.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 33)) (1.5.1) Requirement already satisfied: scrapy-djangoitem==1.1.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 34)) (1.1.1) Requirement already satisfied: service-identity==17.0.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 35)) (17.0.0) Requirement already satisfied: six==1.11.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 36)) (1.11.0) Requirement already satisfied: Twisted==18.9.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 37)) (18.9.0) Requirement already satisfied: urllib3==1.24.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 38)) (1.24.1) Requirement already satisfied: w3lib==1.19.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 39)) (1.19.0) Requirement already satisfied: zope.interface==4.6.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 40)) (4.6.0) Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from PyHamcrest==1.9.0->-r requirements.txt (line 27)) (40.6.2) Installing collected packages: Django Found existing installation: Django 2.1.3 Uninstalling Django-2.1.3: Successfully uninstalled Django-2.1.3 Successfully installed Django-2.1.5 (env) Daddy-Mac-418:property_project_ohio Lucia$ python manage.py warren-mort Found mortgage date of $ 113913.0 date 2013-02-08 02:46:18 on account number: 2143135 Found mortgage date of $ 204300.0 date 2016-04-18 10:03:02 on account number: 823656 Found mortgage date of $ 276454.0 date 2012-09-06 02:36:38 on account number: 7609949 Found mortgage date of $ 80000.0 date 2007-11-21 11:50:46 on account number: 2806428 No mortgage identified on account number: 2140659 No mortgage identified on account number: 2633267 No mortgage identified on account number: 551775 Found mortgage date of $ 111102.69 date 2013-07-30 12:10:41 on account number: 3513815 Found mortgage date of $ 250000.0 date 2005-11-09 12:22:40 on account number: 635879 ...

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ludstuen90/property_project_ohio/issues/112#issuecomment-454750464, or mute the thread https://github.com/notifications/unsubscribe-auth/AGgHK_V_IT8mWb5nWLPsMMLbQcm2Rdb1ks5vDxDzgaJpZM4ZyEf1 .

Cool thanks. Sorry my internet is a bummer!

On Jan 16, 2019, at 6:50 AM, Lukas Udstuen notifications@github.com wrote:

Ah dang, OK! I can code in the "start where you left off" feature tonight - could also take a Crack at something that handles connection errors!

El mié., 16 de ene. de 2019 08:45, Lucia Walinchus notifications@github.com escribió:

Scraper stopped again last night, looks like a new connection error. I pulled in updates, verified that I was in the same folder as requirements.txt, updated the Django and restarted again:

Found mortgage date of $ 62614.0 date 2018-07-25 11:40:00 on account number: 641472 No mortgage identified on account number: 1805746 Found mortgage date of $ 30000.0 date 2015-11-23 03:02:35 on account number: 515002 Found mortgage date of $ 121500.0 date 2002-10-16 09:44:17 on account number: 2406035 Found mortgage date of $ 85500.0 date 2002-10-08 11:35:19 on account number: 1130512 No mortgage identified on account number: 6144187 Found mortgage date of $ 85000.0 date 2016-02-09 11:55:26 on account number: 1500589 Found mortgage date of $ 46000.0 date 2018-11-27 02:04:13 on account number: 2128586 Found mortgage date of $ 200000.0 date 2012-10-03 11:31:55 on account number: 5303206 Found mortgage date of $ 150000.0 date 2015-02-10 11:38:29 on account number: 2630043 Found mortgage date of $ 25000.0 date 2012-04-19 01:18:00 on account number: 1142666 Found mortgage date of $ 60000.0 date 2013-12-06 11:00:17 on account number: 4000498 Found mortgage date of $ 119000.0 date 2014-11-21 09:33:28 on account number: 2633763 Found mortgage date of $ 130000.0 date 2003-09-19 02:38:00 on account number: 3000680 Found mortgage date of $ 4350000.0 date 2012-02-08 03:04:08 on account number: 5500974 Found mortgage date of $ 119350.0 date 2009-10-06 08:53:20 on account number: 2119986 Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect conn = self._new_conn() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x11b56c278>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=' oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11b56c278>: Failed to establish a new connection: [Errno 60] Operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/ init.py", line 381, in execute_from_command_line utility.execute() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/ init.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv self.execute(*args, *cmd_options) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute output = self.handle(args, options) File "/Users/Lucia/Code/Blue/property_project_ohio/propertyrecords/management/commands/warren-mort.py", line 10, in handle a.download_mortgage_info() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 172, in download_mortgage_info self.identify_most_recent_mortage_for_each() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 151, in identify_most_recent_mortage_for_each self.download_mortgage_detail({prop_to_parse: most_recent_item}) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 121, in download_mortgage_detail mortgage_amount = self.retrieve_document_details(mortgage_info['Id']) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 103, in retrieve_document_details response = requests.request("POST", self.WARREN_DOCUMENT_DETAIL, data=payload, headers=self.HEADERS) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host=' oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/DocumentDetail (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x11b56c278>: Failed to establish a new connection: [Errno 60] Operation timed out',)) (env) Daddy-Mac-418:property_project_ohio Lucia$ it pull origin master bash: it: command not found (env) Daddy-Mac-418:property_project_ohio Lucia$ git pull origin master remote: Enumerating objects: 34, done. remote: Counting objects: 100% (34/34), done. remote: Compressing objects: 100% (10/10), done. remote: Total 38 (delta 25), reused 31 (delta 24), pack-reused 4 Unpacking objects: 100% (38/38), done. From https://github.com/ludstuen90/property_project_ohio

branch master -> FETCH_HEAD e908345 https://github.com/ludstuen90/property_project_ohio/commit/e90834524cfb7f54c484158c58ada584160aca85 ..24988ba https://github.com/ludstuen90/property_project_ohio/commit/24988ba01c16aef0ff3073b550bf8e97fe5099ec master -> origin/master Updating e908345 https://github.com/ludstuen90/property_project_ohio/commit/e90834524cfb7f54c484158c58ada584160aca85 ..24988ba https://github.com/ludstuen90/property_project_ohio/commit/24988ba01c16aef0ff3073b550bf8e97fe5099ec Fast-forward propertyrecords/test_data/{cuyahoga_mort_deed_data.p => cuyahoga_mort_deed_data1.p} | Bin propertyrecords/test_data/cuyahoga_mort_deed_data2.p | Bin 0 -> 12592 bytes propertyrecords/tests.py | 27 ++++++++++++++++++--------- propertyrecords/utils.py | 46 +++++++++++++++++++++++++--------------------- requirements.txt | 2 +- scrapyohio/scrapyohio/spiders/cuyahoga-real.py | 52 ++++++++++++++++++++++++++++++++-------------------- scrapyohio/scrapyohio/spiders/cuyahoga.py | 2 -- 7 files changed, 76 insertions(+), 53 deletions(-) rename propertyrecords/test_data/{cuyahoga_mort_deed_data.p => cuyahoga_mort_deed_data1.p} (100%) create mode 100644 propertyrecords/test_data/cuyahoga_mort_deed_data2.p (env) Daddy-Mac-418:property_project_ohio Lucia$ cd scrapyohio (env) Daddy-Mac-418:scrapyohio Lucia$ ls scrapy.cfg scrapyohio (env) Daddy-Mac-418:scrapyohio Lucia$ cd .. (env) Daddy-Mac-418:property_project_ohio Lucia$ ls README.md env manage.py ohio propertyrecords requirements.txt scrapyohio (env) Daddy-Mac-418:property_project_ohio Lucia$ pip install -r requirements.txt Requirement already satisfied: asn1crypto==0.24.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.24.0) Requirement already satisfied: atomicwrites==1.2.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (1.2.1) Requirement already satisfied: attrs==18.2.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (18.2.0) Requirement already satisfied: Automat==0.7.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (0.7.0) Requirement already satisfied: beautifulsoup4==4.6.3 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (4.6.3) Requirement already satisfied: certifi==2018.10.15 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 6)) (2018.10.15) Requirement already satisfied: cffi==1.11.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 7)) (1.11.5) Requirement already satisfied: chardet==3.0.4 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 8)) (3.0.4) Requirement already satisfied: constantly==15.1.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 9)) (15.1.0) Requirement already satisfied: cryptography==2.4.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 10)) (2.4.1) Requirement already satisfied: cssselect==1.0.3 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 11)) (1.0.3) Collecting Django==2.1.5 (from -r requirements.txt (line 12)) Downloading https://files.pythonhosted.org/packages/36/50/078a42b4e9bedb94efd3e0278c0eb71650ed9672cdc91bd5542953bec17f/Django-2.1.5-py3-none-any.whl (7.3MB) 100% |████████████████████████████████| 7.3MB 1.4MB/s Requirement already satisfied: django-environ==0.4.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 13)) (0.4.5) Requirement already satisfied: hyperlink==18.0.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 14)) (18.0.0) Requirement already satisfied: idna==2.7 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 15)) (2.7) Requirement already satisfied: incremental==17.5.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 16)) (17.5.0) Requirement already satisfied: lxml==4.2.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 17)) (4.2.5) Requirement already satisfied: more-itertools==4.3.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 18)) (4.3.0) Requirement already satisfied: parsel==1.5.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 19)) (1.5.1) Requirement already satisfied: pluggy==0.8.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 20)) (0.8.0) Requirement already satisfied: psycopg2==2.7.6.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 21)) (2.7.6.1) Requirement already satisfied: py==1.7.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 22)) (1.7.0) Requirement already satisfied: pyasn1==0.4.4 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 23)) (0.4.4) Requirement already satisfied: pyasn1-modules==0.2.2 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 24)) (0.2.2) Requirement already satisfied: pycparser==2.19 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 25)) (2.19) Requirement already satisfied: PyDispatcher==2.0.5 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 26)) (2.0.5) Requirement already satisfied: PyHamcrest==1.9.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 27)) (1.9.0) Requirement already satisfied: pyOpenSSL==18.0.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 28)) (18.0.0) Requirement already satisfied: pytest==3.10.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 29)) (3.10.1) Requirement already satisfied: pytz==2018.7 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 30)) (2018.7) Requirement already satisfied: queuelib==1.5.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 31)) (1.5.0) Requirement already satisfied: requests==2.20.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 32)) (2.20.1) Requirement already satisfied: Scrapy==1.5.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 33)) (1.5.1) Requirement already satisfied: scrapy-djangoitem==1.1.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 34)) (1.1.1) Requirement already satisfied: service-identity==17.0.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 35)) (17.0.0) Requirement already satisfied: six==1.11.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 36)) (1.11.0) Requirement already satisfied: Twisted==18.9.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 37)) (18.9.0) Requirement already satisfied: urllib3==1.24.1 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 38)) (1.24.1) Requirement already satisfied: w3lib==1.19.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 39)) (1.19.0) Requirement already satisfied: zope.interface==4.6.0 in ./env/lib/python3.6/site-packages (from -r requirements.txt (line 40)) (4.6.0) Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from PyHamcrest==1.9.0->-r requirements.txt (line 27)) (40.6.2) Installing collected packages: Django Found existing installation: Django 2.1.3 Uninstalling Django-2.1.3: Successfully uninstalled Django-2.1.3 Successfully installed Django-2.1.5 (env) Daddy-Mac-418:property_project_ohio Lucia$ python manage.py warren-mort Found mortgage date of $ 113913.0 date 2013-02-08 02:46:18 on account number: 2143135 Found mortgage date of $ 204300.0 date 2016-04-18 10:03:02 on account number: 823656 Found mortgage date of $ 276454.0 date 2012-09-06 02:36:38 on account number: 7609949 Found mortgage date of $ 80000.0 date 2007-11-21 11:50:46 on account number: 2806428 No mortgage identified on account number: 2140659 No mortgage identified on account number: 2633267 No mortgage identified on account number: 551775 Found mortgage date of $ 111102.69 date 2013-07-30 12:10:41 on account number: 3513815 Found mortgage date of $ 250000.0 date 2005-11-09 12:22:40 on account number: 635879 ...

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ludstuen90/property_project_ohio/issues/112#issuecomment-454750464, or mute the thread https://github.com/notifications/unsubscribe-auth/AGgHK_V_IT8mWb5nWLPsMMLbQcm2Rdb1ks5vDxDzgaJpZM4ZyEf1 .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Hmm is this a connection error? it looks slightly different.

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect conn = self._new_conn() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x111cd29b0>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 89, in download_list_of_recorder_data_items response = requests.request("POST", self.WARREN_INITIAL_SEARCH, data=payload, headers=self.HEADERS) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='oh3laredo.fidlar.com', port=443): Max retries exceeded with url: /OHWarren/Scrap.WebService.Ava/breeze/breeze/Search (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x111cd29b0>: Failed to establish a new connection: [Errno 60] Operation timed out',))

During handling of the above exception, another exception occurred:

So far we have: 1025 properties with a mortgage of 0 38,000 properties with a mortgage 61,475 or so to go

Anyway, it's back up and running again. I rain the process for a few seconds, then stopped it, then verified that the database is pulling in more records, and then started it again. But I figured I should update the progress here.

Also, we need a way to record the value of Land Contracts, which it looks like are in the recorder's office as "LC." For example, this property was sold in 2018 for $180,000:

But they are on the books as $137,710: http://www.co.warren.oh.us/property_search/summary.aspx?account_nbr=1111809

A 23.5% increase.

Re Error: How about this: I'll have the script keep trying, unliminted number of times, regardless of if the connection fails, until we are able to scrape all the records.

Re LC: Good catch! That definitely looks like something we'd want. How do we rank this in terms of priority? Should we stop development on Cuyahoga/Franklin to go back and work on this?

Sorry for getting back to you so late. Yeah we should probably go back and catch the LCs. So I can start writing R scripts once I get a full database.

-Lucia

Lucia Walinchus Executive Director

Phone: 646-397-7761 Email: Lucia@EyeOnOhio.com Twitter: @SoSaysLucia Want to get investigative news first? Sign up for our free newsletter http://EyeOnOhio.com! (Homepage bottom left)

On Mon, Jan 21, 2019 at 4:56 PM Lukas Udstuen notifications@github.com wrote:

Re Error: How about this: I'll have the script keep trying, unliminted number of times, regardless of if the connection fails, until we are able to scrape all the records.

Re LC: Good catch! That definitely looks like something we'd want. How do we rank this in terms of priority? Should we stop development on Cuyahoga/Franklin to go back and work on this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ludstuen90/property_project_ohio/issues/112#issuecomment-456209031, or mute the thread https://github.com/notifications/unsubscribe-auth/AYwbZdus2HmsjSC59KunlupO_4mGHWOeks5vFjehgaJpZM4ZyEf1 .

Hmm:

Found mortgage date of $ 344000.0 date 2016-09-19 12:09:04 on account number: 1131898 Found mortgage date of $ 105000.0 date 2016-11-29 09:43:04 on account number: 6150073 Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request six.raise_from(e, None) File "", line 2, in raise_from File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request httplib_response = conn.getresponse() File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 1331, in getresponse response.begin() File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 321, in begin self.headers = self.msg = parse_headers(self.fp) File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 208, in parse_headers raise LineTooLong("header line") http.client.LineTooLong: got more than 65536 bytes when reading header line

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment raise six.reraise(type(error), error, _stacktrace) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise raise value.with_traceback(tb) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request six.raise_from(e, None) File "", line 2, in raise_from File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request httplib_response = conn.getresponse() File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 1331, in getresponse response.begin() File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 321, in begin self.headers = self.msg = parse_headers(self.fp) File "/Users/Lucia/anaconda3/lib/python3.6/http/client.py", line 208, in parse_headers raise LineTooLong("header line") urllib3.exceptions.ProtocolError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line utility.execute() File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv self.execute(*args, *cmd_options) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute output = self.handle(args, options) File "/Users/Lucia/Code/Blue/property_project_ohio/propertyrecords/management/commands/warren-mort.py", line 10, in handle a.download_mortgage_info() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 196, in download_mortgage_info self.identify_most_recent_mortage_for_each() File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 175, in identify_most_recent_mortage_for_each self.download_mortgage_detail({prop_to_parse: most_recent_item}) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 144, in download_mortgage_detail mortgage_amount = self.retrieve_document_details(mortgage_info['Id']) File "/Users/Lucia/Code/Blue/property_project_ohio/scrapyohio/scrapyohio/scraper_helpers/warren_mortgage.py", line 126, in retrieve_document_details response = s.request("POST", self.WARREN_DOCUMENT_DETAIL, data=payload, headers=self.HEADERS) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/Users/Lucia/Code/Blue/property_project_ohio/env/lib/python3.6/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',))

Also, this isn't as important, but is there a way to display more of the numbers in the page display? There are 1005 pages, so if I want to see how far the mortgage scraping has gone, but I don't know how far it is, I have to start from one, scroll down, click on four, scroll down click on 7 and so on until I see a number for the mortgage.

(I have the scraper back up and running now, I just figured I should give you the error heads up. And let you know why it stopped so we can address it.) Thanks!

Awesome, thanks for all this information! I made a quick Dashboard to help us see how far along the scraper has been. If you download the updates (Stop server, git pull origin master, restart server), and then go to the home page, you'll see something like this (with more accurate numbers)

This is really basic, but I figured something like this will help add some sanity to the scraping process, to get more information on where we're at.

If there's any more information you'd like added to the dashboard, let me know whatever would be helpful.

I'll take a deeper look at adding LC information and the bug tomorrow!

Great thanks!

On Jan 28, 2019, at 9:32 PM, Lukas Udstuen notifications@github.com wrote:

Awesome, thanks for all this information! I made a quick Dashboard to help us see how far along the scraper has been. If you download the updates (Stop server, git pull origin master, restart server), and then go to the home page, you'll see something like this (with more accurate numbers)

This is really basic, but I figured something like this will help add some sanity to the scraping process, to get more information on where we're at.

If there's any more information you'd like added to the dashboard, let me know whatever would be helpful.

I'll take a deeper look at adding LC information and the bug tomorrow!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

@walinchus It looks like we've resolved all the issues in this thread, no? If you agree, would you mind closing?

ludstuen90 / property_project_ohio

Mortgage Fix #112