search

JnBrymn / minglbot

0 stars 0 forks source link

Why does get_relations need to be run multiple times to get everyone? #9

Closed JnBrymn closed 9 years ago

JnBrymn commented 10 years ago

Sometime I'm having to retrieve relations twice to get them all. Try running a get_relations twice in a row. If it pulls back new relations on the run, that's a problem. (Similar to https://github.com/JnBrymn/minglbot/issues/10)

JnBrymn commented 9 years ago

Recreating problem: I had played with neo4j for a while and then I deleted all the relations and nodes and added the screen_name/id uniqueness constraints. Then I ran:

vc_customers = ["NinjaBunny_","AaronBBrown777","itaykahana","tiopaul","MartinLoy","LaineVCampbell","duanegran","jbarciauskas","snoogindoogin","chuckhagenbuch","yournameistoby","blalor","GiantEvolving","cvshumake","insyteful","samkottler","timgoodaire","DynData","egon1024","MathYourLife","j_manero","obfuscurity","bigg33k","alexsergeyev","akachler","mrtazz","ickymettle","lozzd","allspaw","Ryan_Frantz","mikearpaia","d87tech","techwolf359","wcgallego","jimbartus","dominis"]
friends = m.get_friends(vc_customers)
friends = m.hydrate_users(friends)
friends = m.get_friends(vc_customers)

And it had to go retrieve some users again.

INFO:root:Getting friends. ids:([]) screen_names:(['NinjaBunny_', 'AaronBBrown777', 'itaykahana', 'tiopaul', 'MartinLoy', 'LaineVCampbell', 'duanegran', 'jbarciauskas', 'snoogindoogin', 'chuckhagenbuch', 'yournameistoby', 'blalor', 'GiantEvolving', 'cvshumake', 'insyteful', 'samkottler', 'timgoodaire', 'DynData', 'egon1024', 'MathYourLife', 'j_manero', 'obfuscurity', 'bigg33k', 'alexsergeyev', 'akachler', 'mrtazz', 'ickymettle', 'lozzd', 'allspaw', 'Ryan_Frantz', 'mikearpaia', 'd87tech', 'techwolf359', 'wcgallego', 'jimbartus', 'dominis'])
INFO:root:Getting friends of ninjabunny_ from Twitter
INFO:root:Getting friends of aaronbbrown777 from Twitter
INFO:root:Getting friends of itaykahana from Twitter
INFO:root:Getting friends of tiopaul from Twitter

INFO:root:Getting friends of martinloy from Twitter
INFO:root:Getting friends of lainevcampbell from Twitter
INFO:root:Getting friends of duanegran from Twitter
INFO:root:Getting friends of jbarciauskas from Twitter
INFO:root:Getting friends of snoogindoogin from Twitter
INFO:root:Getting friends of chuckhagenbuch from Twitter
INFO:root:Getting friends of yournameistoby from Twitter
INFO:root:Getting friends of blalor from Twitter
INFO:root:Getting friends of giantevolving from Twitter
INFO:root:Getting friends of cvshumake from Twitter
INFO:root:Getting friends of insyteful from Twitter
INFO:root:Getting friends of samkottler from Twitter
INFO:root:Rate limit exceeded. Sleeping
INFO:root:Rate limit exceeded. Sleeping
INFO:root:Rate limit exceeded. Sleeping
INFO:root:Getting friends of timgoodaire from Twitter
INFO:root:Getting friends of dyndata from Twitter
INFO:root:Getting friends of egon1024 from Twitter
INFO:root:Getting friends of mathyourlife from Twitter
INFO:root:Getting friends of j_manero from Twitter
INFO:root:Getting friends of obfuscurity from Twitter
INFO:root:Getting friends of bigg33k from Twitter
INFO:root:Getting friends of alexsergeyev from Twitter
INFO:root:Getting friends of akachler from Twitter
INFO:root:Getting friends of mrtazz from Twitter
INFO:root:Getting friends of ickymettle from Twitter
INFO:root:Getting friends of lozzd from Twitter
INFO:root:Getting friends of allspaw from Twitter
INFO:root:Getting friends of ryan_frantz from Twitter
INFO:root:Getting friends of mikearpaia from Twitter
INFO:root:Rate limit exceeded. Sleeping
INFO:root:Getting friends of d87tech from Twitter
INFO:root:Getting friends of techwolf359 from Twitter
INFO:root:Getting friends of wcgallego from Twitter
INFO:root:Getting friends of jimbartus from Twitter
INFO:root:Getting friends of dominis from Twitter
INFO:root:Hydrating from neo4j. ids:([]) screen_names:([]) users:([User(id=13378422,screen_name=None), User(id=28325059,screen_name=None), User(id=810781,screen_name=None), User(id=91333167,screen_name=None), User(id=2334251,screen_name=None), User(id=66432490,screen_name=None), User(id=9887162,screen_name=None), User(id=112601087,screen_name=None), User(id=20456848,screen_name=None), User(id=14146300,screen_name=None), User(id=9580822,screen_name=None), User(id=264481774,screen_name=None), User(id=260044118,screen_name=None), User(id=928961,screen_name=None), User(id=15782607,screen_name=None), User(id=8627682,screen_name=None), User(id=7152642,screen_name=None), User(id=15696107,screen_name=None), User(id=743222730,screen_name=None), User(id=167499429,screen_name=None), User(id=673803,screen_name=None), User(id=22407045,screen_name=None), User(id=13093162,screen_name=None), User(id=42007556,screen_name=None), User(id=17840694,screen_name=None), User(id=17173207,screen_name=None), User(id=9723702,screen_name=None), User(id=19725644,screen_name=None), User(id=10201312,screen_name=None), User(id=208212889,screen_name=None), User(id=47,screen_name=None), User(id=26751851,screen_name=None), User(id=632343,screen_name=None), User(id=20865431,screen_name=None), User(id=13035,screen_name=None), User(id=13444792,screen_name=None), User(id=354998770,screen_name=None), User(id=22356683,screen_name=None), User(id=40887051,screen_name=None), User(id=305899937,screen_name=None), User(id=6844292,screen_name=None), User(id=16560367,screen_name=None), User(id=5440622,screen_name=None), User(id=966271,screen_name=None), User(id=14375294,screen_name=None), User(id=37211927,screen_name=None), User(id=14586723,screen_name=None), User(id=13334762,screen_name=None), User(id=112614654,screen_name=None), User(id=41036966,screen_name=None), User(id=320846092,screen_name=None), User(id=692763,screen_name=None), User(id=212632603,screen_name=None), User(id=35373186,screen_name=None), User(id=49820803,screen_name=None), User(id=760627034,screen_name=None), User(id=11348282,screen_name=None), User(id=97665755,screen_name=None), User(id=335969218,screen_name=None), User(id=19502601,screen_name=None), User(id=226976689,screen_name=None), User(id=17025041,screen_name=None), User(id=150486075,screen_name=None), User(id=115485051,screen_name=None), User(id=15431118,screen_name=None), User(id=23992877,screen_name=None), User(id=248507370,screen_name=None), User(id=14466413,screen_name=None), User(id=8015252,screen_name=None), User(id=687123,screen_name=None), User(id=15661871,screen_name=None), User(id=11800282,screen_name=None), User(id=9980812,screen_name=None), User(id=313568566,screen_name=None), User(id=765548,screen_name=None), User(id=153268466,screen_name=None), User(id=11522502,screen_name=None), User(id=17194060,screen_name=None), User(id=80703,screen_name=None), User(id=171947193,screen_name=None), User(id=822284,screen_name=None), User(id=38686230,screen_name=None), User(id=2384071,screen_name=None), User(id=14079705,screen_name=None), User(id=13682312,screen_name=None), User(id=18243723,screen_name=None), User(id=360092358,screen_name=None), User(id=24228154,screen_name=None), User(id=15473958,screen_name=None), User(id=15158246,screen_name=None), User(id=398190345,screen_name=None), User(id=523645507,screen_name=None), User(id=1026774504,screen_name=None), User(id=331912537,screen_name=None), User(id=14380900,screen_name=None), User(id=14086716,screen_name=None), User(id=18574825,screen_name=None), User(id=577398586,screen_name=None), User(id=7136992,screen_name=None), User(id=1465659204,screen_name=None)])
INFO:root:Hydrating ids from Twitter. ids:([13378422, 28325059, 810781, 91333167, 2334251, 66432490, 9887162, 112601087, 20456848, 14146300, 9580822, 264481774, 260044118, 928961, 15782607, 8627682, 7152642, 15696107, 743222730, 167499429, 673803, 22407045, 13093162, 42007556, 17840694, 17173207, 9723702, 19725644, 10201312, 208212889, 47, 26751851, 632343, 20865431, 13035, 13444792, 354998770, 22356683, 40887051, 305899937, 6844292, 16560367, 5440622, 966271, 14375294, 37211927, 14586723, 13334762, 112614654, 41036966, 320846092, 692763, 212632603, 35373186, 49820803, 760627034, 11348282, 97665755, 335969218, 19502601, 226976689, 17025041, 150486075, 115485051, 15431118, 23992877, 248507370, 14466413, 8015252, 687123, 15661871, 11800282, 9980812, 313568566, 765548, 153268466, 11522502, 17194060, 80703, 171947193, 822284, 38686230, 2384071, 14079705, 13682312, 18243723, 360092358, 24228154, 15473958, 15158246, 398190345, 523645507, 1026774504, 331912537, 14380900, 14086716, 18574825, 577398586, 7136992, 1465659204]) screen_names:([]) from Twitter
INFO:root:Repairing id-screen_name conflict for id 13378422 and screen_name allspaw
INFO:root:Repairing id-screen_name conflict for id 810781 and screen_name mrtazz
INFO:root:Repairing id-screen_name conflict for id 66432490 and screen_name obfuscurity
INFO:root:Repairing id-screen_name conflict for id 8627682 and screen_name lozzd
INFO:root:Repairing id-screen_name conflict for id 354998770 and screen_name Ryan_Frantz
INFO:root:Repairing id-screen_name conflict for id 22356683 and screen_name ickymettle
INFO:root:Repairing id-screen_name conflict for id 16560367 and screen_name akachler
INFO:root:Repairing id-screen_name conflict for id 335969218 and screen_name d87tech
INFO:root:Repairing id-screen_name conflict for id 150486075 and screen_name mikearpaia
INFO:root:Repairing id-screen_name conflict for id 23992877 and screen_name techwolf359
INFO:root:Repairing id-screen_name conflict for id 11800282 and screen_name wcgallego
    INFO:root:Getting friends. ids:([]) screen_names:(['NinjaBunny_', 'AaronBBrown777', 'itaykahana', 'tiopaul', 'MartinLoy', 'LaineVCampbell', 'duanegran', 'jbarciauskas', 'snoogindoogin', 'chuckhagenbuch', 'yournameistoby', 'blalor', 'GiantEvolving', 'cvshumake', 'insyteful', 'samkottler', 'timgoodaire', 'DynData', 'egon1024', 'MathYourLife', 'j_manero', 'obfuscurity', 'bigg33k', 'alexsergeyev', 'akachler', 'mrtazz', 'ickymettle', 'lozzd', 'allspaw', 'Ryan_Frantz', 'mikearpaia', 'd87tech', 'techwolf359', 'wcgallego', 'jimbartus', 'dominis'])
INFO:root:Getting friends of obfuscurity from Twitter
INFO:root:Getting friends of akachler from Twitter
INFO:root:Getting friends of mrtazz from Twitter
INFO:root:Getting friends of ickymettle from Twitter
INFO:root:Getting friends of lozzd from Twitter
INFO:root:Getting friends of allspaw from Twitter
INFO:root:Getting friends of ryan_frantz from Twitter
INFO:root:Getting friends of mikearpaia from Twitter
INFO:root:Getting friends of d87tech from Twitter
INFO:root:Getting friends of techwolf359 from Twitter
INFO:root:Getting friends of wcgallego from Twitter
JnBrymn commented 9 years ago

The problem appears to be associated with those individuals where had a id-screen_name conflict repaired.

Can I replicate the problem by

friends = m.get_friends("jnbrymn")
friends = m.get_friends("softwaredoug")
friends = m.hydrate_users(friends) #should notice the inconsistency between "jnbrymn" and doug's friend
friends = m.get_friends("jnbrymn") #will this pull down jnbrymn again?
JnBrymn commented 9 years ago

I think the problem is in this line. Apparently friend_found_at and follower_found_at is being lost when id-screen_name conflict is repaired.

JnBrymn commented 9 years ago

replaced by https://github.com/JnBrymn/minglbot/issues/35