Open awxinmin opened 4 years ago
May you try to run Python from shell instead of IDLE and report the last few lines of the output, please?
Twint stops if Twitter stops sending data, so you may have just reached the end of the scraping session
Hi thanks for the reply. yes I am running it in shell but it closes before without warning thus it is impossible to report the last few lines of output. It is unlikely that it has reached the end as I’ve tried scraping it for example from 2018-01-01 to 2018-01-30 but somehow it will stop at around 2018-01-06
Which shell are you using? If I'm not wrong, IDLE closes after Twint stops, may you use gnome-terminal
of something similar?
I am using the default shell that comes with IDLE
You should use a shell without running IDLE
Oh so I just run my code using bash? Just curious, why does IDLE affect this?
I don't know how does IDLE work, but if I remember correctly it exits as soon as your script exits. Using bash should be fine
I have had the same issue with Thonny. Looks like this is an UNRESOLVED issue. I have noticed that reducing record pulls from 5,000 to 1 allows my application to get farther through the process that I have created, but the application still closes several steps after the twint processes have completed.
It may not be a twint issue, but I only encounter this issue when using twint. It's not normal that an IDE would simply terminate instead of stopping with an error message unless something is telling the application to terminate itself.
FINAL UPDATE 6/7/2021: I cannot determine where the issue is coming from. Even after trying several different queries and coming back a few days later, there seems to be no answer as to why this is occurring. Unfortunately, I will have to abandon my project. Social networking sites try to do their best to block web scraping, it could be that something on the Twitter side is closing down any attempt to capture Twitter data.
I will be following this project for future updates, but as of right now, there is not much use for a project that does not work.
pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint
;Command Ran
c = twint.Config()
c.Count = True c.Stats = True c.Database = "run1_exchangehacks.db" c.Debug = True c.Search = "exchange OR hacked"
c.Tor_control_port = 9051 c.Tor_control_password = "" #inserts password
for combi in daterange('2018-01-01','2019-12-18','M'): c.Since = combi[0] # inclusive c.Until = combi[1] # exclusive twint.run.Search(c) time.sleep(2)
daterange() returns --> [['2018-01-01', '2018-01-31'], ['2018-01-31', '2018-02-28'], ['2018-02-28', '2018-03-31'], ['2018-03-31', '2018-04-30'], ['2018-04-30', '2018-05-31'], ...]
Description of Issue
Environment Details