[ISSUE] IDLE keeps closing without error message

[awxinmin]

Make sure you've checked the following:

• [checked] Python version is 3.6;
• [checked] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
• [checked] I have searched the issues and there are no duplicates of this issue/question/request.

Command Ran

c = twint.Config()
c.Count = True c.Stats = True c.Database = "run1_exchangehacks.db" c.Debug = True c.Search = "exchange OR hacked"

c.Tor_control_port = 9051 c.Tor_control_password = "" #inserts password

for combi in daterange('2018-01-01','2019-12-18','M'): c.Since = combi[0] # inclusive c.Until = combi[1] # exclusive twint.run.Search(c) time.sleep(2)

daterange() returns --> [['2018-01-01', '2018-01-31'], ['2018-01-31', '2018-02-28'], ['2018-02-28', '2018-03-31'], ['2018-03-31', '2018-04-30'], ['2018-04-30', '2018-05-31'], ...]

Description of Issue

I have been using trying to use twint to scrape twitter data but ive faced some problems. I am running this using IDLE 3.6.8 as i read previously that twint had some compatibility issues with jupyter notebook. however after scraping about 4k tweets, my IDLE shell will just close without any error messages and scraping stops. I have been facing this problem for quite a few times and I would really appreciate some help

Environment Details

Ubuntu 18.04.3 Running this in IDLE.

[pielco11]

May you try to run Python from shell instead of IDLE and report the last few lines of the output, please?

Twint stops if Twitter stops sending data, so you may have just reached the end of the scraping session

[awxinmin]

Hi thanks for the reply. yes I am running it in shell but it closes before without warning thus it is impossible to report the last few lines of output. It is unlikely that it has reached the end as I’ve tried scraping it for example from 2018-01-01 to 2018-01-30 but somehow it will stop at around 2018-01-06

[pielco11]

Which shell are you using? If I'm not wrong, IDLE closes after Twint stops, may you use gnome-terminal of something similar?

[awxinmin]

I am using the default shell that comes with IDLE

[pielco11]

You should use a shell without running IDLE

[awxinmin]

Oh so I just run my code using bash? Just curious, why does IDLE affect this?

[pielco11]

I don't know how does IDLE work, but if I remember correctly it exits as soon as your script exits. Using bash should be fine

[puddlejumper90]

I have had the same issue with Thonny. Looks like this is an UNRESOLVED issue. I have noticed that reducing record pulls from 5,000 to 1 allows my application to get farther through the process that I have created, but the application still closes several steps after the twint processes have completed.

It may not be a twint issue, but I only encounter this issue when using twint. It's not normal that an IDE would simply terminate instead of stopping with an error message unless something is telling the application to terminate itself.

FINAL UPDATE 6/7/2021: I cannot determine where the issue is coming from. Even after trying several different queries and coming back a few days later, there seems to be no answer as to why this is occurring. Unfortunately, I will have to abandon my project. Social networking sites try to do their best to block web scraping, it could be that something on the Twitter side is closing down any attempt to capture Twitter data.

I will be following this project for future updates, but as of right now, there is not much use for a project that does not work.