search

EU-ECDC / epitweetr

ECDC Early warning tool using Twitter data
European Union Public License 1.2
55 stars 14 forks source link

dependencies aborted #52

Closed Abdouzster closed 2 years ago

Abdouzster commented 2 years ago

I keep having the dependencies aborted with the error below. It is taking forever to convert files.

Max number of retries reached failed while migrating any old json files to embeded database getting dependencies Error in store_geo(lines, created_dates, chunk_size = chunk_size): The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!

forchard commented 2 years ago

Hi Abdouzster, If it is taking too long, Maybe you can choose not to migrate the files to the new format. You will not see any impact on reports or alerts. The migration allows you to use a few features with your old data:

If this is not an option, can you tell us how many files were successfully migrated and after how much time? Thanks!

Abdouzster commented 2 years ago

Hi Forchard, I will try to quantify the number of files but it is more about every round ending up with an error. Now dependencies aborted with another error below Max number of retries reached failed while migrating any old json files to embeded database getting dependencies Error in store_geo(lines, created_dates, chunk_size = chunk_size): The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!The server was not able to produce a timely response to your request. Please try again in a short while!Send failure: Connection was abortedSend failure: Connection was abortedSend failure: Connection was abortedSend failure: Connection was abortedSend failure: Connection was abortedSend failure: Connection was aborted

Thanks

Abdouzster commented 2 years ago

Hi Forchard, How to skip data migration? Should I just change the working directory to a clean one or no need for that? Thanks, Abdelhamid

forchard commented 2 years ago

Hi @Abdouzster Just move the files within $EPI_HOME/tweets/search and $EPI_HOME/tweets/geo to $EPI_HOME/tweets/search_archive and $EPI_HOME/tweets/geo_archive

Anyway, since the data collection has not been running from more than one week you can also start on a new clean $EPI_HOME

Best Francisco

Abdouzster commented 2 years ago

Hi Francisco, I moved the files and cleaned the directory. I am waiting for dependencies. how long should it take? It is still showing me the message "Embeded database is running, please turn it OFF in order to download dependencies. You can do this on the task scheduler on windows or manually kill the process on other platforms." I am sure the database is not running Thanks, Abdelhamid

forchard commented 2 years ago

Hi, This task should not be long (30 minutes max)

To test if the embedded database is running please check http://localhost:8080/ping on your web browser. You should not be able to reach the page, otherwise the database (or something else) is using that port.

You may also check

Of this troubleshoot does not help, then please share the logs on the detect loop we should see if there is any error in there (like in internet access issue. The detect log file should be on c:/Users/[you]/epitweetr/detect.log.

Abdouzster commented 2 years ago

Thanks Francisco! I passed this situation. However, I have the search loop error that is keep happening. 2022-03-10 09:14:54 [INFO]: -------> searching for topic Botulism from until (last tweet) Error in value[3L] : argument is not interpretable as logical Error in if (n1%%n2) warning("length(e1) not a multiple length(e2)"): argument is not interpretable as logical

forchard commented 2 years ago

Does the search works for other topics or Botulism is your the first topic being tried? This error may be related to an issue on you topics.json file or your topic.xlsx file, can you share them? Thanks,

Abdouzster commented 2 years ago

The search execute until specific topic and stop. not related to botulism. it was west nile. I removed it and it stoped at botulism. I am currently using the default file. here is the sequence of execution again:

epitweetr::search_loop("C:/Users/abdou/Documents/Twitter") 2022-03-10 09:14:43 [INFO]: -------> searching for topic Measles from 1501703489705193476 until (last tweet) Requesting token on behalf of user... Waiting for authentication in browser... Press Esc/Ctrl + C to abort Authentication complete. 2022-03-10 09:14:48 [INFO]: -------> searching for topic Rubella from 1501691422805663747 until (last tweet) 2022-03-10 09:14:48 [INFO]: -------> searching for topic Mumps from 1501691341771739138 until (last tweet) 2022-03-10 09:14:49 [INFO]: -------> searching for topic Dengue from 1501703534877847557 until (last tweet) 2022-03-10 09:14:50 [INFO]: -------> searching for topic Haemorrhagic fever from 1501702062614134795 until (last tweet) 2022-03-10 09:14:51 [INFO]: -------> searching for topic Avian influenza from 1501691123395436558 until (last tweet) 2022-03-10 09:14:51 [INFO]: -------> searching for topic Chikungunya from 1501691495564337155 until (last tweet) 2022-03-10 09:14:52 [INFO]: -------> searching for topic Poliomyelitis from 1501691907734441989 until (last tweet) 2022-03-10 09:14:52 [INFO]: -------> searching for topic Tuberculosis from 1501691859910995975 until (last tweet) 2022-03-10 09:14:53 [INFO]: -------> searching for topic Anthrax from 1501691914663383041 until (last tweet) 2022-03-10 09:14:53 [INFO]: -------> searching for topic Crimean-Congo haemorrhagic fever from 1501676314721402888 until (last tweet) 2022-03-10 09:14:54 [INFO]: -------> searching for topic Botulism from 1501687047416860684 until (last tweet) 2022-03-10 09:14:54 [INFO]: -------> searching for topic Botulism from until (last tweet) Error in value[3L] : argument is not interpretable as logical Error in if (n1%%n2) warning("length(e1) not a multiple length(e2)"): argument is not interpretable as logical

Abdouzster commented 2 years ago

topics20220309.xlsx

forchard commented 2 years ago

Hi Abdel, There seems to be an issue on the format of the query on Botullism on the default file, please change the END of the second query from "- peso" to "-peso"

There may be other issues on the query. It seems that we did not updated this file on last release and the Twitter API v2 it is more strict than the v1.

@lauespinosa can you share the last version of the topics file so I can upload and prepare a new delivery package?

Thanks Abdelhamid! (in the mean time ypu will need to look on failing queries and fix them manually.

Best, Francisco

Abdouzster commented 2 years ago

Hi Francisco, Removing the space didn't work. I removed the second row for west nile virus and the second row for botulism. the loop worked but stopped again at STEC-VTEC. I think there is other cause for this. It might be a language or accent problems. It worked after I removed the second line in STEC-VTEC. Thanks, Abdelhamid