search

kinshukdua / LiveActionMap

An attempt to map the areas with active conflict in Ukraine using twitter data and NLP.
https://www.live-action-map.com
MIT License
172 stars 15 forks source link

Dockerize #16

Closed laurin closed 2 years ago

laurin commented 2 years ago

Closes #7

laurin commented 2 years ago

We should add some documentation to the readme.

laurin commented 2 years ago

@kinshukdua The tweets.txt is not supposed to be persisted, right?

kinshukdua commented 2 years ago

@laurin it is supposed to persist for a short while. I'm thinking of letting the tweets accumulate for a bit (say 30 minutes or an hour) and then be deleted. I'm currently handling this with a cronjob but the feature being part of the docker container would help a lot.

laurin commented 2 years ago

Ok, the problem is that the tweets will be appended to the existing files, so most of there are a lot of duplicates. If the 100 tweets max returned by the Twitter API are not enough we could retrieve multiple pages or properly store the tweets with an id.

kinshukdua commented 2 years ago

You're right, we probably need to store the tweets with an id, to prevent duplicates. I think a database is an overkill for storing simple tweets. Instead lets just store the tweets with ids, text and link in a sorted order. Reading the file should barely take any time so we can just check the ids.

DomiiBunn commented 2 years ago

Try firebase, it's simple yet perfect for these solutions

laurin commented 2 years ago

I think we should take the discussion regarding the storage of tweets to another issue