Twitter import: Add linked websites to indexing queue only, if yet not in index

opensemanticsearch / open-semantic-etl

Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database

https://opensemanticsearch.org/etl

GNU General Public License v3.0

254 stars 69 forks source link

Twitter import: Add linked websites to indexing queue only, if yet not in index #102

Open opensemanticsearch opened 4 years ago

opensemanticsearch commented 4 years ago

Add linked websites to indexing queue only, if yet not in index to spare resources because indexed only once even in linked in many tweets.

NetwarSystem commented 4 years ago

What is happening here? I have a lot of experience handling Twitter API data at scale.