search

aigents / aigents-java

Aigents Java Core Platform
MIT License
30 stars 12 forks source link

Refactor Siter so extensions can override it and add custom plugins/adapters for online/social media processing #20

Closed akolonin closed 4 years ago

akolonin commented 4 years ago

What is done so far:

  1. Imager is moved to separate class and renamed to ContentLocator
  2. Siter has constructors changed and init function changed with respective argument revamping

What will come next:

  1. Siter will be split in Siter and WebCrawler.
  2. Siter will hold overall crawling framework and be configurable at Body level so one can create/extend/override it
  3. WebCrawler will do actual web crawling and implement Crawler interface (this interface will be also implemented by Redditer, Twitterer and Discourser), so you could extend override the WebCrawler itself or add custom Crawlers
  4. The current readChannel method of Redditer, Twitterer and Discourser will be moved to Crawled interface renamed to "crawl"
  5. RSSer will be created and implementing Crawled interface as an example of custom crawler (do can do Arxiv and PsyArxiv plugins)
akolonin commented 4 years ago

https://github.com/aigents/aigents-java/commit/8e2e4092e34b89065bce2f6b276f4ea85f696932

akolonin commented 4 years ago

Completed with https://github.com/aigents/aigents-java/commit/3cfe6dab0660474b62157860b8135ca800d618bd