-
It would be nice to add to Frontera an optional crawling strategy for topical crawling. It could take dictionary of words describing some topic as input and crawl from seed urls searching for document…
-
**Description**
We found some discrepancies in the reports from both crawlers. We need to investigate whether we can achieve the same results with SEMrush or if it makes sense to continue running Co…
-
- [ ] Talk about the complexity of the algorithm running tim used.
- [x] Web characterization **[6]**
- [x] Methods for sampling, Web dynamics, Estimating freshness and age, Characterization of We…
-
In a 7/9 discussion with Anna, she requested that we develop an approach to design and build site wide search functionality into the NCIDS documentation site. https://designsystem-dev.cancer.gov/devel…
-
**Is your feature request related to a problem? Please describe.**
The Ultimate aim of the project is to detect illicit websites. As of now the algorithm uses graph knowledge to target suspicious lin…
-
I'm using docker-compose on windows 10
when I run "docker-compose up " everything works fine
elasticsearch works fine, DDT tool works fine but the crawler won't work, when I use deep crawling and us…
-
Whereas the code for Yserbius and Twinion can only run Yserbius style games (and even more specifically, only Yserbius and Twinion), Cawdor runs on a new client/server dungeon crawler engine that is s…
-
### Pitch
In August, GPTbot block was merged into the code https://github.com/mastodon/mastodon/pull/26396. Now, Google has a robots.txt policy for Bard and future Google AI models with user agent Go…
-
I would love to create a gpt out of a github repo. Can you please add this?
K thx bai
-
```
from trafilatura.spider import focused_crawler
crawl_start_url = 'https://cloud.google.com/docs'
to_visit, known_links = focused_crawler(homepage=crawl_start_url, max_seen_urls=1000, max_known_…