Unable to crawl lbry.tv / odysee.com

yacy / yacy_search_server

Distributed Peer-to-Peer Web Search Engine and Intranet Search Appliance

http://yacy.net

Other

3.36k stars 424 forks source link

Open kodxana opened 3 years ago

kodxana commented 3 years ago

I'm trying to crawl links from this websites but only what I could crawl is main page and favicons. Any ideas what might be issue?

frankenstein91 commented 3 years ago

These pages interactively load their content and do not deliver it directly.

kodxana commented 3 years ago

Hmm so what do you sugest?

frankenstein91 commented 3 years ago

In my view, there are two very good solutions. Either you use a sitemap or an RSS feed. The solution with RSS is already suggested for Peertube and explained in the ticket: https://github.com/yacy/yacy_search_server/issues/399#issuecomment-771099797