Why omnivore fails to fetch content in some webpages

omnivore-app / omnivore

Omnivore is a complete, open source read-it-later solution for people who like reading.

https://omnivore.app

GNU Affero General Public License v3.0

11.85k stars 601 forks source link

Why omnivore fails to fetch content in some webpages #3925

Open natarslan opened 2 months ago

natarslan commented 2 months ago

I realise that once in a while omnivore cant fetch the article content both in Android app and in pc web browser. I was wondering if there are specific reasons behind this or is it just a random thing?

The current fail is in this page: https://climeworks.com/press-release/climeworks-switches-on-worlds-largest-direct-air-capture-plant-mammoth

jacksonh commented 2 months ago

Hey, that particular one seemed to fetch OK for me. Do you mind posting the error you see?

Typically when we can't fetch a URL its some sort of blocking enabled by the site. Often it is CloudFlare "bot protection" which can be quite aggressive.

natarslan commented 2 months ago

Hey,

I double checked the web page. It actually works in PC (web browser now). But it still doesnt work on Android (version 14). Here comes the screenshot.

What I also checked:

Made sure that NetGuard (vpn) is not blocking internet access to omnivore -> fetch fails
Totally disabled NetGuard in case it was somehow interfering with Omnivore -> fetch still fails

omnivore|100

jacksonh commented 2 months ago

When you say works on PC, you saved it with the browser extension there?

natarslan commented 2 months ago

Yes in desktop browser it works. But in android app the page didnt work (see the screenshot).

jacksonh commented 2 months ago

Yes in desktop browser it works. But in android app the page didnt work (see the screenshot).

but you only saved the page once? Or you saved on Android also?

natarslan commented 2 months ago

I tried saving the page

first on android (failed to fetch - so i removed the item)
and then on my desktop browser (failed first, then worked).
Lastly, I deleted the page from browser and tried it again on Android (failed).

I can confirm that it now works on Android also.

The question is, is there anything I can do to "help" omnivore fetch the pages (can disabling adblockers etc help)?