search

thp / urlwatch

Watch (parts of) webpages and get notified when something changes via e-mail, on your phone or via other means. Highly configurable.
https://thp.io/2008/urlwatch/
Other
2.81k stars 352 forks source link

Wait for selector? #763

Closed yuis-ice closed 4 months ago

yuis-ice commented 1 year ago

As #548, some websites don't load properly (one serious example would be jp.mercari.com/) with just a wait_until. I think it'd be better to enable users to use wait_for to wait for a selector, or enables users to inject/eval a custom JavaScript script within the filter chain so it allows users much more capability more than the waiting issue. Thanks.

thp commented 1 year ago

With urlwatch 2.28, the docs say: https://urlwatch.readthedocs.io/en/latest/jobs.html#browser

wait_until: Either load, domcontentloaded, networkidle, or commit (see Advanced Topics)

See also: Configuring how long browser jobs wait for pages to load

Related docs: https://dev.to/checkly/avoiding-hard-waits-in-playwright-and-puppeteer-272

Jamstah commented 5 months ago

Have put a very naive implementation in as #810