🚨 There's a new, more advanced version of this bot that you can checkout for free 🚨

Visit notifast.me to learn more

webscraper-bot

Discord web scraping bot used to scrape websites with dynamic content and send notifications when there is a new item.

Instructions

1.Adding bot to your server

This version development has been paused but you can try out the new version or self host it (e.g. using Render or Railway),

2.Adding new scraper job

Use /create-job command and fill in the form:

• name - name of the job
• url - url that the bot will scrape
• selector - selector for an a tag with link to element (examples below)
• interval - interval of the job in minutes (min 1 minute)
• active? - default true, if the job should be active right away (you can always enable/disable job with [enable/disable]-job commands)
• channel? - defualt channel where the command was run which channel should be messaged when the new item appear
• clean? - default true, if the query params should be ignored (essential for some sites like Ebay)

After that you can do basic CRUD opeartions on jobs with commands like /list-jobs /update-job /delete-job...

3.Waiting...

After that you simply wait for the job run and it will send message when new items are found:

You can also run the job manually with /run-job. Note that the first run will get all the elements that will be on the webiste.

How to get selector

The selector is a querySelectorAll string that the bot uses to get unique a tags linking to items you want to scrape. To verify if your selector is working you can run document.querySelectorAll(<your selector>) in browser console and check if the function returns items you want to scrape.

Examples of selectors

• Ebay: .srp-river-results .s-item__image a
• Otodom: [data-cy=listing-item-link]
• Olx: [data-cy=l-card] > a

Feel free to contribute, there are a lot of things to improve :)