Bounty #1 : Linkedin scraper

[putxe]

🏆 Rewards: 2000 EXD 🏆

💡 Project Overview

We offer an exciting bounty for developers who can create a Python scraper that fetches relevant posts and comments from LinkedIn.com. The scraper should be able to log in to a LinkedIn account, generate a search URL based on provided keywords, and extract posts and comments posted within the last 120 seconds.

For the complete, detailed technical specifications and information about constraints, rules, interface, parameters, and wanted output format, please refer to our CONTRIBUTING.md file.

🔍 Task Description

Your role, as a Python developer, is to create a powerful scraper that accomplishes the following tasks:

• Visit linkedin.com and automatically log in with a provided account.
• Generate a search URL based on the given keyword in the input.
• The search URL format should be: https://www.linkedin.com/search/results/content/?keywords={the-keyword}&sortBy=%22date_posted%22
• Extract all posts and comments from the LinkedIn page that are not older than 120 seconds.
• You can find the scraper interface and further technical specifications in our CONTRIBUTING.md file.
• Additionally, we have provided an example of how to read the input parameters dictionary in this gist.

✨ Integration and Authentication

As a developer, it's crucial to integrate the scraper to maximize efficiency and user experience.

• Once the scraper visits linkedin.com and logs in successfully, it should be redirected to the homepage without requiring authentication again.
• You can achieve this by utilizing cookie storage to maintain the login session. (_recommended local file to create/use: linkedincookies.pkl)
• A good, reasonable amount of logs per scraping session is appreciable.

🔗 Submission

Let's make the web scraping experience on LinkedIn seamless and productive! 🚀🌟

To participate in this bounty, please create a Pull Request and link it to this issue. We look forward to reviewing your submission and wish you happy coding!

[whizsid]

I am attempting to this.

[whizsid]

@putxe @MathiasExorde Linkedin is asking for a confirmation code from an email after logged using an automated browser. So we have to pass IMAP configurations of the passed e-mail box. Do you have any concerns regarding this?

[MathiasExorde]

Our usual approach has been to inform users about the required action through logs and future push notifications using 'ntfy.sh'. It is the plan for Twitter & other scrapers. Then, we leave it to the users to take the necessary action, which may vary in nature (e.g., email reconfirmation like Twitter or completing a Captcha).

However, for the purposes of this experiment, we are considering the possibility of implementing IMAP and automating the process of reading emails to see if it is feasible. This would allow us to determine whether we should automate email processing or continue with our current method of notifying users about the required action. Feel free to proceed as you see fit! If we notice that this introduces a significant amount of complexity or additional work on your end, we will, of course, take that into consideration