create web scraping cheat sheet

[hannesdatta]

When scraping, you often need to use various types of code snippets over and over again.

The purpose of this issue is to create a "web scraping cheat sheet".

Desired features:

• contains all elements from this class
• making a connection w/ requests vs. selenium
• parsing information using beautifulsoup and selenium
• relate to challenges #2.1 - #2.4 in https://tiu.nu/scraping (at least)
• be styled in the layout of tilburg science hub cheat sheets, see here for the pdf, here for the pptx template

Deliverable:

• two files: PPTX and rendered PDF of the cheatsheet, committed to this folder
• ideally, a link from the dprep site (somewhere in the .md files) to this file (but I could link it, too)

Bonus point policy for https://odcm.hannesdatta.com:

• Students enrolled in [odcm.hannesdatta.com]() and [dprep.hannesdatta.com]() can obtain bonus points for working on this question (with a maximum of .5 on their final grade, see course syllabi).
• Successfully solving this issue within two weeks of posting it will account for .2 points, which includes up to two revision requests.
• If you would like to work on this issue, please post a brief description about how you plan to tackle it, in how much time, etc. Usually, the one who posts first gets to work on the issue. If more students request to work on this issue, the course coordinator will decide based on the proposed solutions.
• Solving this PR after the deadline halves the obtainable points.

[BastiaanVanHouten]

Personally, I would separate the different parts of collecting the web data into individual sections, such as creating a JSON file, the part with requesting to scrape (including the header) and setting a timer etc.

In each section, I would provide an explanation for each line of code, making it easier for others to understand the seperate lines.

Finally, I would combine all parts of the code, showing how they work together to create a complete web scraping script.

[hannesdatta]

Sounds good. You can work on the issue if you're up to it. Comment here if you accept!

[BastiaanVanHouten]

I accept!

[BastiaanVanHouten]

So far I have split up the code into 8 sections. These sections are

1. Importing the necessary packages.
2. Creating a function with a while loop.
3. Sending a Get request. 4.Extract information href_attribute.
4. Wrapping code into a json file. 6.Read and process the file using JSON.
5. Using time to pause the code.
6. Using panda to order the content of the file.

For these different sections I have written down line by line what the code does. Expect for section 6 that did not need explanation because it was repetitive. Moreover, with the different parts that needed additional explanation I have made a light gray text box. Finally, I have made a musterd colored text box that explains in 6 steps how to find the tags, attributes etc. on the website.

I have one major question: Would you like me to not explain the code line by line but give a brief description of what the code does? This will create more space. This space I could possibly use for the challanges section.

Please inform me on what you think is essential and currently left out or what is currently included but is not really necessary to have in the cheat sheet.

I have made two versions. Both are included in the powerpoint file. Webscraping cheatsheet.pptx

Best, Bas