Closed mohammad-fahs closed 2 months ago
@mohammad-fahs Website: OLX https://www.dubizzle.com.lb/
OLX is a popular online marketplace where users can buy and sell a wide range of products and services. It operates globally and allows individuals to list items such as electronics, vehicles, real estate, and more. Users can search for local listings, negotiate prices, and connect directly with sellers or buyers. OLX aims to simplify transactions by providing a platform for classified ads that is easy to use and accessible via both web and mobile applications.
Data Scraped: Mobile phones products
Console Screenshot for Scraped Data
Java Code
package com.douaaObeid.web_Scraper2;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.stereotype.Component;
import java.net.URL;
import java.util.List;
import java.util.Scanner;
@Component
public class OLXScrapperRunner implements CommandLineRunner {
OLXScraper scrapingService = new OLXScraper();
public static void main(String[] args) {
SpringApplication.run(WebScraper2Application.class, args);
}
@Override
public void run(String... args) throws Exception {
Scanner scanner = new Scanner(System.in);
while (true) {
System.out.println("Enter link to scrape or 'exist' to stop:");
String link = scanner.nextLine();
if (link.equalsIgnoreCase("exist")) {
break;
}
try {
System.out.println("Scraping link: " + link);
List<OLX> OLXList = scrapingService.scrapeOLX(link);
System.out.println("Scraping complete. Found " + OLXList.size() + " items.");
for (OLX olx : OLXList) {
System.out.println(olx.toString());
}
} catch (Exception e) {
System.err.println("An error occurred during scraping: " + e.getMessage());
e.printStackTrace();
}
}
}
}
Task 2 :
Objective:
In this task, you will initialize a new Spring Boot project, add the JSoup dependency, and write a simple Java program to scrape data from a website of your choice. The goal is to get hands-on experience with setting up a Spring Boot project, using an external library (JSoup), and applying web scraping techniques.
Instructions:
com.yourname
web-scraper
Web Scraper
com.yourname.webscraper
Spring Web
dependency (to allow adding more features later).pom.xml
file in the root directory of your project.<dependencies>
tag:pom.xml
file and allow IntelliJ to update the Maven project to download the JSoup library.com.yourname.webscraper
package that implements theCommandLineRunner
interface.run
method, use JSoup to connect to the website you chose and scrape the data.Submit Your Work:
CommandLineRunner
.Resources that can help: