spinlud / linkedin-jobs-scraper

147 stars 40 forks source link

#Add options to events.scraper.data #42

Open HamedOsama opened 1 year ago

HamedOsama commented 1 year ago

This pull request fixies issue #41

First, I have added options property to IData interface as optional property in (events.ts)

export interface IData {
    query: string;
    location: string;
    jobId: string;
    jobIndex: number; // Job index during search, only useful for debug
    link: string;
    applyLink?: string;
    title: string;
    company: string;
    companyLink?: string;
    companyImgLink?: string;
    place: string;
    date: string;
    description: string;
    descriptionHTML: string;
    insights: string[];
    options?: IQueryOptions; // new update
}

Then I add the options to emitter in (AnonymousStrategy.ts)

this.scraper.emit(events.scraper.data, {
    query: query.query || "",
    location: location,
    jobId: jobId!,
    jobIndex: jobIndex,
    link: jobLink!,
    ...jobApplyLink && { applyLink: jobApplyLink },
    title: jobTitle!,
    company: jobCompany!,
    place: jobPlace!,
    description: jobDescription! as string,
    descriptionHTML: jobDescriptionHTML! as string,
    date: jobDate!,
    insights: [],
    options: query.options // new update
});

Also, I update the filters example and log the filter after data scrapped.

import {
    LinkedinScraper,
    relevanceFilter,
    timeFilter,
    typeFilter,
    experienceLevelFilter,
    events,
} from "..";

(async () => {
    // Each scraper instance is associated with one browser.
    // Concurrent queries will run on different pages within the same browser instance.
    const scraper = new LinkedinScraper({
        headless: true,
        slowMo: 10,
    });

    // Add listeners for scraper events
    scraper.on(events.scraper.data, (data) => {
        console.log(
            `Title: ${data.title} \n
            Company: ${data.company} \n
            filters ${data?.options?.filters ? JSON.stringify(data.options.filters) : ""}`
        );
    });

    await scraper.run({
        query: "",
        options: {
            filters: {
                // See documentation on how find this url
                companyJobsUrl: "https://www.linkedin.com/jobs/search/?f_C=1441%2C17876832%2C791962%2C2374003%2C18950635%2C16140%2C10440912&geoId=92000000&lipi=urn%3Ali%3Apage%3Acompanies_company_jobs_jobs%3BcbFm1gYoRwy%2FxVRQWbGyKw%3D%3D&licu=urn%3Ali%3Acontrol%3Ad_flagship3_company-see_all_jobs",
                relevance: relevanceFilter.RELEVANT,
                time: timeFilter.MONTH,
                // type: typeFilter.FULL_TIME,
                experience: experienceLevelFilter.MID_SENIOR,
            }
        }
    }, {
        optimize: true,
        locations: ["United States"],
        limit: 10,
    });

    // Close browser
    await scraper.close();
})();
HamedOsama commented 1 year ago

@spinlud can you check this?