linkedtales / scrapedin

LinkedIn Scraper (currently working 2020)
Apache License 2.0
597 stars 174 forks source link

Only first accomplishment returned #102

Closed lhl2617 closed 4 years ago

lhl2617 commented 4 years ago

Hi,

When scraping for accomplishment sections for detailed accomplishments (added in #89, by @EmaSuriano), only the first item is returned. Was wondering if you are seeing the same bug.

Thanks

lhl2617 commented 4 years ago

I just need this for an ad-hoc projects scrape, so this is code that works for me

const scrapProjectsPanel = async (page) => {
    const openingButton = await page.$(
        `.pv-accomplishments-block.projects button`,
    );

    if (openingButton) {
        await openingButton.click();
        await new Promise((resolve) => {
            setTimeout(() => {
                resolve();
            }, 500);
        });

        let showMoreButton;

        while (showMoreButton = await page.$(`.pv-profile-section__see-more-inline`)) {
            await showMoreButton.click();
            await new Promise((resolve) => {
                setTimeout(() => {
                    resolve();
                }, 250);
            });
        }

        const selectors = await page.$$(`.pv-accomplishment-entity`)

        const got = await Promise.all(selectors.map((selector) =>
            Object.keys(template.projects.fields)
                .reduce(scrapSelectorFields(selector, template.projects), Promise.resolve({}))
        ))

        return got;
    }
}

I will PR when I have time - but if others have time-critical uses for this, please go ahead and PR for me

cmtr commented 4 years ago

Proposal for this issue in PR #118