[Google Jobs API] Slow response times and failed queries

[NateSkiles]

A customer reached out letting us know they were receiving 503 errors from the Google Jobs API.

Additionally, I'm unable to get a response in the Playground:

https://serpapi.com/playground?engine=google_jobs&q=Barista&no_cache=true

It looks like this started around 2024-07-23 11:58:10 UTC.

Intercom

[schaferyan]

Another user reported this:

Front

[schaferyan]

Thank you @NateSkiles, can you also add an intercom link for the customer who reported this?

[NateSkiles]

Another customer reported this:

Intercom

[sonika-serpapi]

Another customer reported this:

Intercom

[schaferyan]

Another customer reported this:

Intercom

[marm123]

Another customer reported this:

Intercom

[marm123]

Another user reported this:

Intercom

[martin-serpapi]

Another high-volume customer reported this:

Intercom

[stuartskelton]

Ditto for me too.

[martin-serpapi]

Another customer reported this:

Intercom

[martin-serpapi]

Another user reported this:

Intercom

[martin-serpapi]

Another high-volume customer reported this:

Intercom

[aciddjus]

Thank you, everyone, for your patience while we work on this issue. We apologize for the inconvenience.

We have deployed a partial fix, adding support for the new Google Jobs layout, and will continue to improve our parser in the coming days.

[alexbarron]

We received another report of this issue:

Intercom

It does appear success rates are increasing.

[stuartskelton]

I can confirm I am seeing an improvement, thanks for your help.

[schaferyan]

Another high volume customer report (from before the partial fix):

Front

[Freaky]

Success rates have been stable at 100% for a few hours now:

Average response times actually look better than before:

Known issues:

• No pagination support
• related_links are unavailable
• chips support isn't yet implemented

[schaferyan]

Another customer reported this:

Intercom

[NateSkiles]

Another customer reported this (specifically unavailable related_links):

Front

[martin-serpapi]

Another user reported this:

Intercom

[alexbarron]

Received a report specifically asking about related_links:

Intercom

[alexbarron]

Received a request specifically for pagination:

Intercom

[alexbarron]

Received a couple reports about apply_link missing in the encoded job_id:

Front Intercom

[alexbarron]

This could be a separate issue, but while helping a customer, I noticed their Google Jobs query didn't return results if it was all caps. The same query in all lower case letters returned results as expected. The query in all caps worked before the Google Jobs layout change which is why I suspect this may be related.

Playground upper case

Playground lower case

For comparison, I can run the query in all caps in actual Google Jobs and get results so it does seem related to us.

Google Jobs Query

Front

If this is a separate issue, please let me know and I'll create a new issue.

[Freaky]

Thanks @alexbarron, that's an interesting example! It can be reproduced directly on Google by specifying hl and gl:

Unsetting gl appears to work in this case: Playground (note you will need to unset the field manually as Playground JavaScript automatically resets Localization fields based on Geographic Location on load - we should perhaps revisit this behaviour)

Another problematic case I've noticed is Ireland, where setting gl or hl gives empty result pages: Playground. Unsetting both is the needed workaround in this case.

I need to investigate further to see what other combinations cause problems.

[sonika-serpapi]

Another customer reported this.

Intercom

[martin-serpapi]

Another user reported this:

Intercom

[martin-serpapi]

Another customer reported this:

Front

[Freaky]

A new update to the API has been deployed.

• Pagination support has been added. This requires client-side changes, as the numeric start parameter has been replaced with an opaque next_page_token available from serpapi_pagination.next_page_token. Searches with a start parameter will fail with an appropriate error message instead of returning empty results.
• location and via extraction has been improved and should now work across all locales and without the word via or its localised equivalent.
• extensions extraction has been improved.
• detected_extensions now includes additional extractions for things like dental_coverage and health_insurance.
• job_highlights are now extracted from JavaScript structures instead of being parsed out of description text.
• share_link has been added, providing a URL to link back to the job on Google.
• apply_options provides a list of job application links that were previously only available from the google_jobs_listing engine. This replaces the undocumented single link that was previously encoded in job_id.

Remaining work:

• chips support. There's a new filtering interface and the work here will likely make it into other engines as well.
• related_links. These require an additional subrequest for each job - they may end up in the google_jobs_listing engine.
• thumbnail is not yet properly extracted for paginated responses.
• Updated documentation for the new fields.
• Prettified HTML for paginated responses. The raw pages are usable, if a bit messy.

Thanks for everyone's patience.

[stuartskelton]

FYI, the playground needs updating https://serpapi.com/playground?engine=google_jobs&q=Golang&location=London%2C+England%2C+United+Kingdom&google_domain=google.co.uk&gl=uk&hl=en&start=0

get this error

{
"error":
"The `start` parameter has been removed, please use `next_page_token` for pagination."
}

[Freaky]

Thank you @stuartskelton - seems to be an issue with how we clear our Cloudflare cache after deployment. I manually cleared https://serpapi.com/playground?engine=google_jobs and now see the updated form.

I'll open an internal issue to improve this.

[sonika-serpapi]

One high volume customer was not sure how to use the next_page_token parameter. Can we perhaps add to the documentation on the Google Jobs API page to specify where users can get the token from to specify in the parameter? Intercom

[SoulEvill]

Thanks for updating the API.

I've just implemented the next_page_token for my application which usually pull 5 pages result.

The data looks odd - for example, thumbnail

after page 1 are all empty (it does have a url value but it is point to an empty gif)

Here is screenshot on playground on page 2 where you can see thumbnail are all empty.

[martin-serpapi]

Another customer reported the issue with the thumbnails in the response when paginating. I was able to reproduce it in the Playground:

Playground | Inspect

Intercom

[martin-serpapi]

A customer reported that there's an issue with the formatting of the results in the Playground when paginating. I was able to reproduce it:

Playground | Inspect

Front

[Freaky]

@SoulEvill @martin-serpapi Thanks for the reports - I forgot to mention thumbnails weren't being extracted from paginated pages correctly

I've opened the following new issues to track the remaining work now that the core issues of slow response times and failed queries is resolved:

• 1809 to track prettified HTML output for paginated pages on the Playground

• 1810 to track thumbnail extraction on paginated pages

• 1811 to track the new search filtering interface that has replaced chips

• 1812 to track the missing related_links section

• 1813 to track the problem of empty results with some gl and hl parameters

[ArmanNisch]

@Freaky Question: If the next_page_token token is the only way to paginate, how can we call, let's say, page 20 without having to first go through all the prior 19 pages?

With the start this was obviously no issue.

[martin-serpapi]

Thank you for the question @ArmanNisch . Unfortunately, there's no way to skip pages when using the next_page_token for pagination. Each response from the Google Jobs API contains the next_page_token value for the next page of results.

Please note that this is a limitation from Google, and not our API. I'm sorry for any inconvenience it may cause. If you have any follow-up questions, just let me know.

[hilmanski]

A customer requested that we should still scrape the original image link.

Sidenote, they mentioned, that the current SerpApi link didn't last very long. AFAIK, it should be last for a month.

Front

[hilmanski]

A customer complained about the update from start to next_page_token. Let's make this clear in the documentation, either informing about the update or clarifying where users can get the value from.

[aciddjus]

We have added a small update to our documentation regarding the next_page_token and start parameters.