URLs fail to open against a local query-server: dailymotion and yandex

[cclauss]

I'm submitting a ...

• [x] bug report
• [ ] feature request

Current behavior:

When running a query-server locally on Python2, the following URLs do not open:

open http://localhost:7001/api/v1/search/dailymotion?query=fossaisa&num=1
open http://localhost:7001/api/v1/search/yandex?query=fossaisa&num=1

Expected behavior:

These URLs would produce similar results to similar URLs for other scrapers.

Steps to reproduce:

#!/bin/sh
URL_ROOT="http://localhost:7001/api/v1/search/"
SEARCH_OPTIONS="?query=fossaisa&num=1"
# Uses the Mac OSX open command to launch a url into the default browser
# These urls open as expected
open "${URL_ROOT}ask${SEARCH_OPTIONS}"
open "${URL_ROOT}baidu${SEARCH_OPTIONS}"
open "${URL_ROOT}bing${SEARCH_OPTIONS}"
open "${URL_ROOT}duckduckgo${SEARCH_OPTIONS}"
open "${URL_ROOT}exalead${SEARCH_OPTIONS}"
open "${URL_ROOT}google${SEARCH_OPTIONS}"
open "${URL_ROOT}mojeek${SEARCH_OPTIONS}"
open "${URL_ROOT}parsijoo${SEARCH_OPTIONS}"
open "${URL_ROOT}quora${SEARCH_OPTIONS}"
open "${URL_ROOT}yahoo${SEARCH_OPTIONS}"
open "${URL_ROOT}youtube${SEARCH_OPTIONS}"

sleep 10  # allow time for the urls above to finish

# These urls do not open as expected
open "${URL_ROOT}dailymotion${SEARCH_OPTIONS}"
open "${URL_ROOT}yandex${SEARCH_OPTIONS}"

[AnshulMalik]

I tried all engines, yandex and dailymotion's results are not coming up, the process is running forever.

[cclauss]

@AnshulMalik You are correct. Those two scrapers seem to be failing. The others work on master.

[harsh-98]

@AnshulMalik yandex is failing because the site is redirecting to catpcha though dailymotion is working fine

[cclauss]

Thank you @harsh-98 I opened #387 so we can put some focus on resolving the captcha issue.

Would it be possible for you to remove dailymotion from https://github.com/fossasia/query-server/pull/385/files#diff-eff0198ff37b58d7351b62ca25b3f095R30 and tell us why it is failing? I know that it is failing but not why.

[harsh-98]

@cclauss on my system it isn't failing.