[YouTube] Could not fetch long search terms suggestions (>= 100 characters, including spaces)

opusforlife2 commented 3 years ago

Checklist

[x] I am using the latest version - 0.25.2
[x] I checked, but didn't find any duplicates (open OR closed) of this issue in the repo.
[x] I have read the contribution guidelines given at https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md.
[x] This issue contains only one bug. I will open one issue for every bug report I want to file.

Steps to reproduce the bug

Paste a 99 character long body of text into the search bar.
See that it works fine.
Paste a 100 character long body of text into the search bar.
Wait for a second or two for the suggestion code to work.
See the error snackbar pop up.

This only happens for Youtube, and not the other services.

I haven't investigated this much. But @ShareASmile did, and found out the 100 character threshold!

Actual behaviour

Snackbar error.

Expected behavior

No snackbar error.

Logs

Exception

User Action: get suggestions
Request:
Content Country: CH
Content Language: en-IN
App Language: en_IN
Service: YouTube
Version: 0.20.9
OS: Linux Android 10 - 29
Crash log

org.schabi.newpipe.extractor.exceptions.ParsingException: Could not parse json response
    at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSuggestionExtractor.suggestionList(YoutubeSuggestionExtractor.java:72)
    at org.schabi.newpipe.util.ExtractorHelper.lambda$suggestionsFor$2(ExtractorHelper.java:118)
    at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$kgV3vMftlbQH0zdv8hszrHLyMLU.call(Unknown Source:4)
    at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43)
    at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
    at io.reactivex.rxjava3.internal.operators.single.SingleOnErrorReturn.subscribeActual(SingleOnErrorReturn.java:38)
    at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
    at io.reactivex.rxjava3.internal.operators.single.SingleToObservable.subscribeActual(SingleToObservable.java:35)
    at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13102)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableMap.subscribeActual(ObservableMap.java:33)
    at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13102)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableZip$ZipCoordinator.subscribe(ObservableZip.java:110)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableZip.subscribeActual(ObservableZip.java:72)
    at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13102)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableMaterialize.subscribeActual(ObservableMaterialize.java:28)
    at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13102)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableSwitchMap$SwitchMapObserver.onNext(ObservableSwitchMap.java:127)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableFilter$FilterObserver.onNext(ObservableFilter.java:52)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableConcatMap$ConcatMapDelayErrorObserver$DelayErrorInnerObserver.onNext(ObservableConcatMap.java:498)
    at io.reactivex.rxjava3.observers.SerializedObserver.onNext(SerializedObserver.java:113)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableDebounceTimed$DebounceTimedObserver.emit(ObservableDebounceTimed.java:143)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableDebounceTimed$DebounceEmitter.run(ObservableDebounceTimed.java:168)
    at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
    at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:919)
Caused by: com.grack.nanojson.JsonParserException: Unexpected token 'OCTYPE' on line 1, char 1
    at com.grack.nanojson.JsonTokener.createParseException(Unknown Source:44)
    at com.grack.nanojson.JsonTokener.createHelpfulException(Unknown Source:119)
    at com.grack.nanojson.JsonTokener.advanceToToken(Unknown Source:95)
    at com.grack.nanojson.JsonParser.advanceToken(Unknown Source:12)
    at com.grack.nanojson.JsonParser.parse(Unknown Source:1)
    at com.grack.nanojson.JsonParser$JsonParserContext.from(Unknown Source:19)
    at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSuggestionExtractor.suggestionList(YoutubeSuggestionExtractor.java:62)
    ... 28 more

ShareASmile commented 10 months ago

I think this exception is reproducible with more than 100 number of characters (including spaces) in search query in NewPipe for Youtube service.

Try searching in the app any text which exceeds 100 characters (including spaces) like following text:-

Chand Se Mukhatib rare song Kumar Sanu Remastered HQ high quality dolby atmos surround sound flac aud

& extractor returns exception as in my following comment in form of a snackbar error.

Now try the search one more time with one less character e.g. remove the last alphabet "d" from search field & no such exception takes place.

Finally you can Calmly count searched characters (including spaces) to determine exact figure above which extractor fails to get suggestions or whatever.

opusforlife2 commented 10 months ago

Nice, that's exactly right!

ShareASmile commented 10 months ago

Latest Error Report to above reported bug to help developers debug the issue..

Exception

User Action: get suggestions
Request: chand se mukhatib rare song Kumar Sanu Remastered HQ high quality dolby atmos surround sound flac audio
Content Country:
Content Language: en-GB
App Language: en
Service: YouTube
Version: 0.25.2
OS: Linux Android 13 - 33
Crash log

org.schabi.newpipe.extractor.exceptions.ExtractionException: Invalid response type (got "text/html; charset=UTF-8", excepted a JSON response) (response code 400)
    at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSuggestionExtractor.suggestionList(YoutubeSuggestionExtractor.java:71)
    at org.schabi.newpipe.util.ExtractorHelper.lambda$suggestionsFor$2(ExtractorHelper.java:110)
    at org.schabi.newpipe.util.ExtractorHelper.$r8$lambda$kgV3vMftlbQH0zdv8hszrHLyMLU(ExtractorHelper.java:0)
    at org.schabi.newpipe.util.ExtractorHelper$$ExternalSyntheticLambda14.call(R8$$SyntheticClass:0)
    at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43)
    at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
    at io.reactivex.rxjava3.internal.operators.single.SingleToObservable.subscribeActual(SingleToObservable.java:36)
    at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13262)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableMap.subscribeActual(ObservableMap.java:33)
    at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13262)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableZip$ZipCoordinator.subscribe(ObservableZip.java:110)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableZip.subscribeActual(ObservableZip.java:72)
    at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13262)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableMaterialize.subscribeActual(ObservableMaterialize.java:28)
    at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13262)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableSwitchMap$SwitchMapObserver.onNext(ObservableSwitchMap.java:128)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableConcatMap$ConcatMapDelayErrorObserver$DelayErrorInnerObserver.onNext(ObservableConcatMap.java:500)
    at io.reactivex.rxjava3.observers.SerializedObserver.onNext(SerializedObserver.java:114)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableDebounceTimed$DebounceTimedObserver.emit(ObservableDebounceTimed.java:156)
    at io.reactivex.rxjava3.internal.operators.observable.ObservableDebounceTimed$DebounceEmitter.run(ObservableDebounceTimed.java:181)
    at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
    at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
    at java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
    at java.lang.Thread.run(Thread.java:1012)

AudricV commented 10 months ago

This behavior happen even on the official clients (I only tested the web ones), YouTube returns Google's 400 HTTP HTML responses when a suggestion has more than 100 characters saying Your client has issued a malformed or illegal request. That’s all we know.

Workarounds would be to return an empty search suggestion list when a search suggestion request has more than 100 characters (I prefer this solution) or when we get a 400 HTTP response with a HTML Content-Type HTTP header.

For reference, Piped backends block requests to search suggestions with more than 100 characters, returning an error response saying that query is too long.

TeamNewPipe / NewPipe