Youtube cannot be play or extracted

[schlector-code]

Since Yesterday YouTube cannot play, can you take a look at it with priority?

[nhCoder]

I will look into it tonight

[craftbox2245]

Error : While getting YouTube Data: Attempt to invoke virtual method 'int java.lang.String.lenght()' on a null object reference YouTube can not extracted Please Help us as soon as possible @nhCoder

[nhCoder]

its works now IDK but it works please check and tell me again

[schlector-code]

its works now IDK but it works please check and tell me again No, it still not working, failing with number of videos like Vevo videos.

[hhyeok1026]

I confirmed that there was a problem on November 23rd.

The response value from YouTube has changed. However, there were times when a specific video of a specific device appeared as an existing response. So it could seem like it was going well. When it came out with the existing response structure, the new changed values were mixed.

So I modified it to work with YouTube's new changes and made sure it works on all devices.

======================================================= I want to do a pull request, but I have to do the project I am doing now, so I will leave the source here. ================================================== =====

1.PlayerResponse.java The playerJs value in Response.java has been moved to PlayerResponse.

private String playerJs; //add

//add
public void setPlayerJs(String playerJs){
this.playerJs = playerJs;
}

public String getPlayerJs(){
if( playerJs.startsWith("http") && playerJs.contains("youtube.com") ){
return playerJs.replace("\\", "");
}
else{
return "https://www.youtube.com" + playerJs.replace("\\", "");
}
}

=============================================

2. YoutubeStreamExtractor.java

2-1) Change member variable of YoutubeStreamExtractor class.

//String regexPlayerJson="(?<=ytplayer.config\\s=).*?((\\}(\n|)\\}(\n|))|(\\}))(?= ;)"; //delete
 String regexPlayerJson="(?<=ytInitialPlayerResponse\\s=).*?(\\}\\]\\})(?=;)"; //add  //The regular expression I created may be unstable..

//private Response response; //delete
PlayerResponse playerResponse; // From local variable to member variable

2-2) in doInBackground()

 playerResponse = parseJson(jsonBody); //Receive as a member variable
            //response.setPlayerJs(RegexUtils.matchGroup(PlayerBaseRegex, body)); //delete playerResponse.setPlayerJs(RegexUtils.matchGroup(PlayerBaseRegex, body)); //add

2-3) Change parseJson() to the source below.

private PlayerResponse parseJson(String body) throws Exception {
JsonParser parser=new JsonParser();
return new GsonBuilder().serializeNulls().create().fromJson(parser.parse(body), PlayerResponse.class);
}

2-4)in parseUrls()

//decodedSig = CipherManager.dechiperSig(URLDecoder.decode(partCipher.replace("s=", "")), response.getPlayerJs()); //delete
decodedSig = CipherManager.dechiperSig(URLDecoder.decode(url_part.replace("s=", "")), playerResponse.getPlayerJs()); //add

[NiazSagor]

@hhyeok1026 hello. thank you for your updated code and I have made changes on the code according to your code. But the error is still there. can you suggest anything?

[nhCoder]

@hhyeok1026 hello. thank you for your updated code and I have made changes on the code according to your code. But the error is still there. can you suggest anything?

hey no need to modify anything the original code is working

[NiazSagor]

@nhCoder sorry brother, it's still not working. strange though, it was working just fine since I have implemented it. I did not change anything but the error just landed out of no where. I do not know what to do. I have a project that heavily depends on this.

[nhCoder]

just check the apk for test and tell me again

[schlector-code]

@nhCoder Please check with other YouTube video like VEVO videos, it is more likely to fell in the problem. and Test with different device.

[nhCoder]

@nhCoder Please check with other YouTube video like VEVO videos, it is more likely to fell in the problem. and Test with different device.

please send me the links

[hhyeok1026]

@nhCoder

In my case

on a galaxy A9 device https://www.youtube.com/watch?v=gdZLi9oWNZg This video didn't work, But some of the videos work just fine.

however, All of the videos didn't work on other test devices.

[hhyeok1026]

@NiazSagor

I have attached my full source code. Please import additional things you need.

If this doesn't work, I'll show you how I debugged it.

=============================================================== public class YoutubeStreamExtractor extends AsyncTask<String,Void,Void> { Map<String,String> Headers=new HashMap<>(); List adaptiveMedia=new ArrayList<>(); List muxedMedia=new ArrayList<>(); List subtitle=new ArrayList<>(); String regexUrl=("(?<=url=)."); String regexYtshortLink="(http|https)://(www\.|)youtu.be/."; String regexPageLink = ("(http|https)://(www\.|m.|)youtube\.com/watch\?v=(.+?)( |\z|&)"); String regexFindReason="(?<=(class=\"message\">)).*?(?=<)";

//String regexPlayerJson="(?<=ytplayer.config\\s=).*?((\\}(\n|)\\}(\n|))|(\\}))(?=;)";  //before  //ytplayer.config가 일치하고,  // 2020.11.02
String regexPlayerJson="(?<=ytInitialPlayerResponse\\s=).*?(\\}\\]\\})(?=;)";  //before  //ytplayer.config가 일치하고,  // 2020.11.02

//String regexPlayerJson="(?<=ytplayer.config\\s=).*?((\\}(\n|)\\}(\n|))|(\\}))(?=;ytplayer)"; //after

//String regexPlayerJson="(? <= ytplayer.config \ s =).? ((} (\ n |)} (\ n |)) | (})) (? =; ytplayer)";
//String regexPlayerJson="(? <= ytplayer.config \ s =).? ((\} (\ n |) \} (\ n |)) | (\})) (? =; ytplayer)";

String PlayerBaseRegex = "(?<=PLAYER_JS_URL\":\").*?(?=\")";

ExtractorListner listener;
private ExtractorException Ex;
List<String> reasonUnavialable=Arrays.asList(new String[]{"This video is unavailable on this device.","Content Warning","who has blocked it on copyright grounds."});
Handler han=new Handler(Looper.getMainLooper());
//private Response response;
private YoutubeMeta ytmeta;

PlayerResponse playerResponse;

public YoutubeStreamExtractor(ExtractorListner EL) {
    this.listener = EL;
    Headers.put("Accept-Language", "en");
}

public YoutubeStreamExtractor setHeaders(Map<String, String> headers) {
    Headers = headers;
    return this;
}

public YoutubeStreamExtractor useDefaultLogin() {
    Headers.put("Cookie", Utils.loginCookie);
    return setHeaders(Headers); 
}

public Map<String, String> getHeaders() {
    return Headers;
}

public void Extract(String VideoId) {
    this.execute(VideoId);
}

@Override
protected void onPostExecute(Void result) {

    //LogUtils.log("YoutubeStreamExtractor - onPostExecute() 들어옴.");

    if (Ex != null) {
        listener.onExtractionGoesWrong(Ex);
    } else {
        listener.onExtractionDone(adaptiveMedia, muxedMedia,subtitle, ytmeta);
    }
}

@Override
protected void onPreExecute() {

    //LogUtils.log("YoutubeStreamExtractor - onPreExecute() 들어옴.");

    Ex = null;
    adaptiveMedia.clear();
    muxedMedia.clear();

}

@Override
protected void onCancelled() {

    //LogUtils.log("YoutubeStreamExtractor - onCancelled() 들어옴.");

    if (Ex != null) {
        listener.onExtractionGoesWrong(Ex);
    }   
}

@Override
protected Void doInBackground(String[] ids) {

    //LogUtils.log("YoutubeStreamExtractor - doInBackground() 들어옴.");
    String Videoid= Utils.extractVideoID(ids[0]);
    String jsonBody = null;

    try {

        String body = HTTPUtility.downloadPageSource("https://www.youtube.com/watch?v=" + Videoid + "&has_verified=1&bpctr=9999999999", Headers);
        //LogUtils.log("body = " + body);

        jsonBody = parsePlayerConfig(body);

        //PlayerResponse playerResponse=parseJson(jsonBody);
        playerResponse = parseJson(jsonBody);
        //response.setPlayerJs(RegexUtils.matchGroup(PlayerBaseRegex, body));
        playerResponse.setPlayerJs(RegexUtils.matchGroup(PlayerBaseRegex, body));

        ytmeta = playerResponse.getVideoDetails();
        subtitle=playerResponse.getCaptions() !=null ? playerResponse .getCaptions().getPlayerCaptionsTracklistRenderer().getCaptionTracks(): null;

        //Utils.copyToBoard(jsonBody);

        if (playerResponse.getVideoDetails().getisLive()) {
            parseLiveUrls(playerResponse.getStreamingData());
        } else {
            StreamingData sd=playerResponse.getStreamingData();
            //LogUtils.log("sizea= " + sd.getAdaptiveFormats().length);
            //LogUtils.log("sizem= " + sd.getFormats().length);

            adaptiveMedia = parseUrls(sd.getAdaptiveFormats());
            muxedMedia =    parseUrls(sd.getFormats());
            //LogUtils.log("sizeXa= " + adaptiveMedia.size());
            //LogUtils.log("sizeXm= " + muxedMedia.size());
        }
    }
    catch (Exception e) {
        //LogUtils.log(Arrays.toString(e.getStackTrace()));
        //Ex = new ExtractorException("Error While getting Youtube Data:" + e.getMessage());

        StringWriter errors = new StringWriter();
        e.printStackTrace(new PrintWriter(errors));

        Ex = new ExtractorException("Error While getting Youtube Data:" + errors.toString());
        this.cancel(true);
    }
    return null;
}

/*this function creates Json models using Gson*/
private PlayerResponse parseJson(String body) throws Exception {
    JsonParser parser=new JsonParser();

    //LogUtils.log("body : " + body);

    //LogUtils.log("출력한다!!!!!!!!!!!!");
    //LogUtils.longlog(body);

    //response = new GsonBuilder().serializeNulls().create().fromJson(parser.parse(body), Response.class);
    //Log.v("또안되","response : " + response);
    //Log.v("또안되","response.getArgs() : " + response.getArgs());
    //Log.v("또안되","response.getArgs().getPlayerResponse() : " + response.getArgs().getPlayerResponse());

    //return new GsonBuilder().serializeNulls().create().fromJson(response.getArgs().getPlayerResponse(), PlayerResponse.class);
    return new GsonBuilder().serializeNulls().create().fromJson(parser.parse(body), PlayerResponse.class);
}

/*This function is used to check if webpage contain steam data and then gets the Json part of from the page using regex*/
private String parsePlayerConfig(String body) throws ExtractorException {

    if (Utils.isListContain(reasonUnavialable, RegexUtils.matchGroup(regexFindReason, body))) {
        throw new ExtractorException(RegexUtils.matchGroup(regexFindReason, body));
    }

    if (body.contains("ytplayer.config")) {
        return RegexUtils.matchGroup(regexPlayerJson, body);
    } else {
        throw new ExtractorException("This Video is unavialable");
    }
}

/*independent function Used to parse urls for adaptive & muxed stream with cipher protection*/

private List<YTMedia> parseUrls(YTMedia[] rawMedia) {
    List<YTMedia> links=new ArrayList<>();
    try {
        for (int x=0; x < rawMedia.length; x++) {

            YTMedia media=rawMedia[x];
            //LogUtils.log(media.getCipher() != null ? media.getCipher(): "null cip");
            LogUtils.log(media.getSignatureCipher() != null ? media.getSignatureCipher(): "null cip");

            if (media.useCipher()) {
                String tempUrl = "";
                String decodedSig = "";

                //for (String partCipher:media.getCipher().split("&")) {
                for (String partCipher : media.getSignatureCipher().split("&")) {

                    if (partCipher.startsWith("s=")) {

                        try{
                            //decodedSig = CipherManager.dechiperSig(URLDecoder.decode(partCipher.replace("s=", "")), response.getAssets().getJs());
                            //decodedSig = CipherManager.dechiperSig(URLDecoder.decode(partCipher.replace("s=", "")), null);
                            //decodedSig = CipherManager.dechiperSig(URLDecoder.decode(partCipher.replace("s=", "")), response.getPlayerJs());
                            decodedSig = CipherManager.dechiperSig(URLDecoder.decode(partCipher.replace("s=", "")), playerResponse.getPlayerJs());
                        }catch(Exception e){
                            e.printStackTrace();
                        }
                    }

                    if (partCipher.startsWith("url=")) {
                        tempUrl = URLDecoder.decode(partCipher.replace("url=", ""));

                        for (String url_part:tempUrl.split("&")) {
                            if (url_part.startsWith("s=")) {
                                //decodedSig = CipherManager.dechiperSig(URLDecoder.decode(url_part.replace("s=", "")), response.getAssets().getJs());
                                //decodedSig = CipherManager.dechiperSig(URLDecoder.decode(url_part.replace("s=", "")), null);
                                //decodedSig = CipherManager.dechiperSig(URLDecoder.decode(url_part.replace("s=", "")), response.getPlayerJs());
                                decodedSig = CipherManager.dechiperSig(URLDecoder.decode(url_part.replace("s=", "")), playerResponse.getPlayerJs());
                            }
                        }
                    }
                }

                String  FinalUrl= tempUrl + "&sig=" + decodedSig;

                media.setUrl(FinalUrl);
                links.add(media);
            } else {
                links.add(media);
            }
        }
    }
    catch (Exception e) {
        e.printStackTrace();
        Ex = new ExtractorException(e.getMessage());
        this.cancel(true);
    }
    return links;
}

/*This funtion parse live youtube videos links from streaming data  */
private void parseLiveUrls(StreamingData streamData) throws Exception {
    if (streamData.getHlsManifestUrl() == null) {
        throw new ExtractorException("No link for hls video");
    }
    String hlsPageSource= HTTPUtility.downloadPageSource(streamData.getHlsManifestUrl());
    String regexhlsLinks="(#EXT-X-STREAM-INF).*?(index.m3u8)";
    List<String> rawData= RegexUtils.getAllMatches(regexhlsLinks, hlsPageSource);
    for (String data:rawData) {
        YTMedia media=new YTMedia();
        String[] info_list= RegexUtils.matchGroup("(#).*?(?=https)", data).split(",");
        String live_url= RegexUtils.matchGroup("(https:).*?(index.m3u8)", data);
        media.setUrl(live_url);
        for (String info:info_list) {
            if (info.startsWith("BANDWIDTH")) {
                media.setBitrate(Integer.valueOf(info.replace("BANDWIDTH=", "")));
            }
            if (info.startsWith("CODECS")) {
                media.setMimeType((info.replace("CODECS=", "").replace("\"", "")));
            }
            if (info.startsWith("FRAME-RATE")) {
                media.setFps(Integer.valueOf((info.replace("FRAME-RATE=", ""))));
            }
            if (info.startsWith("RESOLUTION")) {
                String[] RESOLUTION= info.replace("RESOLUTION=", "").split("x");
                media.setWidth(Integer.valueOf(RESOLUTION[0]));
                media.setHeight(Integer.valueOf(RESOLUTION[1]));
                media.setQualityLabel(RESOLUTION[1] + "p");
            }
        }
        //LogUtils.log(media.getUrl());
        muxedMedia.add(media);
    }
}

public interface ExtractorListner {
    void onExtractionGoesWrong(ExtractorException e);
    void onExtractionDone(List<YTMedia> adativeStream, List<YTMedia> muxedStream, List<YTSubtitles> subList, YoutubeMeta meta);
}

}

==============================================================================

public class PlayerResponse { private PlayabilityStatus playabilityStatus; private StreamingData streamingData; private YoutubeMeta videoDetails; private Captions captions; private String playerJs;

public void setPlayerJs(String playerJs){
    this.playerJs = playerJs;
}

public String getPlayerJs(){
    if( playerJs.startsWith("http") && playerJs.contains("youtube.com") ){
        return playerJs.replace("\\", "");
    }
    else{
        return "https://www.youtube.com" + playerJs.replace("\\", "");
    }
}

public void setCaptions(Captions captions) {
    this.captions = captions;
}

public Captions getCaptions() {
    return captions;
}

public void setPlayabilityStatus(PlayabilityStatus playabilityStatus) {
    this.playabilityStatus = playabilityStatus;
}

public PlayabilityStatus getPlayabilityStatus() {
    return playabilityStatus;
}

public void setStreamingData(StreamingData streamingData) {
    this.streamingData = streamingData;
}

public StreamingData getStreamingData() {
    return streamingData;
}

public void setVideoDetails(YoutubeMeta videoDetails) {
    this.videoDetails = videoDetails;
}

public YoutubeMeta getVideoDetails() {
    return videoDetails;
}

public class Captions{
    private PlayerCaptionsTracklistRenderer playerCaptionsTracklistRenderer;

    public void setPlayerCaptionsTracklistRenderer(PlayerCaptionsTracklistRenderer playerCaptionsTracklistRenderer) {
        this.playerCaptionsTracklistRenderer = playerCaptionsTracklistRenderer;
    }

    public PlayerCaptionsTracklistRenderer getPlayerCaptionsTracklistRenderer() {
        return playerCaptionsTracklistRenderer;
    }

    public class PlayerCaptionsTracklistRenderer{
        private List<YTSubtitles> captionTracks;

        public void setCaptionTracks(List<YTSubtitles> captionTracks) {
            this.captionTracks = captionTracks;
        }

        public List<YTSubtitles> getCaptionTracks() {
            return captionTracks;
        }
    }
}

public class PlayabilityStatus{
    private String status;
    private boolean playableInEmbed;

    public void setStatus(String status) {
        this.status = status;
    }

    public String getStatus() {
        return status;
    }

    public void setPlayableInEmbed(boolean playableInEmbed) {
        this.playableInEmbed = playableInEmbed;
    }

    public boolean isPlayableInEmbed() {
        return playableInEmbed;
    }
}

}

[hhyeok1026]

@NiazSagor

In my app that I edited, I'm getting reviews from users saying that it doesn't work. Don't trust my source code.

What I have fixed and what is already existing may have to coexist.

[NiazSagor]

@hhyeok1026 thank you for the code. now what I have faced, some urls are working and some are not.

some of the links that are not working 1WEk-ecKW00 Fmcwap9uwLI P0ysy7Bwjy0

some links that are working kpmsdqQwNW8

I am confused about how to deal with the problem. How to know for sure which links are going to work and which are not.

The error :

2020-11-27 15:25:29.991 29442-29442/com.angik.exoplayer I/Sagor: Error While getting Youtube Data:java.lang.NullPointerException: Attempt to invoke virtual method 'int java.lang.String.length()' on a null object reference at java.io.StringReader.<init>(StringReader.java:50) at com.google.gson.JsonParser.parseString(JsonParser.java:47) at com.google.gson.JsonParser.parse(JsonParser.java:98) at com.angik.exoplayer.YoutubeStreamExtractor.parseJson(YoutubeStreamExtractor.java:145) at com.angik.exoplayer.YoutubeStreamExtractor.doInBackground(YoutubeStreamExtractor.java:111) at com.angik.exoplayer.YoutubeStreamExtractor.doInBackground(YoutubeStreamExtractor.java:29) at android.os.AsyncTask$2.call(AsyncTask.java:333) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:764)

[hhyeok1026]

@NiazSagor

I will check the video. I posted a commit. Check it out. #42 After checking the video above, if there is no problem, I will do a pull request.

// The above 4 videos, I checked it in my modified program. I found itag 140, all worked fine.

// It seems that there are 3 cases as the body value that was responded to the request.

"case1" - Structure applied to the original. "case2" - Structure applied by my modifications. "case3" - Case where both 1 and 2 coexist

So I checked if I could bring it to number 1 first, and if it didn't, I made it to number 2.

The above cases may be different for each video, on a specific device. I saw a case that "case1" was changed to "case2" and "case1" was no longer visible on the device. When I first analyzed this issue, I saw "case3". Now almost "case2" is visible.

// p.s. In the code I posted earlier, I found another case in the regex and modified it a bit.