V2.7.0: InnertubeError: Grid not found!

titong0 commented 1 year ago

Steps to reproduce

I don't know why I got this, it was working one reload before.

Failure Logs

InnertubeError: Grid not found!
This is a bug, want to help us fix it? Follow the instructions at https://github.com/LuanRT/YouTube.js/blob/main/docs/updating-the-parser.md or report it at https://github.com/LuanRT/YouTube.js/issues!
    at Function._Parser_printError (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\index.js:293:29)
    at Function.parseItem (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\index.js:155:77)
    at Function.parseArray (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\index.js:170:37)
    at new SectionList (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\classes\SectionList.js:15:41)
    at Function.parseItem (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\index.js:150:32)
    at new Tab (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\classes\Tab.js:18:40)
    at Function.parseItem (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\index.js:150:32)
    at Function.parseArray (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\index.js:170:37)
    at new SingleColumnBrowseResults (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\classes\SingleColumnBrowseResults.js:12:37)
    at Function.parseItem (C:\Users\Tiktok\Desktop\programacao\auto-heardle-frontend\node_modules\youtubei.js\dist\src\parser\index.js:150:32) {
  info: {
    items: [
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object]
    ],
    trackingParams: 'CAQQ6IsCGAAiEwif_b6--7b8AhUyT0gAHYpGCfg=',
    header: { gridHeaderRenderer: [Object] }
  },
  date: 2023-01-08T02:50:48.950Z,
  version: '2.7.0'
}

Expected behavior

no crashing

Current behavior

it crashes

Version

Default

Anything else?

No response

Checklist

[X] I am running the latest version.
[X] I checked the documentation and found no answer.
[X] I have searched the existing issues and made sure this is not a duplicate.
[X] I have provided sufficient information.

LuanRT commented 1 year ago

Did anything else happen after this warning? If so, it would be nice if you could tell me what your program was doing when this popped up. Thanks!

titong0 commented 1 year ago

the programming crashed because I had the following code

if (!grid?.items) throw new Error();

It sometimes works and others it just doesn't. I have no clue what the issue is.

LuanRT commented 1 year ago

What I mean is what method (from the library) caused the warning to appear on your application. The condition !grid?.items is failing because when a node is not found it will default to null. I can just add the missing node but more context would be appreciated, as I can't seem to be able to reproduce.

titong0 commented 1 year ago

Sorry for thew lack of activity on this, after a lot of experimenting I still don't know when this happens. My code does the following:

Mostly code calling the library

const youtube = Innertube.create();
const client = await youtube;
const artist = await client.music.getArtist(artistId);

// both of these are MusicCarouselShelf | undefined
const { albums, singles } = removeUnnecesarySections(artist.sections);

// calling getFullSection is what causes the crash
getFullSection(albums).then((albums) => {
    if (!albums) return null;
    return Promise.all(albums.map((album) => getSongsFromAlbum(album)));
  });

Utilities

const removeUnnecesarySections = ( sections: (MusicCarouselShelf | MusicShelf)[]) => {
  const albums = sections.find((section) => {
      if (section.is(MusicCarouselShelf))
        return section.header?.title.text === "Albums";
    })
    ?.as(MusicCarouselShelf);
  const singles = sections.find((section) => {
      if (section.is(MusicCarouselShelf))
        return section.header?.title.text === "Singles";
    })
    ?.as(MusicCarouselShelf);

  return { albums, singles };
};

the function where the crash (sometimes) happens

const getFullSection = async (section: MusicCarouselShelf | undefined) => {
  if (!section) return null;

  const client = await youtube;
  const button = section.header?.more_content;
  if (!button) {
    // return just the items since there are no other albums
    if (section.contents[0] instanceof MusicTwoRowItem) {
      return section.contents as ObservedArray<MusicTwoRowItem>;
    }
    throw new Error(`no "see more button" neither 'album.contents'`);
  }
  const page = await button.endpoint.call(client.actions, {
    parse: true,
    client: "YTMUSIC"
  });

  // when the crash happens, this 'page' object is filled with nulls (example at the end)
  console.log(page);

  if (!page) throw new Error("No page");

  const single_col = page.contents.item().as(SingleColumnBrowseResults);

  const grid = single_col.tabs
    .firstOfType(Tab)
    ?.content?.as(SectionList)
    .contents.find((a) => a.is(Grid)) as Grid;

  if (!grid?.items) throw new Error("No grid for this section");
  const items = grid?.items.as(MusicTwoRowItem);
  return items;
};

The returned object from calling button.endpoint.call()

{
  actions: null,
  actions_memo: Memo(0) [Map] {},
  contents: SuperParsedResult {},
  contents_memo: Memo(3) [Map] {
    'SectionList' => [ [SectionList] ],
    'Tab' => [ [Tab] ],
    'SingleColumnBrowseResults' => [ [SingleColumnBrowseResults] ]
  },
  header: SuperParsedResult {},
  header_memo: Memo(1) [Map] { 'MusicHeader' => [ [MusicHeader] ] },
  sidebar: null,
  sidebar_memo: Memo(0) [Map] {},
  live_chat_item_context_menu_supported_renderers: null,
  live_chat_item_context_menu_supported_renderers_memo: Memo(0) [Map] {},
  on_response_received_actions: null,
  on_response_received_actions_memo: Memo(0) [Map] {},
  on_response_received_endpoints: null,
  on_response_received_endpoints_memo: Memo(0) [Map] {},
  on_response_received_commands: null,
  on_response_received_commands_memo: Memo(0) [Map] {},
  continuation: null,
  continuation_contents: null,
  metadata: null,
  microformat: null,
  overlay: null,
  refinements: null,
  estimated_results: null,
  player_overlays: null,
  playback_tracking: null,
  playability_status: undefined,
  streaming_data: undefined,
  current_video_endpoint: null,
  endpoint: null,
  captions: null,
  video_details: undefined,
  annotations: [],
  storyboards: null,
  endscreen: null,
  cards: null
}

titong0 commented 1 year ago

updated to V2.9.0 and it seems to always work now.

titong0 commented 1 year ago

update: after using a bit, it is happening again. Even in V2.9,0.

titong0 commented 1 year ago

Hi there, any update on this? I still can't seem to get it to work consistently

LuanRT commented 1 year ago

Hi @titong0, I've been trying to reproduce this but unfortunately had no success. Do you know if this happens consistently with a specific artist? That would probably help.

LuanRT commented 1 year ago

Ah also, next time this happens please post the contents of the contents_memo map. That will most definitely give us some more info about what page InnerTube is returning in place of the actual data.

titong0 commented 1 year ago

Hi @titong0, I've been trying to reproduce this but unfortunately had no success. Do you know if this happens consistently with a specific artist? That would probably help.

Hi. No, it does not seem to happen with any specific artist. the problem seems to be caused by the 'see more' button, so any artist with several albums will ocassionally bug.

Ah also, next time this happens please post the contents of the contents_memo map. That will most definitely give us some more info about what page InnerTube is returning in place of the actual data.

I'll try using the library and if it crashes I'll paste it here.

LuanRT commented 1 year ago

Got it. Thank you!

titong0 commented 1 year ago

Endpoint object that used 'call' method

{
  "type": "NavigationEndpoint",
  "payload": {
    "browseId": "UCRY5dYsbIN5TylSbd7gVnZg",
    "params": "6gPjAUdxY0JXcGdCQ3BVQkNpUjVkRjl3WVdkbFgzTnVZWEJ6YUc5MFgyMTFjMmxqWDNCaFoyVmZjbVZuYVc5dVlXd1NIekI1U1dkVU9Ua3dRVE5uWm1OT01sbzBRbTlUZUVSdlgzSk5ORFppUW1jYVRBQUFaVzRBQVVGU0FBRkJVZ0FCQUVaRmJYVnphV05mWkdWMFlXbHNYMkZ5ZEdsemRBQUJBVU1BQUFFQUFRQUFBUUVBVlVOU1dUVmtXWE5pU1U0MVZIbHNVMkprTjJkV2JscG5BQUh5MnJPcUNnWkFBVWdBVUF3",
    "browseEndpointContextSupportedConfigs": {
      "browseEndpointContextMusicConfig": {
        "pageType": "MUSIC_PAGE_TYPE_ARTIST"
      }
    }
  },
  "metadata": {
    "api_url": "/browse"
  }
}

Return value

{
  "actions": null,
  "actions_memo": {},
  "contents": {},
  "contents_memo": {},
  "header": {},
  "header_memo": {},
  "sidebar": null,
  "sidebar_memo": {},
  "live_chat_item_context_menu_supported_renderers": null,
  "live_chat_item_context_menu_supported_renderers_memo": {},
  "on_response_received_actions": null,
  "on_response_received_actions_memo": {},
  "on_response_received_endpoints": null,
  "on_response_received_endpoints_memo": {},
  "on_response_received_commands": null,
  "on_response_received_commands_memo": {},
  "continuation": null,
  "continuation_contents": null,
  "continuation_contents_memo": {},
  "metadata": null,
  "microformat": null,
  "overlay": null,
  "refinements": null,
  "estimated_results": null,
  "player_overlays": null,
  "playback_tracking": null,
  "current_video_endpoint": null,
  "endpoint": null,
  "captions": null,
  "annotations": [],
  "storyboards": null,
  "endscreen": null,
  "cards": null
}

LuanRT commented 1 year ago

@titong0 Thank you very much for your help! After hours of debugging I finally found the root cause of this issue, it turns out YouTube Music doesn't like some of the randomly generated visitor id the library creates :⁠-⁠P

If you can, please try v3.3.0 to confirm this is fixed.

titong0 commented 1 year ago

Hi there! I finnally got a chance to try it out. I had to change most Imports but now I'm getting the following error: Module not found: Package path ./dist/src/parser/map is not exported from package ...\node_modules\youtubei.js (see exports field in ...\node_modules\youtubei.js\package.json) and this is the export object I found under node_modules\youtubei.js\package.json:

 "exports": {
    ".": {
      "node": {
        "import": "./dist/src/platform/node.js",
        "require": "./bundle/node.cjs"
      },
      "deno": "./dist/src/platform/deno.js",
      "types": "./dist/src/platform/lib.d.ts",
      "browser": "./dist/src/platform/web.js",
      "default": "./dist/src/platform/web.js"
    },
    "./agnostic": {
      "types": "./dist/src/platform/lib.d.ts",
      "default": "./dist/src/platform/lib.js"
    },
    "./web": {
      "types": "./dist/src/platform/lib.d.ts",
      "default": "./dist/src/platform/web.js"
    },
    "./web.bundle": {
      "types": "./dist/src/platform/lib.d.ts",
      "default": "./bundle/browser.js"
    },
    "./web.bundle.min": {
      "types": "./dist/src/platform/lib.d.ts",
      "default": "./bundle/browser.min.js"
    }
  },

I'm not really sure what causes this, I'm importing the libraries' classes like this:

import {
  Grid,
  MusicCarouselShelf,
  MusicShelf,
  MusicTwoRowItem,
  SectionList,
  SingleColumnBrowseResults,
  Tab,
} from "youtubei.js/dist/src/parser/map";

I tried deleting node_modules and installing with @latest but nothing worked.

absidue commented 1 year ago

@titong0 v3 cleaned up the platform support and changed how the imports work, you can now import those specific classes like this:

import { YTNodes } from 'youtubei.js';

const tab = new YTNodes.Tab(/* code */);

v3 changelog: https://github.com/LuanRT/YouTube.js/releases/tag/v3.0.0

titong0 commented 1 year ago

Thanks! I'm not sure wether the grid error has been solved because now Im getting something else from nextjs.


error - unhandledRejection: TypeError: First parameter has member 'readable' that is not a ReadableStream.
    at Y (C:...\node_modules\next\dist\compiled\@edge-runtime\primitives\streams.js:234:11)
    at C:\..\node_modules\next\dist\compiled\@edge-runtime\primitives\streams.js:1775:50
    at ReadableStream.pipeThrough (C:...\node_modules\next\dist\compiled\@edge-runtime\primitives\streams.js:1778:6)
    at fetchFinale (C:...\node_modules\undici\lib\fetch\index.js:973:52)
    at mainFetch (C:...\node_modules\undici\lib\fetch\index.js:773:5)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)```
I don't know wether this is a next.js thing because it wasn't happening before updating to 3.3.0. Sorry if it has nothing to do

LuanRT / YouTube.js