[Bug]: `PageIterator` skip last track

[LWJerri]

For example, I have a playlist with 45 tracks, and when I use PageIterator for getPlaylistTracks I receive a partial tracks list. Here minimal code to reproduce the problem:

const playlistTracksIterator = new PageIterator((opts) =>
  getPlaylistTracks(client, PLAYLIST_ID, opts),
);
const playlistTracksList = await playlistTracksIterator.collect();

console.log(preparedTracksList.length); // 44, because last track is missed.

[LWJerri]

@MellKam, a bug isn't fixed.

const playlistTracksIterator = new PageIterator((opts) =>
  getPlaylistTracks(this.client, env.SPOTIFY_PLAYLIST_ID, opts),
);
const playlistTracksList = await playlistTracksIterator.collect();

const preparedTracksList: Prisma.TrackCreateManyInput[] = playlistTracksList.map((item) => {
  return { id: item.track.id, addedAt: item.added_at, playlistId: env.SPOTIFY_PLAYLIST_ID };
});

console.log(preparedTracksList);

[
  {
    id: '0tl24SxNQgrDQYkQ333wTJ',
    addedAt: '2023-06-30T01:53:19Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '2g0wZFEqqhYuemuR0LF8Uy',
    addedAt: '2023-06-30T01:53:21Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '2WTOrgc5j48naKt1dzIh14',
    addedAt: '2023-06-30T01:53:23Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '64CLNJgz3xmK7P22wWyNGx',
    addedAt: '2023-06-30T01:53:25Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '4NFxFhEqXaJKKXL5cmeO5k',
    addedAt: '2023-06-30T01:53:28Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '2JoIAjiaEU5zXn0cZd79A8',
    addedAt: '2023-06-30T01:53:29Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '30Q0KoiQQ66sKtEl5OG0RK',
    addedAt: '2023-06-30T01:53:31Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '1FZSfHFsN9QMO94PyQnA4h',
    addedAt: '2023-06-30T07:49:40Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '6ANTdU7nub0PjrRZHs2f3G',
    addedAt: '2023-06-30T07:49:42Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '6Nw87i9LU9RyIxAKQb09TT',
    addedAt: '2023-06-30T07:49:44Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '0ycdbYphNKIlhfFrdvyE53',
    addedAt: '2023-06-30T07:49:46Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '3ob2q5w8uZQksLz4WYGmi0',
    addedAt: '2023-06-30T07:50:06Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '4B9TclyfxksAluMJeSCZ4h',
    addedAt: '2023-07-03T14:15:47Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '6gbmylJ7sB7NFfMfTQHosf',
    addedAt: '2023-07-08T22:26:54Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '42rF3FLOvTSXRpVowkwTGY',
    addedAt: '2023-07-08T23:28:20Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '607Gx4Z3NAiqGCGJg9925F',
    addedAt: '2023-07-08T23:28:21Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '4NgherJDfCXWBZBYheM4jK',
    addedAt: '2023-07-08T23:28:22Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '1iMsmdz6NFv30BfFn6H8cE',
    addedAt: '2023-07-08T23:28:23Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '5UGAXwbA17bUC0K9uquGY2',
    addedAt: '2023-07-08T23:28:24Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '3nx1LABlssQw0JCmHTWMn3',
    addedAt: '2023-07-08T23:28:25Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '7wQbjBKlkfy4Cf1kq2fHVn',
    addedAt: '2023-07-08T23:28:26Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '7KWS7lQc8XpBMF6wLFIluY',
    addedAt: '2023-07-08T23:28:27Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '6gTbAJOvivFXwkOEhSIOtS',
    addedAt: '2023-07-08T23:28:27Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '5Shuabm4PbfAe9k57iU8ZB',
    addedAt: '2023-07-08T23:28:28Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '5w5pdKt5Ca3YbLXCQzu8CC',
    addedAt: '2023-07-08T23:28:29Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '7iLmh60QhnumkevZlMVbp3',
    addedAt: '2023-07-08T23:28:30Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '2k1qANbSg71HQO1Zz4udCx',
    addedAt: '2023-07-08T23:28:31Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '4Go1KOwxEUJ9zkH8i2uZBo',
    addedAt: '2023-07-09T19:57:19Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '3lBPDrw1TNad6VZuwfiKEt',
    addedAt: '2023-07-09T20:20:02Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '2iHJyy8g0PhIryKyI2sfRi',
    addedAt: '2023-07-09T20:20:57Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '6tOWUHyvwy2oUOrhTXHCJD',
    addedAt: '2023-07-09T20:21:14Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '1s9e1EBT6eYbDOneeBB6r0',
    addedAt: '2023-07-23T19:55:09Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '1gNcPHAiVIQZmqJFJdt3ti',
    addedAt: '2023-07-25T08:51:27Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '4Iuefv4wqnrO64DaEK6PpW',
    addedAt: '2023-07-25T08:56:19Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '2xFEozG5FIAYttcrZiIEoJ',
    addedAt: '2023-08-08T12:40:43Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '1SgRz27RpBS5VKq1c30roJ',
    addedAt: '2023-08-22T10:18:27Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '25omkMCQxe3Xz1WkmIBBHg',
    addedAt: '2023-08-27T16:36:48Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '43aTeMcUIlwSrWzgkPHxSL',
    addedAt: '2023-08-29T21:43:53Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '7EbS93HCklBNflPfv82mvH',
    addedAt: '2023-09-07T21:02:31Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '7xBe5rIJQxctV4rCQQSKvj',
    addedAt: '2023-09-08T07:48:54Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '0daHbdrLvUmkh81rnolMcG',
    addedAt: '2023-09-08T08:00:47Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '30PG0mWTRb3c4bNN9MYhL3',
    addedAt: '2023-09-09T09:30:27Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '0RgKtaVv27Nff2y29qaD8W',
    addedAt: '2023-09-24T17:40:40Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  },
  {
    id: '4QboxcXr7KnL3C4OtiyxIb',
    addedAt: '2023-09-25T18:42:16Z',
    playlistId: '0Z78i4Fm10pnQFRZ9NLdBa'
  }
]

The last track (id: 6hxa6m7Z7J2yKDDoz90ecM) from the playlist (id: 0Z78i4Fm10pnQFRZ9NLdBa) is still missing.

[MellKam]

@LWJerri Make sure you updated to the last version. That's really strange cause I even wrote a test for it and it all works. Can you reproduce it with other playlist or it is just with this one? I even tried it with some random spotify playlist and it works.

Also a small tip. You can provide a maximum limit for endpoint you trying to reach, so you make less request overall.

const playlistTracksIterator = new PageIterator((opts) =>
  getPlaylistTracks(client, PLAYLIST_ID, opts),
  { limit: 50 } // 50 is max limit for this endpoint
);

[LWJerri]

I tried this code on a playlist from spotify with exactly 100 tracks and ended up with 98 tracks after .collect(). I also set the maximum limit per request to 50 tracks, it looks like the 2 missing tracks are the result of two requests with limit of 50 track.

[LWJerri]

Okay, now I have package v1.0.0-rc11 and still have only 99 songs instead of 100. Here my code:

const playlistTracksIterator = new PageIterator(
  (opts) => getPlaylistTracks(this.client, env.SPOTIFY_PLAYLIST_ID, opts),
  { limit: 50 },
);

const playlistTracksList = await playlistTracksIterator.collect();

console.log(playlistTracksList.length);

[MellKam]

Just tried to test it with my 2023 wrapped playlist and it passed

Deno.test("PageIterator: real playlist", async () => {
    const pageIterator = new PageIterator(
        (opts) => getPlaylistTracks(client, "37i9dQZF1Fa1IIVtEpGUcU", opts),
        { limit: 50 },
    );

    const items = await pageIterator.collect();
    assertEquals(items.length, 100);
});

[LWJerri]

Just tried to test it with my 2023 wrapped playlist and it passed

Deno.test("PageIterator: real playlist", async () => {
  const pageIterator = new PageIterator(
      (opts) => getPlaylistTracks(client, "37i9dQZF1Fa1IIVtEpGUcU", opts),
      { limit: 50 },
  );

  const items = await pageIterator.collect();
  assertEquals(items.length, 100);
});

Well, can you join my stream to show you in live what happens?

twitch.tv/LWGerry

[LWJerri]

:shipit:, bc solved.