Example on how to use paging

[FlorianLoch]

Hello folks, Just started coding for some new side project using this library to get some data from Spotify. Basic usage is well documented and fairly easily, thanks for that! B I could not find an example using pagination. Taking a rough look at the code and the documentation I could not really see how pagination would work (except taking the URL contained in “next” and handling the whole thing on your own which does not feel right to me). Searching the web I also didn’t find an example but found people asking the same question on stackoverflow (answering it themselves afterwards by saying use “getItems()” which, to my understanding, only gives you the results of the current page) - so I think an example or some explanation in the README would be great.

Thanks. ;)

Cheers Florian

[FlorianLoch]

I saw the remark on asking questions on SO - but I think explaining this in the documentation would be helpful for the project itself, not only for me asking this ;)

[dargmuesli]

This wrapper currently only mirrors what the Spotify web api provides. Adding methods that allow navigating to different pages would be a nice pull request :)

[FlorianLoch]

Hm, I kind of expected this to be the answer but still hoped for another one ;) But yes, this would be very useful - I guess in most cases of information retrieval one would like to have an easy option to get the full answer.

I will implement something for my project and if I am happy with it I might consider creating a PR and contributing it back. Although, there are quite some ways to implement this so some discussion in advance might be good. Let me think about a proper way of adding this. ;)

[Selbi182]

Kind of late, but I did create a hacky solution for retrieving a full paging response some time ago: https://github.com/Selbi182/SpotifyDiscoveryBot/blob/master/src/main/java/spotify/bot/api/SpotifyCall.java

Then you just pass the builder into the method, like so:

List<TrackSimplified> tracksOfAlbum = SpotifyCall.executePaging(spotifyApi.getAlbumsTracks(id));

One of the traps I fell into was not realizing early on that there are actually two types of paging used, one called PagingCursorbased and one simply Paging. In my case, the former cost me some headache, so I only implemented it for string-based cursors for now. As such, this might not be pull-request worthy yet, but it could probably be patched up with some effort (hah).

[jordigarcl]

What about making those requests returning pagination objects implement something like

fun next(nextUrl: String): GetCurrentUsersSavedAlbumsRequest

So there is an easy way to iterate over results while leveraging serialization?

[FlorianLoch]

Sorry to get back to this just yet... I actually implemented the pagination handling as iterator triggering new requests only when necessary. A simple, not yet fully finished approach: https://github.com/FlorianLoch/audiobooks-on-spotify-crawler/blob/master/src/main/java/ch/fdlo/hoerbuchspion/crawler/util/PaginationIterator.java

I consider the iterating approach to be superior to the list based one as it possibly avoids requests not being necessary (say you just want to fetch 200 entries not all of them). The approach linked also makes use of the builder for creating additional requests. I have to admit, I was not aware that there are two types of pagination...

As I see there is some interest in this I might do a PR based on my linked code.

[FlorianLoch]

Additionally I wrote some decorators implementing the IHttpManager interface taking care of stuff like the rate limit (which becomes a thing iterating pages as mentioned by @Selbi182). As this is a separate, although linked, concern it should not be handled by the iterator itself. Same actually holds true for handling errors and triggering retries etc.

https://github.com/FlorianLoch/audiobooks-on-spotify-crawler/blob/master/src/main/java/ch/fdlo/hoerbuchspion/RateLimitMonitoringHttpManager.java

[dargmuesli]

What about the PR you mentioned? :)

[dargmuesli]

Let's go with Kotlin for an example here:

fun <T> getAllPagingItems(requestBuilder: AbstractDataPagingRequest.Builder<T, *>): List<T> {
    val list = arrayListOf<T>()

    do {
        val paging = requestBuilder.build().execute()
        list.addAll(paging.items)
        requestBuilder
                .offset(paging.offset + paging.limit)
    } while (paging.next != null)

    return list
}

Closing this issue as an example is now provided. I'd still welcome PRs as mentioned above though!

[cbismuth]

Here are my Java wrappers :

 private <T> List<T> fetchAllPagingItems(final AbstractDataPagingRequest.Builder<T, ?> builder) {
    try {
        final List<T> items = Lists.newArrayList();

        Paging<T> paging;
        do {
            paging = builder.build().execute();

            final T[] pagingItems = paging.getItems();

            items.addAll(Arrays.asList(pagingItems));

            builder.offset(paging.getOffset() + paging.getLimit());
        } while (paging.getNext() != null);

        return items;
    } catch (final Exception e) {
        throw new RuntimeException(e);
    }
}

private List<Artist> fetchAllFollowedArtists(final GetUsersFollowedArtistsRequest.Builder builder) {
    try {
        final List<Artist> items = Lists.newArrayList();

        PagingCursorbased<Artist> paging;
        do {
            paging = builder.build().execute();

            final Artist[] artists = paging.getItems();

            items.addAll(Arrays.asList(artists));

            final String lastArtistId = artists[artists.length - 1].getId();

            builder.after(lastArtistId);
        } while (paging.getNext() != null);

        return items;
    } catch (final Exception e) {
        throw new RuntimeException(e);
    }
}

[dadoonet]

Wondering if GetPlaylistsItemsRequest should extend AbstractDataRequest<PagingCursorbased<PlaylistTrack>> instead of AbstractDataRequest<Paging<PlaylistTrack>>?

I was trying to implement a way to retrieve more than 100 elements from my playlist and did not find how so far. Anyone succeeded?

[Selbi182]

@dadoonet Maybe this will help, it's how I implemented getting all tracks of a playlist.

https://github.com/Selbi182/SpotifyDependencies/blob/master/src/main/java/spotify/services/PlaylistService.java#L63 https://github.com/Selbi182/SpotifyDependencies/blob/master/src/main/java/spotify/api/SpotifyCall.java#L96

[dadoonet]

Thanks a ton @Selbi182 !