Cursor-Based Pagination

[richardklint]

With the latest API release, certain pagination calls are not working. See https://help.shopify.com/en/api/guides/paginated-rest-results

[tareqtms]

@richardklint We don't have any specific functions to handle paginated results. What update do you want from us?

[richardklint]

This is just to track progress towards getting the new pagination working.

[KiloooNL]

This is just to track progress towards getting the new pagination working.

Pagination is quite easy, this works for me, it may help you:

<?php
$shopify = new PHPShopify\ShopifySDK;

// Shopify paginiation
$productsCount = $shopify->Product->count();
echo '<br>Total Shopify products in Shopify: ' . $productsCount;

$page = 1;   // First page
$limit = 25; // Limit of products per page
$getShopifyProducts = array(); // Create an array for Shopify products

// Work out how many pages there are
$totalPages = ceil($productsCount / $limit);
while ($page <= $totalPages) {
        echo '<br><b>Current page: ' . $page . ' / ' . $totalPages . '</b>';
        $params = array("limit" => $limit, "page" => $page);
        array_push($getShopifyProducts, $shopify->Product->get($params));

        // Go to the next page.
        $page++;
} 

// Now that we have all the products, print the array
print_r($getShopifyProducts);
?>

[zealnagar]

@KiloooNL you are right but after the Curson based pagination in 2019-07 and above they provide next and previous link in header when you pass limit query string parameter. So we need support to access CurlResponse class getHeader method statically. So we can iterate the pages and get relevant info based on the individual use case. @tareqtms

[shawnhind]

You can also use since_id based pagination which they still say is the most efficient (moreso than cursor-based) and has been available in the API for quite a long time.

The cursor pagination in this plugin would be a nice convenience thing to have though.

Using a since_id is the fastest solution and should be the first choice for pagination when order doesn’t matter.

This can also be implemented right now with no change needed to the library.

Ref: https://www.shopify.com/partners/blog/relative-pagination

[tareqtms]

@zealnagar Ok, will have a look before the next release.

[mailramk]

Hi, When can we expect this enhancement implemented as the support for pagination is stopping from April 2020. This is the screenshot from the mail received from Shopify about this.

Thanks Ram

[rick-li-dev]

Hi, this issue should be in top priority. If page filter is removed on April 1, 2020, this SDK will be useless. I am looking forward to hear there is a support for this soon.

Thanks Rick

[shawnhind]

@rickcow take a look at my comment above in this thread. since_id pagination is still supported in this new API and has nothing to do with page filtering being removed. You can use this in this SDK in its current state very easily and isn't going away and is actually even more efficient than cursor based pagination.

Example to retrieve all Products:

$lastId = 1;
$products = [];

do {
    $results = $client->Product->get(['limit' => 250, 'since_id' => $lastId]);
    $products = array_merge($products, $results);
    $lastId = end($results)['id'];
} while (sizeof($results) > 0);

[Coennie]

@shawnhind agreed since_id works perfectly, for most cases... Unfortunately not all endpoinds support since_id. Take the inventoryLevel for instance, this does give the pagination headers, but does not take the since_id.

Regards Coen

[shawnhind]

@Coennie oh okay. Thanks, wasn't aware that this wasn't available for all endpoints, yeah in that case this would be necessary then. I only use the Page, Blog, Product, and ProductCollection endpoints which all have it so I just made an assumption that they all did.

[shamsicodebase]

This is just to track progress towards getting the new pagination working. I also emailed you. Please reply. Thank You

[tareqtms]

Hello Everyone, I am extremely sorry to keep you waiting. I will keep it at the top priority for the next couple of weeks and hopefully come up with a solution. Thanks

[ranjitatwal30]

Awaiting for an update on this one...

[zamliyconsulting]

Hello Everyone, I am extremely sorry to keep you waiting. I will keep it at the top priority for the next couple of weeks and hopefully come up with a solution. Thanks

Is this helpful? https://community.shopify.com/c/Shopify-APIs-SDKs/How-to-parse-Link-data-from-header-data-in-PHP/m-p/570407#M38050

https://stackoverflow.com/questions/57801924/how-to-create-pagination-in-shopify-rest-api-using-php

https://github.com/slince/shopify-api-php

[tareqtms]

Hi everyone,

I have created a new release today (v1.1.6) with an available pull request for the new pagination system. I reviewed the codes and fixed some coding issues, but I couldn't test it 100%. At the final build the next page could be retrieved by the following example code:

$productResource =  $shopify->Product();
$products = $productResource->get(['limit'=>50]);
$nextPageProducts = $productResource->get($productResource->getNextPageParams());
/* in short: you need to reuse the resource to get the params before making another resource */**

I beg your help to test it out fully and suggest changes or make new pull request to make it better.

[zealnagar]

@tareqtms I have tested the change. I found one bug for getting the previous link and I have fixed. For that Pull Request has been created. You can review and merge it if proper. For getting paginated data following code worked for me.

$productResource = $shopify->Product();
$products        = $productResource->get(['limit' => 250]);
$next_page       = $productResource->getNextPageParams();
while ($next_page) {
    $new_products = $productResource->get($productResource->getNextPageParams());
    $products     = array_merge($products, $new_products);
    $next_page    = $productResource->getNextPageParams();
}
dd(json_encode($products));

getApi() method gave me error. So i removed and it worked.

[tareqtms]

@zealnagar Thanks, I have merged the pull request and created a new build v1.1.7. I also removed the getApi() function from my comment, that was wrong and was giving me the same problem, but forgot to remove that part when copying the codes from another comment.

[zamliyconsulting]

@tareqtms I have tested the change. I found one bug for getting the previous link and I have fixed. For that Pull Request has been created. You can review and merge it if proper. For getting paginated data following code worked for me.

$productResource = $shopify->Product();
$products        = $productResource->get(['limit' => 250]);
$next_page       = $productResource->getNextPageParams();
while ($next_page) {
    $new_products = $productResource->get($productResource->getNextPageParams());
    $products     = array_merge($products, $new_products);
    $next_page    = $productResource->getNextPageParams();
}
dd(json_encode($products));

This does not work for me and I have no idea what I am doing wrong. It keeps on giving me 250 products instead of the 2317 products i have.

$temp = getProductsNEW(); echo count($temp);

function getProductsNEW() { global $shopify; $productResource = $shopify->Product(); $products = $productResource->get(['limit' => 250]); $next_page = $productResource->getNextPageParams(); while ($next_page) { $new_products = $productResource->get($productResource->getNextPageParams()); $products = array_merge($products, $new_products); $next_page = $productResource->getNextPageParams(); } return $products; }

[zealnagar]

@zamliyconsulting I have copied your code and executed it on my system. It is working. In the output console, 1st execution is with limit 1 filter, and in 2nd execution is with limit 250. In both the case, I got count as 4 since I have 4 products in my store.

Don't know why you are getting only 250. Please make sure the following requirement meets.

1. Update the package to latest version i.e v1.1.7
2. While setting SDK config, you have passed API version >= 2019-07.

[zamliyconsulting]

@zamliyconsulting I have copied your code and executed it on my system. It is working. In the output console, 1st execution is with limit 1 filter, and in 2nd execution is with limit 250. In both the case, I got count as 4 since I have 4 products in my store.

Don't know why you are getting only 250. Please make sure the following requirement meets.

1. Update the package to latest version i.e v1.1.7
2. While setting SDK config, you have passed API version >= 2019-07.

I downloaded the code yesterday, so I assume it is the latest version. API version is already passed in the ShopifySDK.php file (though line 313 was left as 2019-04), so I do not include it in my config array.

Well, since the purpose of this code is to test with more than 250 products, why are you testing with only 4 products? You don't need any of this for 4 products.

[tareqtms]

@zamliyconsulting I think you are facing the issue because of the default version is '2019-04'. You need '2019-07' to make the cursor-based pagination work. I should have made the change to the default version actually. Will do in the next build.

[tareqtms]

Just created a new release with the ApiVersion fix.

[zamliyconsulting]

Just created a new release with the ApiVersion fix.

Thank you but I had already fixed it. Nonetheless, I downloaded it and tested with same results. Has anybody actually tested this with more than 250 products in their system?

[tareqtms]

@zamliyconsulting Are you confirming that it's not working for you after updating the ApiVersion too?

[zamliyconsulting]

@zamliyconsulting Are you confirming that it's not working for you after updating the ApiVersion too?

Yes, I just downloaded your latest version 5 minutes ago. Same result. I also tried including the api version in the config array, same result. $next_page array before the loop is coming empty even though I have 2317 products.

[zealnagar]

@zamliyconsulting I have tested with more than 250 products please see the attached screenshot. It is working absolutely fine.

[zealnagar]

@zamliyconsulting I have updated to the latest version of the package. see my composer.lock

[zealnagar]

@zamliyconsulting let's connect on Google meet and I will help you with this. Please send me google meet link zealnagar.nagar@gmail.com

[zamliyconsulting]

@zamliyconsulting Are you confirming that it's not working for you after updating the ApiVersion too? @zealnagar @tareqtms

OK, i finally found the problem. I still don't know how this was working for you @zealnagar?

@tareqtms - You will need the fix the following in shopifyResource.php: On lines 558 and 559, change x-shopify-api-version to X-Shopify-API-Version On lines 563,564,565, change ['link'] to ['Link']

NOW, it is working!!!!

[zamliyconsulting]

How does this work with Order or InventoryLevel, especially when you need to have parameters other than 'limit'? I could not make it work.

[zealnagar]

How does this work with Order or InventoryLevel, especially when you need to have parameters other than 'limit'? I could not make it work.

Here is the solution for parameter other than limit

$filters = [
    'limit'              => 250,
    'fulfillment_status' => 'unfulfilled',
];
$order_resource = $shopify->Order();
$orders         = $order_resource->get($filters);
$next_page      = $order_resource->getNextPageParams();
while ($next_page) {
    $filters          = array_merge($filters, $next_page);
    $next_page_orders = $order_resource->get($filters);
    $orders           = array_merge($orders, $next_page_orders);
    $next_page        = $order_resource->getNextPageParams();
}

now in the $filter you can pass other parameters to like I have passed.

[arosenhagen]

@zamliyconsulting Are you confirming that it's not working for you after updating the ApiVersion too? @zealnagar @tareqtms

OK, i finally found the problem. I still don't know how this was working for you @zealnagar?

@tareqtms - You will need the fix the following in shopifyResource.php: On lines 558 and 559, change x-shopify-api-version to X-Shopify-API-Version On lines 563,564,565, change ['link'] to ['Link']

NOW, it is working!!!!

This works for me too. We had the problem, that executing an API logic from a local php file went well but running it via a serverless function in AWS lambda only returned those 250 product limit.

[zamliyconsulting]

How does this work with Order or InventoryLevel, especially when you need to have parameters other than 'limit'? I could not make it work.

Here is the solution for parameter other than limit

$filters = [
    'limit'              => 250,
    'fulfillment_status' => 'unfulfilled',
];
$order_resource = $shopify->Order();
$orders         = $order_resource->get($filters);
$next_page      = $order_resource->getNextPageParams();
while ($next_page) {
    $filters          = array_merge($filters, $next_page);
    $next_page_orders = $order_resource->get($filters);
    $orders           = array_merge($orders, $next_page_orders);
    $next_page        = $order_resource->getNextPageParams();
}

now in the $filter you can pass other parameters to like I have passed.

This is incorrect. You can pass the filters ONLY the first time. Here is the corrected version. It works in this format both for InventoryLevel and Order. BTW, you need API Version 2019-10 for these. They do not work with 2019-07.

$filters = [
    'limit'              => 250,
    'fulfillment_status' => 'unfulfilled',
];
$order_resource = $shopify->Order();
$orders         = $order_resource->get($filters);
$next_page      = $order_resource->getNextPageParams();
while ($next_page) {
    $next_page_orders = $order_resource->get($order_resource->getNextPageParams());
    $orders           = array_merge($orders, $next_page_orders);
    $next_page        = $order_resource->getNextPageParams();
}

[raviMukti]

How does this work with Order or InventoryLevel, especially when you need to have parameters other than 'limit'? I could not make it work.

Here is the solution for parameter other than limit

$filters = [
    'limit'              => 250,
    'fulfillment_status' => 'unfulfilled',
];
$order_resource = $shopify->Order();
$orders         = $order_resource->get($filters);
$next_page      = $order_resource->getNextPageParams();
while ($next_page) {
    $filters          = array_merge($filters, $next_page);
    $next_page_orders = $order_resource->get($filters);
    $orders           = array_merge($orders, $next_page_orders);
    $next_page        = $order_resource->getNextPageParams();
}

now in the $filter you can pass other parameters to like I have passed.

This is incorrect. You can pass the filters ONLY the first time. Here is the corrected version. It works in this format both for InventoryLevel and Order. BTW, you need API Version 2019-10 for these. They do not work with 2019-07.

$filters = [
    'limit'              => 250,
    'fulfillment_status' => 'unfulfilled',
];
$order_resource = $shopify->Order();
$orders         = $order_resource->get($filters);
$next_page      = $order_resource->getNextPageParams();
while ($next_page) {
    $next_page_orders = $order_resource->get($order_resource->getNextPageParams());
    $orders           = array_merge($orders, $next_page_orders);
    $next_page        = $order_resource->getNextPageParams();
}

@zamliyconsulting hi zamliy, I have tried the code above, but this gives me a very long time to get a response, does this repeat up to all the orders in the order history? I provide a filter for orders that are paid and also unfulfilled. I have checked the total order on the dashboard, there are only 500 more orders, but this takes approximately 4 minutes

$shopify = new ShopifySDK($config);
        $filters = [
            'limit' => 250,
            'financial_status' => 'paid',
            'fulfillment_status' => 'unfulfilled'
        ];
        $orders_resource = $shopify->Order();
        $orders = $orders_resource->get($filters);
        $next_page = $orders_resource->getNextPageParams();
        while ($next_page) {
            $next_page_orders = $orders_resource->get($orders_resource->getNextPageParams());
            $orders = array_merge($orders, $next_page_orders);
            $next_page = $orders_resource->getNextPageParams();
        }
        return $orders;

[zamliyconsulting]

@zamliyconsulting hi zamliy, I have tried the code above, but this gives me a very long time to get a response, does this repeat up to all the orders in the order history? I provide a filter for orders that are paid and also unfulfilled. I have checked the total order on the dashboard, there are only 500 more orders, but this takes approximately 4 minutes

$shopify = new ShopifySDK($config);
        $filters = [
            'limit' => 250,
            'financial_status' => 'paid',
            'fulfillment_status' => 'unfulfilled'
        ];
        $orders_resource = $shopify->Order();
        $orders = $orders_resource->get($filters);
        $next_page = $orders_resource->getNextPageParams();
        while ($next_page) {
            $next_page_orders = $orders_resource->get($orders_resource->getNextPageParams());
            $orders = array_merge($orders, $next_page_orders);
            $next_page = $orders_resource->getNextPageParams();
        }
        return $orders;

Nobody accused Shopify of being fast :-) You can try adding 'created_at_min' and/or 'created_at_max' to the filters to see if gets faster.

[raviMukti]

@zamliyconsulting hi zamliy, I have tried the code above, but this gives me a very long time to get a response, does this repeat up to all the orders in the order history? I provide a filter for orders that are paid and also unfulfilled. I have checked the total order on the dashboard, there are only 500 more orders, but this takes approximately 4 minutes

$shopify = new ShopifySDK($config);
        $filters = [
            'limit' => 250,
            'financial_status' => 'paid',
            'fulfillment_status' => 'unfulfilled'
        ];
        $orders_resource = $shopify->Order();
        $orders = $orders_resource->get($filters);
        $next_page = $orders_resource->getNextPageParams();
        while ($next_page) {
            $next_page_orders = $orders_resource->get($orders_resource->getNextPageParams());
            $orders = array_merge($orders, $next_page_orders);
            $next_page = $orders_resource->getNextPageParams();
        }
        return $orders;

Nobody accused Shopify of being fast :-) You can try adding 'created_at_min' and/or 'created_at_max' to the filters to see if gets faster.

haha okay if that's the case I'll try your advice first, thanks in advance

[raviMukti]

@zamliyconsulting hi zamliy, I have tried the code above, but this gives me a very long time to get a response, does this repeat up to all the orders in the order history? I provide a filter for orders that are paid and also unfulfilled. I have checked the total order on the dashboard, there are only 500 more orders, but this takes approximately 4 minutes

$shopify = new ShopifySDK($config);
        $filters = [
            'limit' => 250,
            'financial_status' => 'paid',
            'fulfillment_status' => 'unfulfilled'
        ];
        $orders_resource = $shopify->Order();
        $orders = $orders_resource->get($filters);
        $next_page = $orders_resource->getNextPageParams();
        while ($next_page) {
            $next_page_orders = $orders_resource->get($orders_resource->getNextPageParams());
            $orders = array_merge($orders, $next_page_orders);
            $next_page = $orders_resource->getNextPageParams();
        }
        return $orders;

Nobody accused Shopify of being fast :-) You can try adding 'created_at_min' and/or 'created_at_max' to the filters to see if gets faster.

@zamliyconsulting hi zamliy, I get an error like this - Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1679360 bytes) in D:\Project\Web\marketplace-slim\vendor\phpclassic\php-shopify\lib\CurlResponse.php on line 22 it's because the infinite loop isn't?

i have updated my code to something like this :

$shopify = new ShopifySDK($config);
        $filters = [
            'limit' => 250,
            'financial_status' => 'paid',
            'fulfillment_status' => 'unfulfilled',
            'created_at_min ' => '2020-04-09',
            'created_at_max' => '2020-04-10'
        ];
        $orders_resource = $shopify->Order();
        $orders = $orders_resource->get($filters);
        $next_page = $orders_resource->getNextPageParams();
        while ($next_page) {
            $next_page_orders = $orders_resource->get($orders_resource->getNextPageParams());
            $orders = array_merge($orders, $next_page_orders);
            $next_page = $orders_resource->getNextPageParams();
        }
        return $orders;

[zamliyconsulting]

@zamliyconsulting hi zamliy, I get an error like this - Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1679360 bytes) in D:\Project\Web\marketplace-slim\vendor\phpclassic\php-shopify\lib\CurlResponse.php on line 22 it's because the infinite loop isn't?

i have updated my code to something like this :

$shopify = new ShopifySDK($config);
        $filters = [
            'limit' => 250,
            'financial_status' => 'paid',
            'fulfillment_status' => 'unfulfilled',
            'created_at_min ' => '2020-04-09',
            'created_at_max' => '2020-04-10'
        ];
        $orders_resource = $shopify->Order();
        $orders = $orders_resource->get($filters);
        $next_page = $orders_resource->getNextPageParams();
        while ($next_page) {
            $next_page_orders = $orders_resource->get($orders_resource->getNextPageParams());
            $orders = array_merge($orders, $next_page_orders);
            $next_page = $orders_resource->getNextPageParams();
        }
        return $orders;

You need the Y-m-dTH:i:s-P date format for those 2 date. Also, try ini_set('memory_limit', '-1'); on top of your page.

[cbepxpa3ym]

This is crazy how you guys use pagination with "while". Here is the correct way with "do-while":

$productResource = $shopify->Product();
$pageParams = [
    'limit' => 100,
    'product_type' => 'some type',
    'fields' => 'id,variants'
];
do {
    $productsList = $productResource->get($pageParams);
    // place your logic here
    $pageParams = $productResource->getNextPageParams();
} while($pageParams);

[hurta2yaisel]

I'm trying to get the orders with pagination using this code and I can't get the next page info. I can only get the last 15 orders. I'm using the version 1.1.14 of this library (default ApiVersion 2020-01). What I'm doing wrong?

$shopify = ShopifySDK::config($config);
$orderParams = array(
    "financial_status" => "paid",
    "limit" => 50
);
$orderResource = $shopify->Order();
$pending_orders = array();
do {
    $json_data = $orderResource->get($orderParams);
    $pending_orders = array_merge($pending_orders, $json_data);
    $orderParams = $orderResource->getNextPageParams();

    file_put_contents('recovery.json', json_encode($pending_orders));
    //var_dump($json_data);
    var_dump($orderParams);
    var_dump(count($pending_orders));
    var_dump($orderResource->getNextLink());
} while ($orderParams);

[zamliyconsulting]

I'm trying to get the orders with pagination using this code and I can't get the next page info. I can only get the last 15 orders. I'm using the version 1.1.14 of this library (default ApiVersion 2020-01). What I'm doing wrong?

$shopify = ShopifySDK::config($config);
$orderParams = array(
    "financial_status" => "paid",
    "limit" => 50
);
$orderResource = $shopify->Order();
$pending_orders = array();
do {
    $json_data = $orderResource->get($orderParams);
    $pending_orders = array_merge($pending_orders, $json_data);
    $orderParams = $orderResource->getNextPageParams();

    file_put_contents('recovery.json', json_encode($pending_orders));
    //var_dump($json_data);
    var_dump($orderParams);
    var_dump(count($pending_orders));
    var_dump($orderResource->getNextLink());
} while ($orderParams);

I will give you what is working. You can figure out the rest.

$filters = [
    "status" => "any",
    "limit" => 250
];

$orderResource = $shopify->Order();
$orders = $orderResource->get($filters);
$next_page = $orderResource->getNextPageParams();
while ($next_page) {
    $new_orders = $orderResource->get($orderResource->getNextPageParams());
    $orders = array_merge($orders, $new_orders);
    $next_page = $orderResource->getNextPageParams();
}
return $orders;

[cbepxpa3ym]

I will give you what is working.

The code he uses works too. And that code (the loop) is professional comparing to what you have provided. If you do not understand the difference then you have to change your specialty, I think. lol

[cbepxpa3ym]

What I'm doing wrong?

Do you work on the Private app? If not then you have to request access to all orders. Also are you sure you have more than 15 paid orders?

[hurta2yaisel]

What I'm doing wrong?

Do you work on the Private app? If not then you have to request access to all orders. Also are you sure you have more than 15 paid orders?

It is a public application and yep, I have 54 paid orders. Let me try requesting access to all orders.

[hurta2yaisel]

What I'm doing wrong?

Do you work on the Private app? If not then you have to request access to all orders. Also are you sure you have more than 15 paid orders?

I found what I was missing. I had to include the filter by status ("status" => "any"), otherwise it only brought the unfulfilled orders. Access to all orders was not necessary.

$shopify = ShopifySDK::config($config);
$orderParams = array(
    "status" => "any",
    "financial_status" => "paid",
    "limit" => 50
);
$orderResource = $shopify->Order();
$pending_orders = array();
do {
    $json_data = $orderResource->get($orderParams);
    $pending_orders = array_merge($pending_orders, $json_data);
    $orderParams = $orderResource->getNextPageParams();

    file_put_contents('recovery.json', json_encode($pending_orders));
    var_dump($orderParams);
} while ($orderParams);
var_dump($pending_orders);
var_dump(count($pending_orders));

[lexthor]

Hi everyone,

I have created a new release today (v1.1.6) with an available pull request for the new pagination system. I reviewed the codes and fixed some coding issues, but I couldn't test it 100%. At the final build the next page could be retrieved by the following example code:

$productResource =  $shopify->Product();
$products = $productResource->get(['limit'=>50]);
$nextPageProducts = $productResource->get($productResource->getNextPageParams());
/* in short: you need to reuse the resource to get the params before making another resource */**

I beg your help to test it out fully and suggest changes or make new pull request to make it better.

I need something like this but for discount codes. right now the prev and next links are empty.