Where to find data from CURL request?

[AndreasEisenbergerDGTLS]

Hello,

I am currently facing a problem where I am using Data Director to import some products from an API endpoint. This endpoint delivers only up to 500 items per response and if there are more it adds a pagination by also sending an URL where I can get the next 500 items.

Now I was trying to get the initial response from the CURL request in the "result callback function" in the mapping and put some logic in there to get the remaining page, but I cant seem to get that response.

Is there some setting for this?

Best regards Andreas

[BlackbitDevs]

You can add a raw data field with __source as xpath / JmesPath / CSV column (whatever type of import format you have). This will give you the temporary file path of your cURL request response.

But actually you can simply add a raw data field for this nextPage url. If it is present, call the same dataport with this URL in the result callback function. This way you can loop through all pages until nextPage is empty.

To call the dataport you can use the Call dependent import template for the result callback function. In the virtuel field DEPENDENT DATAPORT PARAMETERS return this:

return ['nextPage' => $params['value']];

and assign the raw data field nextPage. Then you will have to configure the import resource for this dataport to something like curl {{ nextPage|default("http://example.org/api") }} - this will use the nextPage parameter from DEPENDENT DATAPORT PARAMETERS if present - and if not it will use the default / start URL.

Beware to adjust result callback function to check if nextPage is not empty - otherwise the dependent dataport will get called even if there is no nextPage resulting in an endless loop.