Is there a limit to the size of a pipeline variable in Azure Data Factory?

[Willem-J-an]

Is there a limit to the size of a pipeline variable in Azure Data Factory? Could I keep appending petabytes of data to such a pipeline variable? Or until I run out of memory on the runtime?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 36f49a06-90b1-cbe1-3853-2c38277b79f8
• Version Independent ID: 02ff6f6c-3ef6-b48f-060d-b290419e0aba
• Content: Azure subscription limits and quotas - Azure Resource Manager
• Content Source: articles/azure-resource-manager/management/azure-subscription-service-limits.md
• Service: azure-resource-manager
• Sub-service: management
• GitHub Login: @davidsmatlak
• Microsoft Alias: v-dasmat

[BharathNimmala-MSFT]

@Willem-J-an Thank you for your query, we will look into it and get back to you at the earliest.

[BharathNimmala-MSFT]

@Willem-J-an Apologies for delayed response on this , i am checking further with our internal PG team on this query. Will circle back at the earliest.

[BharathNimmala-MSFT]

@Willem-J-an Thank you for being patient on this. Post validation with our Internal team it has been called out that SetVariable is not recommended for appending big data into it and it would be better to understand for your scenario to see if dataflow would be a good fit.

If you have further queries on this please send an email to AzCommunity@microsoft.com  with detailed explanation of your scenario and a link to this thread so we can verify and direct the request accordingly .  We'll continue to investigate this with you directly and will update the thread with any findings that may help others. Thank you

At this point it appears that there's no further edits or updates needed to the documentation. We will go ahead and close this issue, however if you have any further queries please tag us in your reply and will be happy to reopen the thread and continue the conversation. Thank you