Incremental deploy feature causes payload limit issue

[anfredrick]

We have an instance of data factory with close to 1000 objects. We tried to use the incremental deploy feature, the deployment went fine. But, the pipelines could not start and failed with ErrorCode=RequestContentTooLarge. Upon investigation, we found that the global parameter "adftools_deployment_state" used to hold the MD5 digest for the objects is causing the request size to increase hitting the limit of ADF. Once the parameter value is reset, we could run the pipelines.

It would be good if there is an option to store the deployment state externally instead of the global parameters. Even if there is an option to provide the deployment state as an input json file, it will help us to utilize the feature without hitting the request payload limits.

[NowinskiK]

Thanks @anfredrick for raising this issue. I will check that and consider suggested alternative. What external storage would be best for your scenario? Azure Storage?

[MartaUch]

Hi Kamil :) I didn't notice this issue before, but I think it's related to the problem I raised recently as well: https://github.com/Azure-Player/azure.datafactory.tools/issues/377. I guess we can connect those two. Would be really great to have sort of workaround. I was thinking about Azure Storage, SQL table in database..

Thanks! Marta

[NowinskiK]

I tested the behaviour and I can't reproduce this error. What I've done:

• Added artificial 2000 (2k) objects into adftools_deployment_state global parameter
• Deployed all GParams including the big one

Payload sent = 125kB

All Global Parameters seem to be deployed successfully in target ADF:

@anfredrick please provide more details or script which enables me to reproduce the error. Check my changes in this branch: https://github.com/Azure-Player/azure.datafactory.tools/tree/issue-374

[anfredrick]

Hi @NowinskiK, Thanks for the update and apologies for the delayed response.

There is no issue while deploying the parameters. After deployment, if you try to run any pipeline then the pipeline will fail to start with the error as below,

ErrorCode=FlowRunSizeLimitExceeded, ErrorMessage=Triggering the pipeline failed due to large run size. This could happen when a run has a large number of activities or large inputs used in some of the activities, including parameters.

If you could provide an option to write the adftools_deployment_state value into a file (the same file as input for subsequent deployment), it would be relatively simple. We could then decide to store the file into any storage git/azure blob etc., as we need in our deployment pipelines.

[NowinskiK]

Thanks @anfredrick. Now, it's totally different Error & from this description, I know what I need to test. I will check that out.

[NowinskiK]

I still can't reproduce the issue: My adf: BigFactorySample2-7d6cdb5f

GP:

Pipeline:

Debug run:

Trigger run:

What am I missing?

[MartaUch]

Hello @NowinskiK , I’m tracking a little bit this topic, because like I’ve mentioned in a separate bug while ago, I’ve run into the same issue. Could you please tell how many pipelines do you have inside this one pipeline that you’ve triggered? Is it only this wait task? If yes, then I think it’s really not enough to reproduce the error. For example, we have one master pipeline that triggers inside hundreds of pipelines and thus, after adding global parameter created during incremental deployment we influence on the payload size significantly even when this parameter is not used directly by any pipeline. I’ve exchanged with Microsoft support team in this regard and they’ve confirmed that along with our big master pipeline and the parameter we are running into an issue due to big payload. If any of the pipeline inside uses the global parameter from the list, all global parameters are taken into account.

Best regards Marta

[chalabaj]

Hi, all. WE got to the same issue once we turned on incremental deployment. We have also quite large data factory (700 object) resulting in adftools_deployment_state having > 70k characters. I think this parameter is somehow pushed to all running pipeline and might cause loo large request.

• What I also found that we have this issue mainly with larger pipes of 30-40 activities.

• Once I remove few of activities, especially those having dynamic content such as @pipeline.parameters or @pipeline.TriggerId etc... the pipeline starts to work.

• I turned off incremental deployment and set adftools_deployment_state in datafactory object to dummy value "globalParameters": { "adftools_deployment_state": { "type": "object", "value": { "adftoolsVer": "1.8.000", "Algorithm": "MD5", "LastUpdate": "2024-02-09T14:46:04.275Z" } } } once I did this, pipelines started to work again:

• this would explain issues above which cause the error event for smaller pipes, since in our cause, smaller pipes runs ok.

• what i think now is to store adftools_deployment_state somewhere else after deploynt and set it up in extra predeployment step priora datafactgory.tool deployment

[NowinskiK]

Thanks @MartaUch & @chalabaj for additional feedback and details. I'm planning to fix this issue by saving the adftools_deployment_state to a storage provided in configuration. Azure DevOps Agent should have access to a given location, so ADF Global Params won't be affected. Will that work well?

[g20yr]

@NowinskiK Can we adftools_deployment_state to the git itself? This way there is no dependency on storage and adf

[NowinskiK]

I don't think so... Repositories (like GIT) are for keeping a source of code, not status(es) of deployments. Unless I misunderstood your idea.

[g20yr]

@NowinskiK My idea is that if we store it in storage if someone accidentally deletes it, Entire ADF will be deployed. Instead of this, we create a JSON file in the data factory git repository and update the content of the file with adftools_deployment_state every time after deployment and use that file for next deployment.

[chalabaj]

@g20yr how large would be the file? if it is > few MB I would rather put it to storage than git.

[NowinskiK]

@g20yr , @chalabaj new release will support Storage, not GIT, which is not the right place to keep the deployment statuses.

[chalabaj]

@g20yr , @chalabaj new release will support Storage, not GIT, which is not the right place to keep the deployment statuses.

Perfect. We will gladly test it once it is released.

[MartaUch]

Hi @NowinskiK , I’m also looking forward to try out the new approach once it’s released. For now workaround that I’ve implemented works fine in our platform, but definitely approach with storage will be much better and cleaner :) Thanks!

[Prefix42]

Hi @NowinskiK, super keen for the feature and curious if there are any updates. Happy to help out if I can.

[NowinskiK]

The public preview version of this (fixed & modified) feature has been released: https://www.powershellgallery.com/packages/azure.datafactory.tools/1.10.1-preview @chalabaj @MartaUch @Prefix42 @g20yr - please test it and send me feedback here.