config_runbackfilljob fails to run using datafactory studio to run the job

[thecloudman]

🐛 Problem

running pipeline config_runbackfilljob in data factory studio fails to run. Set the scope config in settings.json however once you try run the job you get a myriad of errors

One was export time cannot exceed 84 months One was startdate and enddate must be in the same month and another was Either both startDate and endDate needs to be present or both should not be present."

Date format used for the above tests was DD/MM/YYYY

On Second round of testing using a single month with date format MM/DD/YYYY it worked for one month only

Using the same MM/DD/YYYY and spanning multiple months it fails, this was a backfill pipeline and has a variable start and end date, you would assume you could put in start date 13 months ago and end date current date to get 13 months of backfill but that is not working.

[EliWallic]

have the same with the latest release but no date format is working for me. I also saw the API may a bit old but in the body definition the parameters for start and end are "from" and "to" In the API docs https://learn.microsoft.com/en-us/rest/api/cost-management/generate-detailed-cost-report/create-operation?view=rest-cost-management-2023-07-01-preview its described with "start" and "end" - I also tried to change this but didnt worked either.

[flanakin]

Correct, the pipeline was only designed to handle one month at a time. We plan to change that to support multiple months. We can use this issue to track that.

For the date format, we should use ISO 8601 (yyyy-MM-dd) and maybe remove the date part (e.g., yyyy-MM). That said, I'm thinking about making dates optional and defaulting to the full retention period configured in settings.json. With a smart backfill, we can look at the data you have and only pull what's missing. So if we want 3 months and we already have October, then it would only pull September and August. We do need a way to update only a single month or set of months to reprocess data, but in most cases people are looking for that initial backfill.

We can also improve the documentation here so it's clearer how to use these pipelines.