Support for stopping a stream analytics job if changes are needed

drdamour commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Not even sure this is possible given current architecture, but i know new config keeps popping up in the azure rm provider for the feature block that similar to this EG: soft delete for keyvault. but maybe this is too much beyond it's intention.

Stream analytics jobs (SAJ) require being stopped to apply changes to them. I propose a feature block on the provider that enables auto stop and a return to previous state behaviour for any SAJ that need to have changes applied.

The after apply action could be conifgurable as well. leave_stopped, always_start, return_to_prior_state.

We'd also need to consider what to do in case of a failed apply. leave_stopped seems like the right default.

it might even be appropriate to allow per SAJ config of this in the feature block, though my specific use case would be an all or nothing setting.

New or Affected Resource(s)/Data Source(s)

azurerm_stream_analytics_job

Potential Terraform Configuration

provider "azurerm" {
  features {
    api_management {
      stop_prior_to_apply = true #default false
      limit_to_jobs = [  #default empty, no limit
         "job name 1"
      ]
      run_state_after_apply = prior_state #or stopped or #run
      run_state_after_apply_failure = prior_state #or stopped or #run
    }
}

References

No response

tombuildsstuff commented 2 years ago

hey @drdamour

Thanks for opening this issue.

We actually do this in a few places already, namely when resizing the disks for a Virtual Machine and/or for stopping/starting the stream as needed. In the case of Stream Analytics Jobs that's something we should look into doing automatically, since you've mentioned this needs to be done during an update, it's likely we should be doing this automatically for the update to be able to proceed - and if this is for certain fields then we should be conditionally stopping/starting as needed.

As such I don't think we need to add this to the features block (since it's required for the update to be successful/usable) and so I think doing this automatically should be sufficient here, what do you think?

Thanks!

drdamour commented 2 years ago

@tombuildsstuff yeah if it can be done with resources automatically seems fine, i was under impression cause many different resources can cause it that it'd have to be done at provider level.

it's not a problem for my use case, but i could see people who have a use case where they would NOT want a critical SAJ job to be stopped automatically without their knowledge by some iac engineer though...seems liek some level of opt out would be appropriate

tombuildsstuff commented 2 years ago

@drdamour tbh I think we can call that out as a disclaimer/limitation in the docs for now, I agree with you that it could be useful, but I think in this instance since it's required to fully update the resource I think it's probably worth just enabling this functionality for now (without an opt-out) until there's an explicit feature request for it (since otherwise the Job can't be fully updated, which has other caveats)?

drdamour commented 1 year ago

sounds good, looking forward to a solution to this one

sergeibelov113 commented 9 months ago

Any update on this? Would be very useful :)

cicorias commented 7 months ago

If interested, and ideally for feedback, I've created a GitHub action that can provide the stop/update/start processing for when the ASA Transformation query changes.

GitHub Action - Azure Stream Analytics Job Deployment

baselsalam commented 3 weeks ago

Any update on this?

It's been over 2 years and basic functionality on ASA doesn't work without using Null Resources to stop the job. It's not trivial either because the Job needs to be stopped in various states, even an undocumented one (Idle).

hashicorp / terraform-provider-azurerm