Removing `schedule_info` from `google_data_pipeline_pipeline` does not delete Cloud Scheduler job

[kpocius]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version

1.7.4

Affected Resource(s)

google_data_pipeline_pipeline

Terraform Configuration

No response

Debug Output

No response

Expected Behavior

Removing schedule_info should remove the associated Cloud Scheduler job

Actual Behavior

Cloud Scheduler job remains in place and the pipeline keeps getting executed on the schedule which is now not defined anywhere in the code.

Steps to reproduce

1. Create a pipeline with schedule_info initially set
2. Remove schedule_info block

Important Factoids

No response

References

No response

b/328769966

[ggtisc]

Issue confirmed! After creates a pipeline and then remove the block of the scheduler job it remains (everything through terraform) Followed steps:

1. Create a 'google_data_pipeline_pipeline' resource in terraform
2. Execute 'terraform apply'
3. Edit the 'tf' file removing the 'schedule_info' block
4. Update resources in GCP with a 2nd 'terraform apply'
5. Watch the resource on GCP... The schedule remains

[jonathan-lemos]

This happens because the underlying API doesn't support removing a schedule from a Data Pipeline. This is also why the GCP Web UI lacks this option. Can I ask what the use case of a pipeline without scheduling is? I ask because Data Pipelines is not meant to support pipelines without scheduling, and next steps depend on the use case.

[kpocius]

We are creating our own schedules, so we can control additional parameters such a retries.

For now a workaround is not to set the schedule on google_data_pipeline_pipeline in the first place, and just clean up old schedules manually.

Thanks for looking into this.

[jonathan-lemos]

Makes sense. Unfortunately, our headcount is short these days and it's unlikely that we'll have time to fix this service-side any time soon. For the time being, the workaround will have to do. Sorry.

[kpocius]

No worries, thanks for looking into it.