Failure to remove log results in ResourceAlreadyExistsException

dask / dask-cloudprovider

Cloud provider cluster managers for Dask. Supports AWS, Google Cloud Azure and more...

https://cloudprovider.dask.org

BSD 3-Clause "New" or "Revised" License

132 stars 109 forks source link

Failure to remove log results in ResourceAlreadyExistsException #70

Open NicWayand opened 4 years ago

NicWayand commented 4 years ago

When trying to spin up a new Fargate cluster following @rsignell-usgs 's example, I have to manually delete the log group in cloudwatch, or I get an error message:

ResourceAlreadyExistsException: An error occurred (ResourceAlreadyExistsException) when calling the CreateLogGroup operation: The specified log group already exists

My expected behavior is that it will clean up the log group.

Do I need to close out the cluster for this to happen?

rsignell-usgs commented 4 years ago

I think @jacobtomlinson told me there is a bug there. As a workaround you can add skip_cleanup=True, like:

cluster = FargateCluster(n_workers=1, image='rsignell/pangeo-worker:2020-01-23c',
                         skip_cleanup=True)

NicWayand commented 4 years ago

Thanks @rsignell-usgs, but I still get the ResourceAlreadyExistsException even with the skip clean up as True. (I tried a few times).

jacobtomlinson commented 4 years ago

It sounds like there are two issues here:

If the log group exists already an exception is raised. We should catch this and continue.
The log group is not being correctly cleaned up on shutdown.

A workaround in the meantime would be to set the cloudwatch_logs_group kwarg to the name of your existing log group.

cluster = FargateCluster(n_workers=1, cloudwatch_logs_group="my_log_group")