search

4dn-dcic / tibanna

Tibanna helps you run your genomic pipelines on Amazon cloud (AWS). It is used by the 4DN DCIC (4D Nucleome Data Coordination and Integration Center) to process data. Tibanna supports CWL/WDL (w/ docker), Snakemake (w/ conda) and custom Docker/shell command.
MIT License
70 stars 28 forks source link

Turn off cloudwatch and costexplorer #401

Closed laurentiush closed 3 months ago

laurentiush commented 8 months ago

Hi,

I am running tibanna regularly, but am incurring quite some costs for 2 specific 'usage types':

Now that my pipeline is up and running, I don't really need all the detailed information. Is there a way to turn off this logging and monitoring?

One option could be to remove the rights to access Cost Explorer, but I don't know if this will break tibanna:

https://repost.aws/questions/QUhjMTBYs2QGSFcgto3BdJMA/disable-aws-cost-explorer-permissions

alexander-veit commented 8 months ago

Hi,

currently there is no option to switch it off, but I agree that there should be one. I will look into it.

I think that simply removing Tibanna's permissions to Cloudwatch will probably break it.

laurentiush commented 8 months ago

Thanks! Especially if you have a lot of small tasks those costs can quickly add up.

At some point I tried grouping rules in snakemake to reduce the number of instances required, but somehow the output was not detected by snakemake after transferring it back to S3 and pipelines would always fail because of that.

laurentiush commented 4 months ago

Hi again,

I noticed that you have created a branch which addresses my question and was wondering if you plan to include those changes in a next release.

alexander-veit commented 3 months ago

Hi,

I started working on this some time ago but got distracted with other things. I will try to make some progress soon.

laurentiush commented 3 months ago

Thanks a lot! I see that it is now available.

alexander-veit commented 3 months ago

Yes, I still need to release the new version. I will do that today.