Open EddieLF opened 2 months ago
The auditing module was created to scan the upload bucket & query metamist to work out which files can be safely deleted and which files should be ingested.
It does its job fairly well, however some queries are unoptimized and failing for large datasets. E.g. https://batch.hail.populationgenomics.org.au/batches/454428/jobs/1 https://batch.hail.populationgenomics.org.au/batches/454440/jobs/1 https://batch.hail.populationgenomics.org.au/batches/454445/jobs/1 https://batch.hail.populationgenomics.org.au/batches/454450/jobs/1
The auditing module was created to scan the upload bucket & query metamist to work out which files can be safely deleted and which files should be ingested.
It does its job fairly well, however some queries are unoptimized and failing for large datasets. E.g. https://batch.hail.populationgenomics.org.au/batches/454428/jobs/1 https://batch.hail.populationgenomics.org.au/batches/454440/jobs/1 https://batch.hail.populationgenomics.org.au/batches/454445/jobs/1 https://batch.hail.populationgenomics.org.au/batches/454450/jobs/1