search

jckuester / awsweeper

A tool for cleaning your AWS account
Mozilla Public License 2.0
467 stars 45 forks source link

Discovery Performance #134

Open kadaan opened 2 years ago

kadaan commented 2 years ago

The performance of discovery is pretty slow as it is a serial process for all available resources.

The improvements I see are:

  1. Parallelize discovery by type
  2. Initial filtering of discovered resources. If all data for filtering exists on the resources then apply the filter and take the result. If all data for filtering is not available then include the result for later processing.
  3. Parallelize updating of resource state by using a worker pool which handles resources after 2.
  4. Final filtering of discovered resources, this time with state
  5. Collect all results from 4. and return them as the result from List()
kadaan commented 2 years ago

@jckuester Feedback?

kadaan commented 2 years ago

See: https://github.com/kadaan/awsweeper/tree/speed_up

jckuester commented 2 years ago

I'll have a look :slightly_smiling_face: