One of the ways we can improve AWS ingestion time (#13) is to support a "minimal" mode. In this mode, only a subset of all the data would be ingested, correlating to the resources we support. This will let library users save time and space by ingesting only around 10% (in the case of EC2) of the pricing file.
Implementation details
This can be done by having a new AWS ingestion option, e.g. WithIngestionFilter, that would accept a filtering function. The function would be called for each pricing record, marking the items that should (and shouldn't) be ingested. Terracost might then define two such functions out-of-the-box: a DefaultFilter that ingests everything and a MinimalFilter that only ingests records necessary for the library to work.
The "minimal" mode shouldn't be the default, as it will require re-ingestion each time new resources or attributes of already existing resources become supported.
Abstract
One of the ways we can improve AWS ingestion time (#13) is to support a "minimal" mode. In this mode, only a subset of all the data would be ingested, correlating to the resources we support. This will let library users save time and space by ingesting only around 10% (in the case of EC2) of the pricing file.
Implementation details
This can be done by having a new AWS ingestion option, e.g.
WithIngestionFilter, that would accept a filtering function. The function would be called for each pricing record, marking the items that should (and shouldn't) be ingested. Terracost might then define two such functions out-of-the-box: aDefaultFilterthat ingests everything and aMinimalFilterthat only ingests records necessary for the library to work.The "minimal" mode shouldn't be the default, as it will require re-ingestion each time new resources or attributes of already existing resources become supported.