andyblarblar / transferware.ai

“Shards are glass, sherds are pottery”
0 stars 0 forks source link

Create CSV cache #28

Closed andyblarblar closed 8 months ago

andyblarblar commented 8 months ago

Using the API access lib, create a class that abstracts over a cache of the API. This class would take a path to a directory as an argument, and check the cache in that directory. Using the DB access lib, it will ensure that cache is complete. The cache should have the following layout:

This will be used in the training process, to both train the model and generate an embeddings store.

wyliew19 commented 8 months ago

Updated to JSON cache (later will be parquet if space or speed becomes concern)