search

harrylloyd-bl / convert-a-card

Generating ingestible records from catalogue cards as part of the Convert-a-Card project
1 stars 0 forks source link

Persistent data storage #7

Open harrylloyd-bl opened 1 year ago

harrylloyd-bl commented 1 year ago

There needs to be a better place to keep the stored data. I think we had an overwrite of data as a result of it being kept in the git repo which was never a sensible place to keep it in the first place. This includes both the dataframe of results and images of cards.

Explore some network storage. The data can remain as a pickle rather than a db while the format is still MARC records which are difficult to store naturally. But it should be somewhere sensible rather than in the git repo.

harrylloyd-bl commented 1 year ago

Attempted to use s3 storage instead. Not BL sanctioned but as a working example. Calling the storage works fine but slows down the app, so caching the result of the s3 load call, operating on that and only saving back to s3 when necessary. Fixed in 8e07bdc63004a4060c16e727855be9a711459322

harrylloyd-bl commented 5 months ago

Still using S3 free tier, transition to internal server planned