Use checksums when updating database

WEHI-ResearchComputing / rag

RAG toy example to build on. Copied from https://github.com/pixegami/rag-tutorial-v2

0 stars 1 forks source link

Use checksums when updating database #1

Open edoyango opened 2 months ago

edoyango commented 2 months ago

Currently, populate_database.py only updates the database when it detects a new file. Would be handy to replace files in the database that have since been updated. Perhaps with sha256sum?

edoyango commented 1 week ago

I'll add checksums should probably be used at point of file discovery so as not to parse files unnecessarily. As identified in https://github.com/WEHI-ResearchComputing/rag/issues/11