Granola-Team / mina-indexer

The Mina Indexer is a re-imagined version of the software collectively called the "Mina archive node."
Apache License 2.0
18 stars 10 forks source link

Problem: indexer discriminates between files and links #1218

Closed robinbb closed 2 months ago

robinbb commented 2 months ago

The Indexer watches for incoming files in its 'watch-blocks' directory and its 'blocks' directory. (These are defined by the --block-watch-dir and --blocks-dir command line switches.) If one attempts to put 'links' (hard links) to files into, say, the 'blocks' directory, the Indexer does not ingest them... except upon initial ingestion. After initial ingestion, when the directory is being "watched" for changes, additions of files that are hard links are ignored. (Test provided. See comments, below.) This means that one cannot create a blocks directory without literally copying files into it, even though it is very likely, in a testing scenario, that one wants to put (hard or soft) links into a staged blocks directory to save on disk space. The disk space of all the blocks in on the order of 1 TB, and will only grow over time.

robinbb commented 2 months ago

See https://github.com/Granola-Team/mina-indexer/pull/1224 for a test that alters the file type added to the directories staged during regression testing. Invoke just bt block_copy - it will fail. (Delete the testing directory between invocations, to make sure of an apples-with-applies comparison. It will be something like /mnt/mina-indexer-dev/rev-dae1b5da

robinbb commented 2 months ago

Taking. I am adding to #1224 the solution, and will merge.