Since the WaybackIndexer copies files from the store to a wayback input folder, every warc file exists on the filesystem twice, until its parent target instance is archived. For users who have many target instances in a pre-archived state, we could save a significant amount of storage space by giving them the (configuration) option to use soft links to warc files instead of copies. The structure of the wayback input folder would remain unchanged, but the warc file entries would be soft links to files inside the store directory.
Since the WaybackIndexer copies files from the store to a wayback input folder, every warc file exists on the filesystem twice, until its parent target instance is archived. For users who have many target instances in a pre-archived state, we could save a significant amount of storage space by giving them the (configuration) option to use soft links to warc files instead of copies. The structure of the wayback input folder would remain unchanged, but the warc file entries would be soft links to files inside the store directory.