Closed ahmadtourei closed 2 months ago
Interesting.
Can you check if the index actually exists? spool.index.index_path.exists()
.
I think what is happening is DASCore can't read any of the files in the directory, so it goes through all of them (hence the progress bar) but ends up not creating an index because it doesn't have any recognizable contents. Then when you print the spool it sees there is no index file and tries to index again.
We don't currently support this file format, but perhaps we could. We don't yet support reading formats which have multiple files but Madagascar does something similar so it may be worth looking into. Is there a spec/example file you can share?
The BinaryReader
isn't for a specific format, its just a way that FiberIO
subclasses tell DASCore they need to read the file in binary mode (e.g., open(data_path, 'rb')
) as oppose to using pytables or h5py.
Can you check if the index actually exists?
Yes, it returns True
. However, I cannot see it in the directory using "ls -a" in the terminal.
I'll update you about sharing an example file tomorrow. Thanks!
closed by #384
Description
Although indexing with
sp = dc.spool(data_path).update()
continues to 100% and creates the index file in the data directory (verified by index_path_1==index_path_2==data_path=="/mnt/DAS/data_1" as below), when printing the spool (sp), it surprisingly indexes it again, and results in a zero size spool:DASCore DirectorySpool 🧵 (0 Patches) Path: /mnt/DAS/data_1
Data is in binary ".raw" format. Each data directory has an XML file with metadata. I see the
BinaryReader
class indascore.io
but I'm unsure if we currently support this raw format with XML metadata.Example
Expected behavior
Versions