jpjones76
commented
3 years ago If all you want is the number of gaps in a file, then you don't need to read the entire file into memory. I can come up with a workaround that will scan each record and simply skip the blockettes. The pseudocode would be this, more or less:
- Initialize
ngaps, anArray{Int,1};end_times, anArray{Int64,1};ids, anArray{Array{UInt8,1},1}. - Parse the (48-byte) "Fixed Section of Data Header".
- Read bytes 9-20 into an
Array{UInt8,1}, call itid. Compare to each array inids; if new, append toids, then append a 0 tongaps. Store index asj. - Get record start time, number of samples, and fs.
- Compare to
end_times[j]. If there's a gap, incrementngaps[j]. - Determine blockette size in bytes and
seekpast all blockette data. - Repeat until EOF.
This is much faster than read_data because the latter spends most of its time decompressing blockette data. It also avoids string comparisons.
Were you thinking to parallelize the read and use an async/wait() approach? I suspect that passing data between CPUs will be slower than looping over records, unless you can think of a way to to divide files into arbitrary-length chunks. The latter is hard, though, because we don't know where new records begin in the file. You'd need a reliable test to identify the start of a new record; I know of none.
tclements
This is more of a discussion on mseed than an issue.
I have some mseed files with > 10,000 gaps in a dataset of ~3 million files.
read_datatakes about 20 seconds on the files with 10,000+ gaps each. This is expected and totally fine but I'd like to avoid reading files such as this in the future when I process the rest of the dataset. I can't determine which files are bad a priori.I'm looking for a way to get a rough estimate of the number of gaps in a file. Does this still require reading every blockette? Looking through
parserec!, I'm not sure where to begin.My current solution is to implement something like discussed here https://github.com/JuliaLang/julia/issues/36217 on top of
read_data.