search

CyberResearchLabs / mark6

Mark6 Data Acquisition Software
GNU General Public License v2.0
1 stars 1 forks source link

Stale data #6

Open delapsley opened 13 years ago

delapsley commented 13 years ago

$ fchk_test -v -cf=4vdx -cs=8224 /tmp/disk0_out-4207.vdif Format 4 checker 0x4048b0 Setting data size to 8224 Checking /tmp/disk0_out-4207.vdif Begins[2] 23@7629303.550144 PKT00015992 PSN(00007592A24A) - ++p(000075920EB8) = 0000000000009392 ch-2 PKT00015992 VDX fnum diff 8462 != 1934 ch-2 PKT00015992 VDX sre diff 7629305 != 7629304 (~ 1 secs gap) ch-2 PKT00015992 offset 000131518208 Bytes PKT01922558 PSN(000073C3ECAD) - ++p(000075AFB9D0) = FFFFFFFFFE1432DD ch-2 PKT01922558 VDX fnum diff 28087 != 8778 ch-2 PKT01922558 VDX sre diff 7628334 != 7629366 (~ -1032 secs gap) ch-2 ...

so 15992 packets, then a gap ( ~1 sec ); then almost 2 million packets later, we get some very stale data ( sre = seconds of reference epoch )

$ hops_time -q Vex 23@7628334 23@7629366 2011y270d06h58m54.0000s 2011y270d07h16m06.0000s

i.e. "stale" == from ~17 minutes earlier.

delapsley commented 13 years ago

I believe this is caused by the use of fallocate() to pre-allocate disk space. The capture files are of fixed size (e.g. 30 GB), so if only 29 GB of data is recorded, the remaining 1 GB of data may be from previous capture runs. The solution is to use zero fill during pre-allocate. This is easy to do, and should be done for production, but takes a long time. Best time to do it is during module conditioning.

fallocate() use almost doubles disk throughput.