search

ntop / n2disk

Open source components and extensions for n2disk
503 stars 11 forks source link

n2disk is not archiving files respecting the dump order (-O option) #21

Closed igorribeiroduarte closed 3 years ago

igorribeiroduarte commented 4 years ago

After updating n2disk for 3.2.191128-5095_amd64 version, the index files are not being persisted anymore when using -O (archive) option

cardigliano commented 4 years ago

@igorribeiroduarte please note that index files are not archived with -O (this is the way it currently work), is this the issues?

igorribeiroduarte commented 4 years ago

@cardigliano not sure. I'm not talking about the index files not being saved on the folder specified on -O option, what I'm saying is that when using this option the index files are being created on the index folder passed on -A option and right after they are being deleted, in other words, they are not being persisted. This way, I'm not able to extract pcap files with npcapextract anymore

cardigliano commented 4 years ago

I think this is due to the -O algorithm, it is moving pcap files without respecting the recording order (and index files are removed when the corresponding pcap file is moved). This has been implemented quite some time ago, even before the index implementation, and considered obsolete as nobody was actually using it. The algorithm should be improved to respect the dump order with a fifo.

igorribeiroduarte commented 4 years ago

@cardigliano I understand, but shouldn't the pcaps be moved only after the disk is filled? If they are moved before that, will the indexes keep working? If don't, copying the pcaps instead of moving wouldn't be a better approach? I'm not sure if I'm misunderstanding the usage of -O option. I just want the pcaps to be saved on a slower disk before they are erased from the faster disks I'm using for extractions.

cardigliano commented 4 years ago

@igorribeiroduarte I agree that the behaviour should be changed to reflect what you are saying. --archive|-a is behaving in a similar way, however it just renames the files without moving them. Both have been added in past based on specific requests and should be changed into something more generic that makes more sense.

igorribeiroduarte commented 4 years ago

Right, I'll try to add a custom script as you advised on the other issue while these options are not improved. Thank you.

cardigliano commented 3 years ago

-O now supports the --disk-limit dump schema, and it moves file in the proper order