Performance: creating 200Gb tomb is taking more than 15hours

[demaniak]

SO, perhaps more a question that an issue.

How long is tomb supposed to take to create a new, 200Gb tomb file?

At time of writing it has already been running for 15 hours, and the tomb file is 138Gb.

Now, admittedly, the machine in question is a tiny little Asus EEE PC netbook (probably one of the last produced), and the file is being created on an external USB drive....

Looking at top, dd and mount.ntfs seems to be the CPU hogs. So...any one got some pointers to stats of some sort for comparison?

[hellekin]

Given that you're creating a tomb from a very limited hardware platform, there's nothing strange that it takes so much time.

Note that the idea behind Tomb is to make portable encrypted elements. 200GB seems a lot to me. I wouldn't use it for more than what a USB stick can hold, as moving it from one device to the next (e.g., for backups) would take way too much time. In that case I'd encrypt an entire partition of the device, or the whole device at once.

If you lookup "secure hdd wipe benchmark luks" or something approaching, you should find some comparisons. But it mostly depends on the entropy available on your system, and given that you're already 17 hours into the process, you should be able to estimate what's left to go :)

[demaniak]

@hellekin ahright then, fair enough. The external drive in question is not mine, but I needed somewhere to stash backups while prepping for an OS re-install. SO I figured "hey lets try do to this securely!"

But ok, this duck ain't gonna fly, so I abandoned that idea. Straight cleartext copy now.

Tomb is still a great tool, so will definitely continue using it...just on a smaller scale ;P

Thanks for clearing things up!

[hellekin]

You were only 2 hours away!

[boyska]

If I'm not wrong, tomb use /dev/random as a source for random data. This is basically wrong. http://www.2uo.de/myths-about-urandom/ http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ urandom should be the default, and maybe (but why?) /dev/random should be given as an option.

Even in this case, it is pretty obvious that writing 200GB will never be "fast". But on my computer there is a 7x increase anyway.

[linux-modder]

random or urandom can be used in this case random is likely to run out of entropy faster especially on 200G unless actively typing or some such...

Corey W Sheldon Freelance IT Consultant, Multi-Discipline Tutor Ameridea LLC, Co-Founder, CTO (p) +1 (310) 909-7672

"One must never underestimate the power of boredom...from which

creativity and laziness are borne, which can spark great works of chaos and genius."

Find Me on any of the sites I teach /frequent:

https://gist.github.com/linux-modder/ac5dc6fa211315c633c9

Tox: corey84@toxme.se 9357BC6A5944A08AFC7D1EFFD61F6A73B9EABF8B2FB84ACF1DAC9A1A4D0A4705FFCCD0E5499B PGP: 718BF597 http://pgp.mit.edu/pks/lookup?op=get&search=0xE958C5D6718BF597 FP: 2930

99EB 083D D332 0752 88C4 E958 C5D6 718B F597

On Fri, Jul 17, 2015 at 6:30 PM, BoySka notifications@github.com wrote:

If I'm not wrong, tomb use /dev/random as a source for random data. This is basically wrong. http://www.2uo.de/myths-about-urandom/ http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ urandom should be the default, and maybe (but why?) /dev/random should be given as an option.

Even in this case, it is pretty obvious that writing 200GB will never be "fast". But on my computer there is a 7x increase anyway.

— Reply to this email directly or view it on GitHub https://github.com/dyne/Tomb/issues/210#issuecomment-122436270.

[boyska]

of course random could be used. But if it is so slow than creating tombs is considered unpractical and so encryption is not performed at all, this is a bug.

[linux-modder]

@boyska , not at all secure tombs, like any form of crypto when done right take time, fresh from store drives are churning nonsense randoms to the disk for at least 36 before an encrypted OS goes on. Using the default 512mb BS anything over 10G takes a noticeable time...

[boyska]

there is nothing specific in your comment. You says that "it must take time", which could be sometimes right, but is it a contribute to the issue?

[jaromil]

@boyska you are wrong, Tomb uses /dev/urandom to create (dig) unformatted tombs. This should be so as it makes the process faster.

Perhaps known to most participants to the thread, but good to state anyway: we use a reasonably fast random source to allocate space for un-formatted tomb space to avoid certain attacks based on the fact the attacker has knowledge of the contents of storage support blocks before the tomb was created...

As of increasing speed, I recommend using haveged for increasing system entropy and perhaps speed. But yes, crypto takes time and I think the fact it does is a "good sign".

Also in my TODO since long time is the adoption of an "unsafe" dig mode using fallocate. But I'm not even sure that is really needed, as one could use that directly and then lock the file (as any other existing file) with a Tomb.

Entirely at your own risk, but if you are in a hurry and don't care about such security issues then just dd from /dev/zero or perhaps use fallocate, then forge a key and use it to lock the file. There you go with a sort of shallow tomb.

[boyska]

@boyska you are wrong, Tomb uses /dev/urandom to create (dig) unformatted tombs. This should be so as it makes the process faster.

oh, that's great, I misread the code. sorry for the noise, then.

As of increasing speed, I recommend using haveged

yes, haveged is quite good in this respect. I think I'll add it to the Suggests: debian field!

But yes, crypto takes time and I think the fact it does is a "good sign".

no it's not :)

Also in my TODO since long time is the adoption of an "unsafe" dig mode using fallocate. But I'm not even sure that is really needed, as one could use that directly and then lock the file (as any other existing file) with a Tomb

I think that fallocate does not work on every filesystem. Since it is quite a common case to keep a tomb on a usb key, this might be a problem.

[linux-modder]

Gonna go out there and be the dickhead. If you are worried about time it takes to make a tomb of any size then don't use it security takes time and comes with other sacrifices /compromises only two of which is time and convenience ...

[jaromil]

@boyska good point about fallocate, did not think about compatibility across FS. So far Tomb works only with ext4, but for the future we plan at least one or two FS that have snapshot capability, not sure if fallocate would then support btrfs or zfs.

I will change this TODO item into producing documentation for cases in which a Tomb requires less security but must be created fast, then literally any existing file can be used and reformatted. Up to the user to choose what to use as a start instead of digging a proper Tomb.

[boyska]

I meant something different. VFAT could be used inside the tomb (but why?) even if fallocate is used. Similarly, btrfs, zfs, etc can be used inside a tomb. I was talking about the underlying filesystem, though. I think that fallocate works on any decent, modern filesystem (reiserfs, zfs, xfs, you name it. Never tested it but am pretty sure), which FAT is not ;)

[Narrat]

OT

So far Tomb works only with ext4, but for the future we plan at least one or two FS(...)

What about F2FS? Since there are tombs residing on Flash-mediums, offering the option for a FS that focus on those kind of mediums cannot be bad?

[jaromil]

OK yes I made some confusion, writing github issues from a sunny vacation at 40 degrees is never leading to smart insights really :tongue: Now back. In our conversation lets distinguish between:

1) HOST FS which is the filesystem used to format the storage support hosting the tomb file 2) TOMB FS which is the filesystem used internally in tomb (right now ext4 only)

my considerations:

I cannot see any valid case in which we'd want either 1) or 2) to be FAT.

FSes like F2FS or ZFS or J2FS which are addressing bit-rot issues for flash and other mediums have their bitrot features working effectively only when used as 1) HOST FS

FSes like ZFS or BTRFS which are providing snapshot and rollback capabilities have those as effective only when used as 2) TOMB FS

Looking forward to any other scenario you may propose, then we can proceed with further analysis by building a grid of features and FS for all this.

[boyska]

1) HOST FS which is the filesystem used to format the storage support hosting the tomb file 2) TOMB FS which is the filesystem used internally in tomb (right now ext4 only)

my considerations:

I cannot see any valid case in which we'd want either 1) or 2) to be FAT.

I see the case for HOST fs to be fat. While the "typical" workflow is to keep data on the computer, and the tombkey on a usb drive, I do the contrary. The clear advantage is that doing so I can always have the data in my pocket, so in any of my computers they are synced, without the need to use sshfs, "cloud" or anything else.

My tomb has not much data, so it does not contain only the tomb; actually the tomb is 200MB, while the rest is miscellanous plain files. I like it to be vfat so that I can access those data from any computer.

FSes like F2FS or ZFS or J2FS which are addressing bit-rot issues for flash and other mediums have their bitrot features working effectively only when used as 1) HOST FS

Ok, so the "tomb on a usb drive" workflow is important for you too. Well, if you are serious about your data those filesystem probably are a good choice, so this may be a recommendation for users, but not something to be supported in tomb.

boyska gpg --recv-keys 0x58289ca9

[jaromil]

OK. FAT on HOST makes sense now. Tomb does not contemplate a "typical" workflow BTW :^) its really a tool for anyone to make up easily their own. Yes the HOST FS considerations would mostly be for the documentation, they shall require no change in Tomb's code.

[kolobus]

I would suggest making parameter like "-unsafe" which will use /dev/zero instead of /dev/urandom. With the hell lot BIG RED WARNINGS saying this option is unsafe.

[jaromil]

even better than that there is a tool to allocate free blocks to a file, which still gives some sort of randomness, see fallocate

this can be used outside tomb and inatead of the dig command. Perhaps should be mentioned in the docs.