search

cosmos72 / fstransform

tool for in-place filesystem conversion (for example from jfs/xfs/reiser to ext2/ext3/ext4) without backup
GNU General Public License v2.0
278 stars 28 forks source link

NTFS support? [was: More parallelization?] #15

Open cornytrace opened 7 years ago

cornytrace commented 7 years ago

Hi,

When transforming an ntfs partition to ext2, a lot of the time only one core hits 100% while the rest are basically idle, a sign that the program is cpu bound due to a lack of parallelization. Is there something that can be done about this? Or is it the underlying tools that are single threaded?

Edit: I also noticed that the program remounts ntfs without the big_writes parameter, which significantly slows down the procedure. I have a 450GB ntfs partition with an estimated completion time of 1 day and 20 hours.

cosmos72 commented 7 years ago

Hello,

I am quite sure NTFS support is not enabled by default, i.e. fstransform should refuse to convert NTFS filesystems unless you specify one of the --force-* options. Can you confirm that from your experience?

There are several reasons for the stated "lack of support". The main one is exactly what you wrote: it's extremely slow and cpu-bound, due to the fuse based implementation of ntfs-3g. Option big_writes speeds up a bit fstransform usage pattern, but it's still very slow.

I tried some other tricks, including using kernel-based read-only NTFS driver for some of fstransform steps, but the result is not yet production-grade. Probably non even beta.

If you have more suggestions to improve NTFS support you're very welcome, as I consider NTFS support really useful - maybe even a game changer

cornytrace commented 6 years ago

Yeah, I used the -force option, its a disk with non critical data so I didn't really care if something went wrong. I looked deeper into it and yeah, it was basically ntfs-3g being slow. One feature I could use though is a resume function for fstransform. The strange thing is that it detects you have old loop files, but it doesn't offer to use them. One would think that codepath could be used to make a relatively easy resume function, no?

cosmos72 commented 6 years ago

About adding resume to fstransform: it's an idea I considered several times, but never managed to fully implement. The hard part is already done and tested: fsremap fully supports resuming interrupted jobs. Given that, adding resume to fsmove and fstransform should be quite straightforward. Volunteers welcome :)

About ntfs-3g being slow: it's extremely slow when overwriting parts of a huge file - exactly what fstransform does, since it writes inside a sparse, loop-mounted file as large as the partition. I am positively confident some ntfs-3g profiling and optimization, and maybe adding some cache, would greatly speed up this specific use case. Another very useful speedup would be to extend fuse and ntfs-3g to support ioctl(FS_IOC_FIEMAP), because the fallback ioctl(FIBMAP) is both extremely slow on fuse filesystems and limited by design to 2TB. There was a thread on fuse-devel proposing this, but I never saw any patch landing on libfuse or kernel fuse driver. Again, volunteers welcome :)