NTFS > EXT4 target busy issue

[asanghc]

Ubuntu 17.10 on a remote 10 y/o asus laptop fstransform 0.9.3 /dev/SDB is and 1TB external USB 3.0 spinning HDD

I launched these two commands in two tmux terminals about a week ago: fstransform --old-file-system=ntfs --force-untested-file-systems /dev/sdb1 ext4 (the command proposed to monitor the loop file size - it never exceeded 60%) the conversion went to 100% and looked like it would succeed yesterday but then it threw these errors: 02:55:14 fstransform: disconnected loop device '/dev/loop1' from file '/tmp/fstransform.mount.7857/.fstransform.loop.8107' 02:55:14 fstransform: unmounting device '/dev/sdb1' before disk check 02:55:14 umount: /tmp/fstransform.mount.7857: target is busy.

02:55:14 ERROR! fstransform: command '/bin/umount /dev/sdb1' failed (exit status 32) this is potentially a problem. you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/bin/umount /dev/sdb1'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: CONTINUE

12:47:23 fstransform: running '/sbin/fsck' (disk check) on device '/dev/sdb1' 12:47:23 fsck: fsck from util-linux 2.31.1 12:47:23 fstransform: mounting again device '/dev/sdb1' read-only 12:47:24 mount: ntfs-3g-mount: mount failed: Device or resource busy

12:47:24 ERROR! fstransform: command '/bin/mount /dev/sdb1 /tmp/fstransform.mount.7857 -o ro' failed (exit status 21) this is potentially a problem. you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/bin/mount /dev/sdb1 /tmp/fstransform.mount.7857 -o ro'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: CONTINUE

16:22:09 fstransform: launching '/usr/sbin/fsremap' in simulated mode 16:22:09 fsremap: starting job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1' 16:22:09 fsremap: if this job is interrupted, for example by a power failure, 16:22:09 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -n -q --resume-job=1 -- /dev/sdb1 16:22:09 fsremap: ERROR: error opening loop-file '/tmp/fstransform.mount.7857/.fstransform.loop.8107': No such file or directory

16:22:09 ERROR! fstransform: command '/usr/sbin/fsremap -n -q -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107' failed (exit status 254) this is potentially a problem. you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/usr/sbin/fsremap -n -q -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: CONTINUE

16:39:36 fstransform: launching '/usr/sbin/fsremap' in REAL mode to perform in-place remapping. 16:39:36 fsremap: starting job 2, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.2' 16:39:36 fsremap: if this job is interrupted, for example by a power failure, 16:39:36 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -q --resume-job=2 -- /dev/sdb1 16:39:36 fsremap: ERROR: error opening loop-file '/tmp/fstransform.mount.7857/.fstransform.loop.8107': Transport endpoint is not connected

16:39:36 ERROR! fstransform: command '/usr/sbin/fsremap -q --cmd-umount=/bin/umount -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107' failed (exit status 149) this is potentially a problem. you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/usr/sbin/fsremap -q --cmd-umount=/bin/umount -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER:"

I'll admit I'm lost but these are the steps I attempted to fix it: -killed anything I thought could be keeping /dev/sdb1 busy -after running out of ideas, forced the umount with -l (lazy) -problem still wasn't fixed with that -I ended up force remounting the drive with -o force (probably a stupid idea) -commented out entries in fstab that mount that drive -rebooted (another stupid idea probably) -now any attempts to sudo fsremap -q --resume-job=1|2|3 /dev/sdb1 fail with ERROR: error opening loop-file '(null)': Bad address I'm asking for help here now to try to avoid making things worse (maybe too late?)

What happened and can you halp?!

[cosmos72]

Hello @asanghc,

You chose a very unlucky combination:

• NTFS support is experimental and untested - which means dangerous - in fact you added the option --force-untested-file-systems because by default fstransform would refuse to convert it.

• There is a known issue with fstransform on Ubuntu: by default, almost all versions of Ubuntu open a file manager window on each newly available partition, including loop files. This prevents fstransform from unmounting the partition to perform the fsremap - exactly the error you reported. And since fsremap is never started properly, --resume-job does not find anything to resume.

Despite these issues, and depending on the contents of your /dev/sdb1, it may still be possible to complete the conversion. Here are the steps:

1. if you have a Windows installation available, perform a disk check on /dev/sdb1 from Windows.
2. on Ubuntu, execute as root the commands: mkdir /mnt; mount /dev/sdb1 /mnt; ls -al /mnt
3. if the output of ls command above does not show any file, you are in trouble.
4. if instead it shows the files and directories that were initially on the disk before the conversion, but no file name .fstransform.loop.* (unlikely), then you still have an NTFS volume with your data.
5. if instead there is exactly one file, hopefully named .fstransform.loop.8107, keep reading:
6. perform a filesystem check on the file /mnt/.fstransform.loop.8107 - or whatever is its name: e2fsck -C0 -f /mnt/.fstransform.loop.8107
7. if the filesystem is clean or gets successfully repaired, keep reading. Otherwise you are in trouble.
8. close any file manager window that Ubuntu opened.
9. execute umount /mnt
10. if you have a recent fstransform that includes the command fsmount_kernel, execute the command fsmount_kernel -o ro -t ntfs /dev/sdb1 /mnt. Otherwise execute the "regular" command mount -o ro /dev/sdb1 /mnt. Note: fsmount_kernel is strongly preferred because fsremap will be faster with it.
11. again, close any file manager window that Ubuntu opened.
12. run the command that was printed at the initial error, updated for the fact that /dev/sdb1 is now mounted at /mnt. the command is: fsremap --cmd-umount=/bin/umount -- /dev/sdb1 /mnt/.fstransform.loop.8107 (or whatever is the name of the file you found at step 6)
13. be patient. The conversion of 1TB will likely need 20-30 hours.
14. if you get an error similar to umount: /mnt: target is busy, do not type anything in the terminal where fsremap is running. Instead, locate all file manager windows that Ubuntu opened and close them. Then, in a different terminal, try again umount /mnt and, if it still fails, run fuser -vm /mnt to find which processes are using /mnt and kill them. Once you finally unmounted /mnt, go back to the terminal where fremap is running, and type uppercase CONTINUE then press enter. Conversion should resume.

[cosmos72]

Let me know the outcome :)

[asanghc]

Thank you for the quick reply. I'll try your advise this weekend. (never enough time) This is a remote system which I can only access via SSH. This laptop has no monitor (destroyed in accident) and runs only ubuntu server with no X or GUI. I'm not sure what could have cause the "target busy" issue. I'm not suggesting that your awesome tool might be at fault, I just don't know exactly how I screwed this up. Cheers

[asanghc]

Hi again, since #5 is true, I did #6 successfully and since I have version 0.9.3, it doesn't look like I can use the quicker fsmount_kernel mode. Am I making sense so far? Above which versions is fsmount_kernel available? Must I compile it or is there a newer version available in .deb form or through a ppa? Thank you very much for your assistance.

[cosmos72]

Good news :) Yes, it makes sense.

fsmount_kernel is "just" a speedup. There is no updated .deb package containing it, you would have to compile it from sources: you can download the latest fstransform from https://github.com/cosmos72/fstransform/archive/master.tar.gz, uncompress (tar -xzf master.tar.gz), configure (cd fstransform; ./configure), compile (make) and install it (sudo make install). If you are not comfortable with compiling programs from source, you can do without it and follow the procedure described above anyway.

[asanghc]

I compiled/installed per your instructions without problems or errors. everything goes well for steps 5 through 12 but not proceeding beyond that.

sudo fsremap --cmd-umount=/bin/umount -- /dev/sdb1 /mnt/.fstransform.loop.8107 setting log level to INFO
fsremap: starting job 6, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.6' if this job is interrupted, for example by a power failure, you CAN RESUME it with: fsremap -q --resume-job=6 -- /dev/sdb1 device length is 931.48 gigabytes analyzing device '/dev/sdb1', this may take some minutes ...

it remains stuck at this step indefinitely

[cosmos72]

I guess you executed fsremap with /dev/sdb1 already mounted, right?

In such case, whatever kernel driver is in use for NTFS will be employed for the analysis phase - usually, FUSE ntfs-3g, which as I wrote above, is extremely slow for the analysis.

The faster option I mentioned is (after killing the running fsremap stuck in the analysis phase), to execute as root:

umount /dev/sdb1
fsmount_kernel -o ro -t ntfs /dev/sdb1 /mnt
fsremap --cmd-umount=/bin/umount -- /dev/sdb1 /mnt/.fstransform.loop.8107

The umount command executed by fsremap after the analysis may fail if Ubuntu shows a file manager opened on /mnt (which it usually does) - in such case, close all file managers, umount /mnt manually, then type CONTINUE at fsremap prompt

[asanghc]

Is there a way to monitor progress or at least get confirmation that SOMETHING is happening? This is using the faster fsmount_kernel option. I've tried both mount options and different versions of the fstransform tool and waited over a week each time.

This is a headless system without a GUI and I'm doing everything via SSH and tmux. The original "read only" issue is no longer an issue and there is never a prompt to CONTINUE

Would you be interested in seeing the contents of any of these files? /var/tmp/fstransform$ ls total 72K 4.0K drwxr-xr-x 9 root root 4.0K Jul 23 15:25 . 4.0K drwxrwxrwt 5 root root 4.0K Jul 22 07:49 .. 4.0K drwxr-xr-x 2 root root 4.0K Apr 22 16:22 fsremap.job.1 4.0K drwxr-xr-x 2 root root 4.0K Apr 22 16:39 fsremap.job.2 4.0K drwxr-xr-x 2 root root 4.0K Apr 22 16:40 fsremap.job.3 4.0K drwxr-xr-x 2 root root 4.0K Apr 29 22:21 fsremap.job.4 4.0K drwxr-xr-x 2 root root 4.0K Apr 30 18:32 fsremap.job.5 4.0K drwxr-xr-x 2 root root 4.0K Jul 23 12:12 fsremap.job.6 4.0K drwxr-xr-x 2 root root 4.0K Jul 23 15:25 fsremap.job.7 4.0K -rw-r--r-- 1 root root 1022 Apr 18 11:06 fstransform.log.7761 4.0K -rw-r--r-- 1 root root 3.4K Apr 18 11:08 fstransform.log.7857 28K -rw-r--r-- 1 root root 23K Apr 22 16:39 fstransform.log.8107

[asanghc]

Does this explain anything?

2018-07-23 15:31:16 DEBUG [io/extent_posix.ff_linux_fiemap(189)] ioctl(6, FIEMAP, extents[1024]) failed (Operation not supported), falling back on ioctl(FIBMAP) ...

[cosmos72]

You already listed the most relevant line, no need to send me the logs.

fsremap on NTFS takes ages to analyze the device because ioctl(FS_IOC_FIEMAP) is not supported, and ioctl(FIBMAP) is terribly slow on FUSE ntfs-3g. Using fsmount_kernel speeds up a little ioctl(FIBMAP), as the kernel implementation is merely slow instead of terribly slow.

To monitor the progress of this analysis phase, you can try attaching strace -v -T to the running fsremap and post here a small extract (a few lines showing some calls to ioctl(FIBMAP) are enough)

I understand your frustration: you are experiencing exactly the issues I met too, and that convinced me not to support NTFS (at least not yet). This also the reason why fstransform refuses to convert NTFS file systems unless you force it with --force-untested-file-systems

[cosmos72]

Update: in case I did not mention it yet, since you are stuck at step #12, another option is to have a 1TB free partition on the same system (say /dev/sdx1) and perform as root direct copy of the huge loop file:

dd bs=10M if=/mnt/.fstransform.loop.8107 of=/dev/sdx1

If you have enough free space, this is the fastest solution

[xulianh]

Hello, I arrived here from the topic trying to convert NTFS to EXT4. Initially the problem was fsconvert ERROR: error opening loop-file '(null)': Bad address and following the guide I got stuck at point 12. Sorry in advance for reasking in a such old thread. The error shown in console is

sudo /usr/sbin/fsremap -n -q --resume-job=1 --cmd-umount=/bin/umount -- /mnt/sdf2/.fstransform.loop.10941 fsremap: resuming job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1' ERROR: failed device fstat('/mnt/sdf2/.fstransform.loop.10941'), re-run with option '-f' if you want to continue anyway (AT YOUR OWN RISK): Block device required

adding -f:

sudo /usr/sbin/fsremap -n -q -f --resume-job=1 --cmd-umount=/bin/umount -- /mnt/sdf2/.fstransform.loop.10941 fsremap: resuming job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1' WARN: WARNING: failed device fstat('/mnt/sdf2/.fstransform.loop.10941'), continuing AT YOUR OWN RISK due to '-f': Block device required ERROR: error in device ioctl('/mnt/sdf2/.fstransform.loop.10941', BLKGETSIZE64): Invalid argument

Previously I had a restart and tried to rerun from the point console stated that was the command shown below, but as the loop file wasn't given in the options it didn't run

fsremap: you CAN RESUME it with: /usr/sbin/fsremap -n -q --resume-job=1 -- /dev/sdf2

Any help please?

[cosmos72]

Hi xulianh,

Since you are writing into this issue, I guess you alre already aware that NTFS support is experimental and untested - the disclaimer "use fstransform at your own risk" is even more valid in this case.

Having said that, the last argument of fsremap --resume-job=N ... must be the partition to work on, not the loop file.

I.e.:

fstrasform sudo /usr/sbin/fsremap -n -q -f --resume-job=1 --cmd-umount=/bin/umount -- /dev/sdf2

That's why you get the error

ioctl('/mnt/sdf2/.fstransform.loop.10941', BLKGETSIZE64): Invalid argument

as the ioctl BLKGETSIZE64 is supported only by block devices, not by regular files.

Please try again with the fixed command line and let me know if it helps

[xulianh]

Hi @cosmos72 yes, I knew of course this option was experimental, I'm playing with not critical data, although if possible I'd prefer to recover it.

with this command I get the error as follows, that's the reason I wrote there the location of the loop file instead of the device being working out:

sudo /usr/sbin/fsremap -n -q -f --resume-job=1 --cmd-umount=/bin/umount -- /dev/sdf2 
fsremap: resuming job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1'
ERROR: error opening loop-file '(null)': Bad address

[cosmos72]

Strange, but even if we somehow solve this last error it will not help in your case, because converting NTFS file systems >= 1TB is not expected to work:

it is expected to fail during the fsremap phase because Linux kernel does not support ioctl(FS_IOC_FIEMAP) on NTFS and ioctl(FIBMAP) only supports "small" file systems:

exact limit is fibmap_block_size * (2^31 - 1), where fibmap_block_size may differ from file system block size - so the limit is somewhere between 1TB (fibmap_block_size = 512) and 8TB (fibmap_block_size = 4096).

Since you wrote in #27 that your NTFS file system is 3TB, I can only suggest you to run

e2fsck -C0 -f  /mnt/sdf2/.fstransform.loop.10941

and, if the file system check succeeds, copy that file to a spare 3TB raw partition, i.e. something like

dd bs=16M if=/mnt/sdf2/.fstransform.loop.10941 of=/dev/sdXY

where /dev/sdXY is a spare partition with the same size (or larger than) the loop file.

[franklin76]

analyzing device '/dev/sdb1', this may take some minutes ... ERROR: ff_posix_fibmap(): error, dev_block_count = 3906820096, file_block_count = 3906820096 overflow type (int): File too large ERROR: failed to list file blocks with ioctl(FS_IOC_FIEMAP): Operation not supported

Following all the steps, I received that error. Is there anything I can do to recover my data?

[cosmos72]

There should be only a single huge file inside the NTFS filesystem on /dev/sdb1

You can try to mount the NTFS filesystem then mount the file inside it too.

[franklin76]

Thanks a lot. I have mounted the bg loop file and recovered all my data.