Filesystem corruption when regular file is used as backing swap

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Create a swap file
2. Enable compcache with backing swap using the swap file
3. Swapon ramzswap, error: invalid argument

What version of the product are you using? On what operating system?
Compcache 0.6.  Android OS running on Tmobile G1. 

Please provide any additional information below.
Using the exact same steps with a linux-swap partition will work.  
Compcache gives no errors, the swapon command is the only time an error is 
seen. 

Original issue reported on code.google.com by MikeTayl...@gmail.com on 29 Aug 2009 at 8:17

[GoogleCodeExporter]

Please provide kernel log after you get 'invalid argument' error.

Original comment by nitingupta910@gmail.com on 29 Aug 2009 at 11:57

[GoogleCodeExporter]

Once you get this error, try swapon for this file directly (without ramzswap).
Collect kernel log after both these things are done.

Original comment by nitingupta910@gmail.com on 29 Aug 2009 at 12:02

[GoogleCodeExporter]

When I swapon /dev/block/ramzswap0, this is in the kernel log: 

<4>[ 2047.344390] Swap area shorter than signature indicates

When I swapon /system/sd/swap.file, this is in the kernel log: 

<6>[ 2174.034118] Adding 32348k swap on /system/sd/swap.file.  Priority:-1 
extents:136 across:66844k SS

I've tried setting the disksize lower than the swap device but it still gives 
me the 
same error. 

Original comment by MikeTayl...@gmail.com on 29 Aug 2009 at 9:56

[GoogleCodeExporter]

Fix committed. Can you please test by directly checking out from repository:

hg clone http://compcache.googlecode.com/hg/ compcache

Thanks.

Original comment by nitingupta910@gmail.com on 31 Aug 2009 at 2:09

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Still testing the fixes...

Original comment by MikeTayl...@gmail.com on 31 Aug 2009 at 11:45

[GoogleCodeExporter]

Ok, so backing swap now works with a swap file but the swap file keeps getting 
corrupted.  Every two or three reboots I get this error message in the kernel 
log: 

<6>[   91.641601] ramzswap: NUM_DEVICES not specified. Using default: 1
<6>[   91.651336] ramzswap: Creating 1 devices ...
<6>[   91.698608] ramzswap: Backing swap set to /system/sd/swap.file
<6>[   91.723846] ramzswap: Memory limit set to 18432 KB
<6>[   91.745452] ramzswap: Using backing swap device: /system/sd/swap.file
<3>[   91.787445] ramzswap: Backing swapfile has holes
<3>[   91.793029] ramzswap: Initialization failed: err=-22
<3>[   91.854980] Unable to find swap-space signature

If I delete the swap file and recreate it I don't have a problem.  I'm not sure 
if 
it is a hardware problem on my end or what.  I haven't had this problem in the 
past 
using a swap partition.  I ran e2fsck and it found a lot of bad blocks and 
repaired 
them.  I'm testing it again to see if the errors come back.  Could backing swap 
be 
causing these errors?  

Original comment by MikeTayl...@gmail.com on 1 Sep 2009 at 5:22

[GoogleCodeExporter]

Do you get any problem if you using this swap file directly i.e. without 
ramzswap?

Original comment by nitingupta910@gmail.com on 1 Sep 2009 at 7:57

[GoogleCodeExporter]

I've been testing with only linux swap file for about 2 hours now without a 
single error. I couldn't go 20 
minutes without seeing problems when I had compcache with backing swap turned 
on. I have 2 people 
who will be testing this tomorrow to confirm my results. 

Original comment by MikeTayl...@gmail.com on 1 Sep 2009 at 9:39

[GoogleCodeExporter]

I've had another person confirm this issue.  With backing swap on using paging 
file 
another phone had the exact same problem.  The data was corrupted on both 
phones for 
the entire partition, not just the swap.file.  

I've been using linux swap with the same paging file since last night without a 
single problem.  

Original comment by MikeTayl...@gmail.com on 1 Sep 2009 at 6:16

[GoogleCodeExporter]

It looks like this issue actually damaged my sd card.  I had more problems 
today and 
I finally had to repartition the card.  When I did that, there were blocks that 
couldn't be used.  

Original comment by MikeTayl...@gmail.com on 2 Sep 2009 at 6:29

[GoogleCodeExporter]

I am looking into this (using mtdram to simulate SD cards).

Original comment by nitingupta910@gmail.com on 2 Sep 2009 at 6:47

[GoogleCodeExporter]

I thought I would add this.  I don't know if it is helpful or not.  I was 
trying to 
copy data off of the card so that I could setup on another sdcard that I had.  
When 
I tried copying it failed in the middle of the operation and gave me an I/O 
error.  
I grabbed the kernal log during the failure: 

Original comment by MikeTayl...@gmail.com on 2 Sep 2009 at 7:08

Attachments:

• Kernel.txt

[GoogleCodeExporter]

Have you found anything interesting on this yet? 

Original comment by MikeTayl...@gmail.com on 4 Sep 2009 at 4:44

[GoogleCodeExporter]

No, not yet. I'm trying to setup qemu arm emulator.
For now, I verified this case on x64 and everything was found to be correct. 
ARM has
always been trouble maker!

Original comment by nitingupta910@gmail.com on 4 Sep 2009 at 2:42

[GoogleCodeExporter]

Maybe you can use the android emulator. Compcache works under there. should be 
much 
simpler.

Check this: http://forum.xda-developers.com/showpost.php?p=4022567&postcount=1

Original comment by pedro.m....@gmail.com on 4 Sep 2009 at 3:00

[GoogleCodeExporter]

Thanks for the link. I downloaded the android SDK and it really seems to be much
easier to setup!

Before I go ahead preparing custom image with compcache built-in, can you please
verify if compcache with backing swap file shows this corruption problem even 
on this
emulator? It will be great if this happens on emulator also -- in this case I 
will go
ahead preparing this setup taking more help from you :)

Thanks,
Nitin

Original comment by nitingupta910@gmail.com on 5 Sep 2009 at 9:52

[GoogleCodeExporter]

I tried setting it up with the emulator.  There is no way to setup swapping on 
an 
ext partition with the emulator.  The original phone was not designed to use an 
ext 
partition or a linux swap partition.  I've tried everything I can think of and 
I 
can't get it to work.  I don't think it is supported by the emulator.  

Original comment by MikeTayl...@gmail.com on 9 Sep 2009 at 7:50

[GoogleCodeExporter]

I tried it too. I couldn't even get rzscontrol utility to work on emulator.

I will again try using qemu-arm. BTW, on which filesystem did you create swap 
file
when you encountered this problem. Also, what was flash model name (maybe 
mtdram can
emulate it)?

Original comment by nitingupta910@gmail.com on 9 Sep 2009 at 1:50

[GoogleCodeExporter]

I used ext3.  

Original comment by MikeTayl...@gmail.com on 10 Sep 2009 at 12:43

[GoogleCodeExporter]

Ok, I tried it on qemu-arm but could not repro this issue. Swap file was 
created on
ext3 partition which was then used as backing swap. Good amount of R/W happened 
on
backing swap (see screenshot attached) but no filesystem corruption was 
observed.

Now, I see that ramzswap is not checking if backing swap has any bad blocks. So,
maybe we are writing over bad blocks which can have undefined behavior. I will 
add
this check and maybe the problem will disappear. On disks we really never get 
any bad
blocks, so its difficult to catch these bugs.

I will upload a test version soon. If possible, please try and see if it works 
(Oh!
but then, there is danger of losing another flash).

Original comment by nitingupta910@gmail.com on 10 Sep 2009 at 8:38

Attachments:

• qemu_arm_rzs.png

[GoogleCodeExporter]

Did you get the test version uploaded? 

Original comment by MikeTayl...@gmail.com on 13 Sep 2009 at 7:14

[GoogleCodeExporter]

I will upload in next 2-3 days. Found some other side issues too.

Original comment by nitingupta910@gmail.com on 14 Sep 2009 at 6:54

[GoogleCodeExporter]

Please try patch attached. Its against kernel 2.6.31 (pulled today). If you are
applying these patches to some kernel and get any conflicts, please let me know.

Also, is attached version of rzscontrol utility to be used with this patch -- 
do not
use rzscontrol included with compcache-0.6 if you are using this patch.

When you create swap file, use '-c' option too:
{{{
mkswap -c swap.dd
}}}
If it reports any bad blocks, then ramzswap will not allow using this swap file
(which is what the 'fix' is). Without using this option, ramzswap will not be 
aware
of any bad blocks and this issue might occur again.

Original comment by nitingupta910@gmail.com on 18 Sep 2009 at 1:56

Attachments:

• patch_ramzswap_2.6.31.diff
• rzscontrol_main.tar.gz

[GoogleCodeExporter]

Sorry, I have been very busy and this turned into a lower priority issue for 
me.  I 
will be testing this soon.  I'll update when I know more. 

Original comment by MikeTayl...@gmail.com on 6 Nov 2009 at 10:36

[GoogleCodeExporter]

I applied the patch to 2.6.29.6 kernel and handmerged what conflicts arose. 
There is 
one areas where the kernel sources differ /drivers/staging/Makefile the patch 
wants 
to add 

obj-$(CONFIG_RAMZSWAP)       += ramzswap/

after

obj-$(CONFIG_COWLOOP)       += cowloop/

which doesn't exist. I threw it in as the last line. 

Nonetheless, I am unable to initiate compcache backing swap. I compiled the 
modules against my kernel and include them as I usually do. 

The error I get when running
sh /system/bin/swap

is 

link failed File exists
link failed File exists
link failed File exists
link failed File exists
swapoff: /dev/block/ramzswap0: Invalid argument
rm failed for -fr; Read-only file system
/system/bin/swap: rzscontrol: permission denied
rzscontrol /dev/block/ramzswap0 --memlimit_kb=61440 --
backing_swap=/dev/block/mmcblk0p3 --initvm.swappiness = 40
swapon: /dev/block/ramzswap0: Invalid argument

Original comment by ccyrow...@gmail.com on 8 Nov 2009 at 6:40

[GoogleCodeExporter]

Nitin, do you have any ideas on this? 

Original comment by MikeTayl...@gmail.com on 11 Nov 2009 at 11:00

[GoogleCodeExporter]

> Nitin, do you have any ideas on this? 

No, I don't have any idea about those "link failed File exists" errors.

Original comment by nitingupta910@gmail.com on 12 Nov 2009 at 1:22

[GoogleCodeExporter]

sorry man. 

After patching the kernel

selecting ramzswap in the the config

and building the kernel and modules etc and boot the ROM and load the modules 
then attempt to do this: 
 rzscontrol /dev/ramzswap1 --backing_swap=/dev/block/mmcblk0p3

I get rzscontrol: 1 Syntax error: "(" unexpected

I'm using the rzscontrol I compiled form your source. 

Thanks!

Original comment by ccyrow...@gmail.com on 13 Nov 2009 at 5:11

[GoogleCodeExporter]

Nitin, can you help us out with the syntax error?  See the post above mine. 

Original comment by MikeTayl...@gmail.com on 20 Nov 2009 at 3:42

[GoogleCodeExporter]

> Nitin, can you help us out with the syntax error?  See the post above mine. 

I could not repro the issue:

$ sudo sub-projects/rzscontrol/rzscontrol /dev/ramzswap0
--backing_swap=/dev/mapper/VolGroup-lv_swap
$ sudo sub-projects/rzscontrol/rzscontrol /dev/ramzswap0 --init
$ sudo sub-projects/rzscontrol/rzscontrol /dev/ramzswap0 --stats
BackingSwap:    /dev/mapper/VolGroup-lv_swap
DiskSize:        4194304 kB
MemLimit:         305468 kB
NumReads:              0
NumWrites:             0
<snip>

This sytax error is never printed by rzscontrol directly. So, looks like you are
using some wrapper scripts around rzscontrol which contains some scripting 
error.

Original comment by nitingupta910@gmail.com on 20 Nov 2009 at 4:57

[GoogleCodeExporter]

ccyrowski, MikeTaylor00: is it working for you now?

Original comment by nitingupta910@gmail.com on 12 Dec 2009 at 1:57

[GoogleCodeExporter]

We are tracking two issues here:
1- problem when file is used as backing swap
2- some syntax error (see comment #29)

These issues seems to be completely unrelated, so separate issue should be 
opened for
(2).

Anyway, I'm closing it for now. Please repoen if you hit this again.

Original comment by nitingupta910@gmail.com on 12 Jan 2010 at 1:42

• Changed state: Done

[GoogleCodeExporter]

I ran into a problem when using compcache with a swapfile as backing swap. At 
some 
point my filesystem got corrupted and ramzswap reported holes in the swapfile.
The hardware is a x86 system: fanless thin client style, Via C3 processor, 512 
Mbytes 
of RAM. The kernel is 2.6.33 from kernel.org. Compcache was downloaded at the 
time of 
2.6.33 release. The 'swap free notify' patch is applied. The filesystem is 
ReiserFS, I 
had to run 'reiserfsck --rebuild-tree' to fix it.
I have no idea what was happening prior to filesystem corruption: the system 
log 
doesn't say anything interesting. The system just froze at some point.

Original comment by m.kles...@gmail.com on 14 Mar 2010 at 4:46

[GoogleCodeExporter]

I'm now looking into this.
Reopening the issue.

Original comment by nitingupta910@gmail.com on 14 Mar 2010 at 5:11

• Changed state: Started

[GoogleCodeExporter]

Original comment by nitingupta910@gmail.com on 14 Mar 2010 at 5:13

• Added labels: Priority-Critical
• Removed labels: Priority-Medium

[GoogleCodeExporter]

This issue seems to be due to incorrect handling of regular backing swap file by
ramzswap. Now we take inode mutex before checking swap extents and also mark 
backing
swap file as 'S_SWAPFILE' which prevents movement for file fragments on disk 
(say
during runtime defrag) or any truncation.

Still, this should be thoroughly tested within a VM to be safe. If you do any
testing, please let me know.

Original comment by nitingupta910@gmail.com on 14 Mar 2010 at 7:45

• Changed title: Filesystem corruption when regular file is used as backing swap
• Changed state: Fixed

[GoogleCodeExporter]

Unfortunately, I don't have time for testing.
I abandoned backing swap and started using two swaps with different priorities: 
ramzswap and a swapfile. By the way, wouldn't a setup like that be essentially 
equivalent to ramzswap with backing swap? Doesn't it make backing swap an 
unnecessary 
complication?

Original comment by m.kles...@gmail.com on 15 Mar 2010 at 9:26

[GoogleCodeExporter]

When ramzswap encounters an incompressible page, it can forward it to backing 
swap if
its present. This is not possible to do (or at least very difficult) if these 
swaps
were added separately as usual.

[NOT YET IMPLEMENTED[ Another advantage of backing swap setup is during 
shrinking and
defragmentation of (compressed) memory. Since backing swap is under control of
ramzswap, we can easily send out unused (LRU) pages in compressed memory to 
disk --
in compressed form directly.

Original comment by nitingupta910@gmail.com on 15 Mar 2010 at 9:47