ZFS is stuck at r72 (pool version 8)

[GoogleCodeExporter]

The 119 bits were synchronized with OpenSolaris 72, and pool version 8. The 
current OpenSolaris 
release is 125 and the latest pool version is 19

http://www.opensolaris.org/os/community/zfs/version/19/

Original issue reported on code.google.com by alex.ble...@gmail.com on 24 Oct 2009 at 9:45

• Blocking: #66
• Blocked on: #29

[GoogleCodeExporter]

Here are the locations of the upstream stuff; unfortunately, ZFS does not look 
like it's available as a separate 
download from the rest of ON

http://opensolaris.org/os/project/onnv/

http://opensolaris.org/os/downloads/

http://dlc.sun.com/osol/on/downloads/nightly-bins/

Original comment by alex.ble...@gmail.com on 24 Oct 2009 at 9:59

[GoogleCodeExporter]

hg clone ssh://anon@hg.opensolaris.org/hg/onnv/onnv-gate will get the changes 
from the hg repository. It 
takes its sweet time, but hopefully, it will be possible to triangulate patches 
back from the r72 to a more 
recent build

Original comment by alex.ble...@gmail.com on 24 Oct 2009 at 11:05

[GoogleCodeExporter]

I've been going over the current code we have and looking at what it will take 
to move to a much more recent 
version and I'd like to hear what others think about using the same approach 
the FreeBSD folks used.
The paper[1] and slides[2] from the FreeBSD ZFS wiki page[3] contain roughly 
the same info and outline the 
ideas for;
- Using as much unmodified code from OpenSolaris as possible so that upstream 
updates can be easily 
applied.
- Writing wrapper code where needed to implement/translate features 
lacked/different (sunddi, mutex, etc.) 
in Mac OS X.

For now I'm trying to sync as many files from the b126[4] OpenSolaris codebase 
as possible without breaking 
the build, but most of it is rather minor changes, not the huge changes that 
will actually bring us up to a 
recent version.

Should we invest time in trying to fix issues with the current code or put more 
effort moving to a more recent 
version?

[1] http://2007.asiabsdcon.org/papers/P16-paper.pdf
[2] http://2007.asiabsdcon.org/papers/P16-slides.pdf
[3] http://wiki.freebsd.org/ZFS
[4] http://dlc.sun.com/osol/on/downloads/b126/

Original comment by jason.richard.mcneil on 27 Oct 2009 at 7:59

[GoogleCodeExporter]

I think trying to roll forward probably makes sense, not the least of which is 
that some of the bugs might even 
be addressed by future fixes in ZFS anyway.

A concern is how we go about testing it methodically though ... do we have many 
test cases that we can 
execute from upstream to test whether it works across a variety of different 
loads?

Original comment by alex.ble...@gmail.com on 27 Oct 2009 at 8:57

[GoogleCodeExporter]

We should probably try and get ztest[1] working first as that might handle much 
of the stress testing the zfs 
component.

I've been using fsx[2] to do some stress testing of the current version with 
mixed results. I ran it for about six or 
so hours last night on a file based zpool without error, but sometimes only a 
few minutes in and fsx will hang.

Can't figure out what is going wrong just yet, but I also was thinking that 
future ZFS code might address issues 
that we are having now.

Until ztest is a viable solution, I'm using a small script I wrote[3] for now. 
Not perfect, but it's better than nothing 
and manual processes for now. (At least for my needs)

[1] http://hub.opensolaris.org/bin/view/Community+Group+zfs/ztest
[2] http://fstools.macosforge.org/trac/wiki
[3] 
http://github.com/jasonrm/mac-zfs/blob/f5ec87c162f1939417b63c13ba4566cc27043260/
test/fsx-test.sh

Original comment by jason.richard.mcneil on 27 Oct 2009 at 9:24

[GoogleCodeExporter]

I've created issue 14 for the ztest requirements.

Original comment by alex.ble...@gmail.com on 27 Oct 2009 at 9:28

[GoogleCodeExporter]

Just as a note for others, http://dlc.sun.com/osol/on/downloads/b126/ and all 
the source there seems to be 
worthless (for our needs) as it is not current code, and in some cases, even 
*older* than the original Apple 
provided source… a fact which I only discovered hours after trying to work 
with it…

I'm a git kind of guy (in both ways evidently) so I'm using git-hg[1] and 
fast-export[2] to build a git repo right 
now. It's pretty simple to clone;
git-hg clone ssh://anon@hg.opensolaris.org/hg/onnv/onnv-gate opensolaris

Also, the telescoping script[3] seems to be pretty interesting as well, haven't 
managed to let the script fully build 
the resulting repo yet, but it should finish in the next few hours.

[1] http://github.com/offbytwo/git-hg
[2] http://repo.or.cz/w/fast-export.git
[3] http://github.com/roddi/TelescopeSolarisZFS

Original comment by jason.richard.mcneil on 1 Nov 2009 at 7:15

[GoogleCodeExporter]

Telescope takes an hour on my box.  However, I'm not completely sure we've 
identified
*every* part that needs to be pulled in correctly.  The filemap is pretty big.

Do note that the resulting repository from the opensolaris repo with the filemap
included is *much* smaller both in number of files and number of changes than 
the
full solaris one (as you'd hope).  That makes it a lot easier to work with.

Also note that the whole process *should* be repeatable and incremental.  I 
envision
a future where we have a git repo with the pure Solaris work and then the
mac-specific changes in a branch that's merging that in occasionally.

Original comment by dsalli...@gmail.com on 1 Nov 2009 at 9:03

[GoogleCodeExporter]

One issue with the telescope is that the number of files needed will increase 
over time. The dependencies 
included for misc. headers are there by transitive requirement from the ZFS 
tools; whilst the ZFS-specific code 
is easy enough to obtain, some of the others might not be.

The other concern is what happens after the repo has been cloned - does the 
telescope pick up subsequent 
incremental changes? What about tags?

The Apple fork appears to be based on the onnv_72 tag in the hg repository.

Original comment by alex.ble...@gmail.com on 2 Nov 2009 at 11:23

[GoogleCodeExporter]

Yes, this is why I said it's repeatable and incremental.  I just ran it again:

12 PSARC 2009/571 ZFS Deduplication Properties
11 PSARC/2008/181 Solaris Hotplug Framework
10 PSARC/2009/587 forcedirectio mount flag in nfsstat
9 6892316 channel binding in audit_remote(5) should correctly specify the
initiator/acceptor_addrtype
8 6892849 channel binding in audit_remote(5) should correctly specify the
application_data structure member
7 PSARC 2009/595 delete ssig system call trap
6 6885689 syseventd: [daemon.error] Fatal:attempting to dump core on snv_124
5 6739984 Solaris failed to do any failover for any volume with TPGS.
4 6886511 finger should sort its output by username
3 6886377 memmove() (libc.so.1) function defect
2 6895660 Successful message for FIPS 140 would be nice
1 4636888 iostat doesn't display correct output if -z and -e option are used 
together
0 PSARC/2008/252 Labeled IPsec phase 1

8 seconds for the Solaris pull, and about 5 for the hg convert, and about a 
second
for the git convert.

Original comment by dsalli...@gmail.com on 3 Nov 2009 at 12:00

[GoogleCodeExporter]

I'm a little bit concerned about the rewriting to different locations though. 
Perhaps we'd be better off 
adapting the Mac layout to match opensolaris instead? That way, we could roll 
back to onnv_72, do a 
diff against the Apple code, and then roll forwards. 

Both XCode (and Git) can handle the files in different layouts; we can use 
symlinking or just tell XCode 
where to look to find the new files. 

Any preference? I don't think I'm too wedded to the current directory 
structure. Obviously the Apple 
specific code would stay outside the onnv cache. 

Original comment by alex.ble...@gmail.com on 3 Nov 2009 at 12:06

[GoogleCodeExporter]

I don't have a preference.  I agree that the rewriting feels a bit risky.  I 
did a
reroot ( http://dustin.github.com/2009/01/06/git-reroot.html ) over onnv_72 to 
see
where it would take us and attempted to walk it forward.

The nice thing about the reroot is that it gives us (so far) the most granular 
diffs.
 The change that brought in onnv_72 looks like this:

 347 files changed, 28461 insertions(+), 57529 deletions(-)

:/ (though I'm not sure how much of that is due to bad telescope path 
rewriting...
some of it clearly is).

Original comment by dsalli...@gmail.com on 3 Nov 2009 at 12:18

[GoogleCodeExporter]

I put together a move which enables the Mac project to track the onnv structure:

http://github.com/alblue/mac-zfs/tree/reorg

There's a summary of some of the moves here:

http://wiki.github.com/alblue/mac-zfs/moves

I'm still working on getting it to compile fully (the kext doesn't at the 
moment) but it might be easier to 
track new files in at a later stage if we just track required directories.

Original comment by alex.ble...@gmail.com on 6 Nov 2009 at 1:47

[GoogleCodeExporter]

First of all thanks for your excellent work. If you are considering moving up 
to a
more recent version of ZFS it might be worth considering the very latest one 
that
introduces deduplication 
http://blogs.sun.com/jsavit/entry/deduplication_now_in_zfs.

Original comment by s.sciarr...@gmail.com on 7 Dec 2009 at 11:52

[GoogleCodeExporter]

I've merged my reorg into the master branch, and plan on rolling forward from 
the onnv_72 as I can.

As for dedup, whilst it's in the latest (and the latest is always the goal), 
we're quite a way away. I think the 
most recent is onnv_129. Not only that, but there's some questionable 
performance characteristics on a pool 
once dedup is turned on, that can't be rolled back by later turning it off 
again (without recreating the pool).

Original comment by alex.ble...@gmail.com on 31 Jan 2010 at 9:19

[GoogleCodeExporter]

I'm working on merging the changes from onnv_72 via a Git repository, and it's 
going well so far. I've got 
everything bar the last 12 files merged.

Of course, once that's done, we still need to go through the testing process 
before we can then punt up to a 
later version, but I'm marking this as assigned to me so that people know it's 
happening.

Original comment by alex.ble...@gmail.com on 20 Feb 2010 at 4:41

• Changed state: Started

[GoogleCodeExporter]

Original comment by alex.ble...@gmail.com on 20 Feb 2010 at 4:42

[GoogleCodeExporter]

OK, I've been able to move forwards somewhat in this.

http://github.com/alblue/mac-zfs/commits/maczfs_73 <-> onnv_73
http://github.com/alblue/mac-zfs/commits/maczfs_74 <-> onnv_74
http://github.com/alblue/mac-zfs/commits/maczfs_75 <-> onnv_75

At the moment, I've only checked for compilation effectiveness - there's a 
possibility that we have a runtime dependency missing or other errors. I'm 
making them available for now as a temporary environment, but will probably 
verify execution sanity before making them a part of the official master stream

Original comment by alex.ble...@gmail.com on 19 Jul 2010 at 11:21

[GoogleCodeExporter]

It would be nice to keep up with FreeBSD, as I'm using ZFS on FreeBSD and it 
would be nice to have pool portability in case I needed to import a pool from 
FreeBSD directly on my Mac.

Original comment by jeremy.m...@gmail.com on 22 Aug 2010 at 2:21

[GoogleCodeExporter]

If you want too keep compatibility with FreeBSD, you just need to use zfs pools 
that both operating systems understand.
So, if your FreeBSD creates v14 ZFS pools by defaut, and your Mac cannot import 
ZFS pools newer than v8, you need to create v8 instead of v14 pools on your 
FreeBSD:
zpool create -o version=8 mypool /dev/ada1

Original comment by numise...@gmail.com on 8 Sep 2010 at 3:02

[GoogleCodeExporter]

Now that onnv_75 appears to work, we can make progress again towards onnv_77 
which defined pool version 9 ...

Original comment by alex.ble...@gmail.com on 12 Dec 2010 at 3:03

[GoogleCodeExporter]

Original comment by alex.ble...@gmail.com on 19 Mar 2011 at 8:35

• Changed state: Accepted

[GoogleCodeExporter]

For reference, options discussed around 
https://groups.google.com/d/msg/zfs-macos/Elv3ULUTFS0/SyzXTIzm6PQJ (2012-09-20) 
include: 

a) switch to current FreeBSD

b) a big leap forward and start over …

Original comment by grahampe...@gmail.com on 11 Nov 2012 at 11:10