search

openzfsonosx / zfs

OpenZFS on OS X
https://openzfsonosx.org/
Other
822 stars 72 forks source link

ZFS pool on Core Storage, created with a 20130712 build of zfs-osx, can not be used by ZEVO #41

Closed grahamperrin closed 10 years ago

grahamperrin commented 10 years ago

With a 20130712 build of zfs-osx

Preparation

Destruction of a previous test pool

GPES3E-gjp4-1:bin bbsadmin-l$ sudo ./zfs unmount maczfsprototype
unmountall
zfs_unmount
located '/Users/maczfsprototype'
Unmount successful for /Users/maczfsprototype
GPES3E-gjp4-1:bin bbsadmin-l$ sudo ./zpool destroy maczfsprototype
GPES3E-gjp4-1:bin bbsadmin-l$ clear

Creation of a version 28 pool

GPES3E-gjp4-1:bin bbsadmin-l$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *750.2 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                  Apple_HFS swap                    32.0 GB    disk0s2
   3:                  Apple_HFS disk0s3                 536.9 MB   disk0s3
   4:                  Apple_HFS spare                   671.1 MB   disk0s4
   5:          Apple_CoreStorage                         99.5 GB    disk0s5
   6:                 Apple_Boot Boot OS X               650.0 MB   disk0s6
   7:          Apple_CoreStorage                         616.3 GB   disk0s7
   8:                 Apple_Boot Boot OS X               134.2 MB   disk0s8
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS OS                     *99.2 GB    disk1
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *100.0 GB   disk3
   1:                        EFI EFI                     209.7 MB   disk3s1
   2:                  Apple_HFS OS 100                  99.2 GB    disk3s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk3s3
/dev/disk4
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *8.0 GB     disk4
   1:                        EFI EFI                     209.7 MB   disk4s1
   2:                        ZFS                         7.7 GB     disk4s2
/dev/disk5
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *7.7 GB     disk5
   1:                        EFI EFI                     209.7 MB   disk5s1
   2:          Apple_CoreStorage                         7.4 GB     disk5s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk5s3
/dev/disk6
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *640.1 GB   disk6
   1:                        EFI EFI                     209.7 MB   disk6s1
   2:                        ZFS                         639.8 GB   disk6s2
/dev/disk8
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *7.1 GB     disk8
   1:                        ZFS                         7.1 GB     disk8s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk8s9
/dev/disk9
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     Apple_partition_scheme                        *37.8 MB    disk9
   1:        Apple_partition_map                         32.3 KB    disk9s1
   2:                  Apple_HFS osx.zfs-20130712        37.8 MB    disk9s2
GPES3E-gjp4-1:bin bbsadmin-l$ diskutil unmountDisk /dev/disk8
Unmount of all volumes on disk8 was successful
GPES3E-gjp4-1:bin bbsadmin-l$ sudo ./zpool create -o version=28 -O casesensitivity=insensitive -O normalization=formD -O compression=on -O snapdir=visible -O mountpoint=/Users/maczfsprototype maczfsprototype /dev/disk8
efi_write mate3
efi_write mate
checking path '/dev/disk8'
zfs_mount: unused options: "defaults,atime,dev,exec,rw,suid,xattr,nomand"
GPES3E-gjp4-1:bin bbsadmin-l$

A later review of properties, and an export

GPES3E-gjp4-1:~ bbsadmin-l$ sudo /Volumes/osx.zfs-20130712/load_zfs.sh 
Password:
ZFS loaded... Please add the bin to PATH, ie;
export PATH="$PATH:/Volumes/osx.zfs-20130712/64/bin"
GPES3E-gjp4-1:~ bbsadmin-l$ cd /Volumes/osx.zfs-20130712/64/bin 
GPES3E-gjp4-1:bin bbsadmin-l$ sudo ./zpool import
   pool: maczfsprototype
     id: 4802247225081302216
  state: ONLINE
 status: The pool is formatted using a legacy on-disk version.
 action: The pool can be imported using its name or numeric identifier, though
    some features will not be available without an explicit 'zpool upgrade'.
 config:

    maczfsprototype  ONLINE
      disk3s1   ONLINE
GPES3E-gjp4-1:bin bbsadmin-l$ sudo ./zpool import maczfsprototype
zfs_mount: unused options: "defaults,atime,dev,exec,rw,suid,xattr,nomand"
GPES3E-gjp4-1:bin bbsadmin-l$ sudo ./zpool get all maczfsprototype
NAME             PROPERTY               VALUE                  SOURCE
maczfsprototype  size                   6.56G                  -
maczfsprototype  capacity               0%                     -
maczfsprototype  altroot                -                      default
maczfsprototype  health                 ONLINE                 -
maczfsprototype  guid                   4802247225081302216    local
maczfsprototype  version                28                     local
maczfsprototype  bootfs                 -                      default
maczfsprototype  delegation             on                     default
maczfsprototype  autoreplace            off                    default
maczfsprototype  cachefile              -                      default
maczfsprototype  failmode               wait                   default
maczfsprototype  listsnapshots          off                    default
maczfsprototype  autoexpand             off                    default
maczfsprototype  dedupditto             0                      default
maczfsprototype  dedupratio             1.00x                  -
maczfsprototype  free                   6.56G                  -
maczfsprototype  allocated              280K                   -
maczfsprototype  readonly               off                    -
maczfsprototype  ashift                 0                      default
maczfsprototype  comment                -                      default
maczfsprototype  expandsize             16.0E                  -
maczfsprototype  freeing                0                      local
maczfsprototype  feature@async_destroy  disabled               local
maczfsprototype  feature@empty_bpobj    disabled               local
GPES3E-gjp4-1:bin bbsadmin-l$ sudo ./zfs get all maczfsprototype
NAME             PROPERTY              VALUE                   SOURCE
maczfsprototype  type                  filesystem              -
maczfsprototype  creation              Tue Jul 16  0:01 2013   -
maczfsprototype  used                  184K                    -
maczfsprototype  available             6.46G                   -
maczfsprototype  referenced            113K                    -
maczfsprototype  compressratio         7.39x                   -
maczfsprototype  quota                 none                    default
maczfsprototype  reservation           none                    default
maczfsprototype  recordsize            128K                    default
maczfsprototype  mountpoint            /Users/maczfsprototype  local
maczfsprototype  sharenfs              off                     default
maczfsprototype  checksum              on                      default
maczfsprototype  compression           on                      local
maczfsprototype  zoned                 off                     default
maczfsprototype  snapdir               visible                 local
maczfsprototype  aclinherit            restricted              default
maczfsprototype  canmount              on                      default
maczfsprototype  copies                1                       default
maczfsprototype  version               5                       -
maczfsprototype  utf8only              on                      -
maczfsprototype  normalization         formD                   -
maczfsprototype  casesensitivity       insensitive             -
maczfsprototype  vscan                 off                     default
maczfsprototype  sharesmb              off                     default
maczfsprototype  refquota              none                    default
maczfsprototype  refreservation        none                    default
maczfsprototype  primarycache          all                     default
maczfsprototype  secondarycache        all                     default
maczfsprototype  usedbysnapshots       0                       -
maczfsprototype  usedbydataset         113K                    -
maczfsprototype  usedbychildren        70.5K                   -
maczfsprototype  usedbyrefreservation  0                       -
maczfsprototype  logbias               latency                 default
maczfsprototype  dedup                 off                     default
maczfsprototype  mlslabel              none                    default
maczfsprototype  sync                  standard                default
maczfsprototype  refcompressratio      9.23x                   -
maczfsprototype  written               113K                    -
GPES3E-gjp4-1:bin bbsadmin-l$ sudo ./zpool export maczfsprototype
Unmount successful for /Users/maczfsprototype
GPES3E-gjp4-1:bin bbsadmin-l$ exit
logout

[Process completed]

With ZEVO Community Edition 1.1.1 on Mountain Lion

Unlocking the Core Storage logical volume:

gpes3e-gjp4:~ gjp22$ diskutil cs unlockVolume D0699260-3F5A-47A6-BFA4-B1CE79ADB4ED
Passphrase:
Started CoreStorage operation
Logical Volume successfully unlocked
Logical Volume successfully attached as disk6
Error: -69842: Couldn't mount disk
gpes3e-gjp4:~ gjp22$ diskutil list /dev/disk6
/dev/disk6
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *7.1 GB     disk6
   1:                        ZFS                         7.1 GB     disk6s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk6s9

The issue

gpes3e-gjp4:~ gjp22$ sudo zpool import
Password:
  pool: maczfsprototype
    id: 4802247225081302216
 state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
config:

    maczfsprototype                              UNAVAIL  insufficient replicas
      GPTE_AB543C80-915E-A44D-92BE-5CEFB3C45457  UNAVAIL  corrupted data
gpes3e-gjp4:~ gjp22$

Comparison

For a Core Storage logical volume with which there's no difficulty

gpes3e-gjp4:~ gjp22$ diskutil list /dev/disk3
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *616.0 GB   disk3
   1:                        EFI                         209.7 MB   disk3s1
   2:                        ZFS                         615.7 GB   disk3s2
gpes3e-gjp4:~ gjp22$
lundman commented 10 years ago

I can see a couple of things that are suspect here;

  create -o version=28
maczfsprototype  feature@async_destroy

If we create a pool version 28, then it should not set the feature-flags. They belong only to pool version 5000.

maczfsprototype  ashift                 0

That is clearly undesirable.

grahamperrin commented 10 years ago

In the ZEVO support forum: Alignment at the Core Storage LVM layer, ashift above?

… I guess that if alignment is proper at the (lower) Core Storage layer, then I can accept the default ashift 0 at the (upper) ZFS pool layer. …

grahamperrin commented 10 years ago

Without Core Storage –

ashift                 0                      default

– and: 

feature@async_destroy  disabled               local
feature@empty_bpobj    disabled               local

None of those things are an obstacle to ZEVO Community Edition 1.1.1:

grahamperrin commented 10 years ago

Terminal output below. ZEVO on Mountain Lion. 

gpes3e-gjp4:~ gjp22$ diskutil list /dev/disk6 ; diskutil list /dev/disk7 ; diskutil list /dev/disk8
/dev/disk6
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *7.7 GB     disk6
   1:                        EFI                         209.7 MB   disk6s1
   2:          Apple_CoreStorage                         7.4 GB     disk6s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk6s3
/dev/disk7
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *7.1 GB     disk7
   1:                        ZFS                         7.1 GB     disk7s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk7s9
/dev/disk8
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:             zfs_pool_proxy maczfsprototype        *7.1 GB     disk8
gpes3e-gjp4:~ gjp22$ clear

gpes3e-gjp4:~ gjp22$ sudo zdb -l /dev/disk7s1
--------------------------------------------
LABEL 0
--------------------------------------------
    version: 28
    name: 'maczfsprototype'
    state: 1
    txg: 114
    pool_guid: 4802247225081302216
    hostname: ''
    top_guid: 6208245507873663722
    guid: 6208245507873663722
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 6208245507873663722
        path: '/dev/disk3s1'
        whole_disk: 1
        metaslab_array: 30
        metaslab_shift: 26
        ashift: 9
        asize: 7064780800
        is_log: 0
        create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 1
--------------------------------------------
    version: 28
    name: 'maczfsprototype'
    state: 1
    txg: 114
    pool_guid: 4802247225081302216
    hostname: ''
    top_guid: 6208245507873663722
    guid: 6208245507873663722
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 6208245507873663722
        path: '/dev/disk3s1'
        whole_disk: 1
        metaslab_array: 30
        metaslab_shift: 26
        ashift: 9
        asize: 7064780800
        is_log: 0
        create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 2
--------------------------------------------
    version: 28
    name: 'maczfsprototype'
    state: 1
    txg: 114
    pool_guid: 4802247225081302216
    hostname: ''
    top_guid: 6208245507873663722
    guid: 6208245507873663722
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 6208245507873663722
        path: '/dev/disk3s1'
        whole_disk: 1
        metaslab_array: 30
        metaslab_shift: 26
        ashift: 9
        asize: 7064780800
        is_log: 0
        create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 3
--------------------------------------------
    version: 28
    name: 'maczfsprototype'
    state: 1
    txg: 114
    pool_guid: 4802247225081302216
    hostname: ''
    top_guid: 6208245507873663722
    guid: 6208245507873663722
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 6208245507873663722
        path: '/dev/disk3s1'
        whole_disk: 1
        metaslab_array: 30
        metaslab_shift: 26
        ashift: 9
        asize: 7064780800
        is_log: 0
        create_txg: 4
    features_for_read:
gpes3e-gjp4:~ gjp22$ clear

gpes3e-gjp4:~ gjp22$ diskutil coreStorage list
CoreStorage logical volume groups (3 found)
|
+-- Logical Volume Group 902434C9-0131-4E3A-AE15-2B8B938087AD
|   =========================================================
|   Name:         gjp22-cs
|   Status:       Online
|   Size:         616336003072 B (616.3 GB)
|   Free Space:   0 B (0 B)
|   |
|   +-< Physical Volume 179AADE6-34F1-404C-A994-9FD99C881BA6
|   |   ----------------------------------------------------
|   |   Index:    0
|   |   Disk:     disk0s7
|   |   Status:   Online
|   |   Size:     616336003072 B (616.3 GB)
|   |
|   +-> Logical Volume Family FFCE2FAF-BE8E-4FEF-9F3E-E221C6CBCA11
|       ----------------------------------------------------------
|       Encryption Status:       Unlocked
|       Encryption Type:         AES-XTS
|       Conversion Status:       Complete
|       Conversion Direction:    -none-
|       Has Encrypted Extents:   Yes
|       Fully Secure:            Yes
|       Passphrase Required:     Yes
|       |
|       +-> Logical Volume 0CFAFD38-E79B-40AC-A4BE-63296E6B4331
|           ---------------------------------------------------
|           Disk:               disk3
|           Status:             Online
|           Size (Total):       616017227776 B (616.0 GB)
|           Size (Converted):   -none-
|           Revertible:         No
|           LV Name:            gjp22-cs
|           Content Hint:       Apple_HFS
|
+-- Logical Volume Group A82B1A4B-B312-41AB-B020-F813CCC74C1C
|   =========================================================
|   Name:         OS
|   Status:       Online
|   Size:         99484213248 B (99.5 GB)
|   Free Space:   0 B (0 B)
|   |
|   +-< Physical Volume 18126061-401E-47DB-A215-8CC6400769F6
|   |   ----------------------------------------------------
|   |   Index:    0
|   |   Disk:     disk0s5
|   |   Status:   Online
|   |   Size:     99484213248 B (99.5 GB)
|   |
|   +-> Logical Volume Family 7615C140-F0A4-4BDC-AEFF-A78A55E41B1B
|       ----------------------------------------------------------
|       Encryption Status:       Unlocked
|       Encryption Type:         AES-XTS
|       Conversion Status:       Complete
|       Conversion Direction:    -none-
|       Has Encrypted Extents:   Yes
|       Fully Secure:            Yes
|       Passphrase Required:     Yes
|       |
|       +-> Logical Volume 6F42406A-AB44-40B2-9838-9EAFC5CF1C09
|           ---------------------------------------------------
|           Disk:               disk2
|           Status:             Online
|           Size (Total):       99165437952 B (99.2 GB)
|           Size (Converted):   -none-
|           Revertible:         No
|           LV Name:            OS
|           Volume Name:        OS
|           Content Hint:       Apple_HFS
|
+-- Logical Volume Group 84ADC337-A470-4377-B235-01AD6E3221B0
    =========================================================
    Name:         blooper
    Status:       Online
    Size:         7398711296 B (7.4 GB)
    Free Space:   0 B (0 B)
    |
    +-< Physical Volume D7220B99-6B8A-4DA6-B4AA-F6C25D511D29
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk6s2
    |   Status:   Online
    |   Size:     7398711296 B (7.4 GB)
    |
    +-> Logical Volume Family 783C74C2-9B69-49E3-8169-67959DFE8F95
        ----------------------------------------------------------
        Encryption Status:       Unlocked
        Encryption Type:         AES-XTS
        Conversion Status:       Complete
        Conversion Direction:    -none-
        Has Encrypted Extents:   Yes
        Fully Secure:            Yes
        Passphrase Required:     Yes
        |
        +-> Logical Volume D0699260-3F5A-47A6-BFA4-B1CE79ADB4ED
            ---------------------------------------------------
            Disk:               disk7
            Status:             Online
            Size (Total):       7079936000 B (7.1 GB)
            Size (Converted):   -none-
            Revertible:         No
            LV Name:            blooper
            Content Hint:       Apple_HFS
gpes3e-gjp4:~ gjp22$
grahamperrin commented 10 years ago

In response to suggestions from ilovezfs

gpes3e-gjp4:/ gjp22$ diskutil cs unlockVolume D0699260-3F5A-47A6-BFA4-B1CE79ADB4ED
Passphrase:
Started CoreStorage operation
Logical Volume successfully unlocked
Logical Volume successfully attached as disk13
Error: -69842: Couldn't mount disk
gpes3e-gjp4:/ gjp22$ diskutil list /dev/disk13 ; diskutil list /dev/disk14
/dev/disk13
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *7.1 GB     disk13
   1:                        ZFS                         7.1 GB     disk13s1
   2: 6A945A3B-1DD2-11B2-99A6-080020736631               8.4 MB     disk13s9
/dev/disk14
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:             zfs_pool_proxy                        *7.1 GB     disk14

zpool import -d /dev

gpes3e-gjp4:/ gjp22$ sudo zpool import -d /dev
  pool: maczfsprototype
    id: 4802247225081302216
 state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
config:

    maczfsprototype  UNAVAIL  insufficient replicas
      /dev/disk13s1  UNAVAIL  corrupted data

With force

gpes3e-gjp4:/ gjp22$ sudo zpool import -f maczfsprototype
cannot import 'maczfsprototype': invalid vdev configuration

Slice 2, symbolic linking

gpes3e-gjp4:/ gjp22$ sudo ln -s /dev/disk13s1 /dev/disk13s2
gpes3e-gjp4:/ gjp22$ sudo zpool import
  pool: maczfsprototype
    id: 4802247225081302216
 state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
config:

    maczfsprototype                              UNAVAIL  insufficient replicas
      GPTE_AB543C80-915E-A44D-92BE-5CEFB3C45457  UNAVAIL  corrupted data
gpes3e-gjp4:/ gjp22$
grahamperrin commented 10 years ago

I experimented with setting the ashift value to 9 at time of creation by MacZFS prototype. The setting did not work around the issue.

grahamperrin commented 10 years ago

In the ZEVO support forum:

Getting ashift property values: use zdb, not zpool

With ZEVO Community Edition 1.1.1:

• the zpool get subcommand does not work for the ashift property.

ilovezfs commented 10 years ago

These issues, whatever they may have been, have likely been fixed for a while now.

It is worth noting that we now have even better CoreStorage support for new pools, since it will no longer attempt to partition the logical volume, which is not able to be partitioned since 10.8.5

https://github.com/openzfsonosx/zfs/commit/7083c925ccb86f75b9992101d6b36cba8fc9e028 https://github.com/openzfsonosx/zfs/commit/1e9663fbdc422cefe9b88135d5e4a7023e49e713