search

bolthole / zrep

ZREP ZFS based replication and failover script from bolthole.com
Other
251 stars 57 forks source link

Backup mode: Error: zrep_expire Internal Err caller did not hold fs lock on #160

Closed kingfisher77 closed 4 years ago

kingfisher77 commented 4 years ago

I am trying to implement the backup mode from here http://www.bolthole.com/solaris/zrep/zrep.documentation.html This is my script for testing:

# zpool list -o name -H
# tank
# zbank
#!/bin/bash
set -x
zfs destroy -r tank/test
zfs destroy -r zbank/test
zfs create tank/test
zfs create tank/test/d1
zfs create zbank/test
zfs create zbank/test/d1
zrep init zbank/test/d1 localhost tank/test/d1
zrep failover zbank/test/d1
touch /tank/test/d1/t1
zrep refresh zbank/test/d1
exit

This is what happens when the script runs:

+ zfs destroy -r tank/test
+ zfs destroy -r zbank/test
+ zfs create tank/test
+ zfs create tank/test/d1
+ zfs create zbank/test
+ zfs create zbank/test/d1
+ zrep init zbank/test/d1 localhost tank/test/d1
Setting zrep properties on zbank/test/d1
Creating snapshot zbank/test/d1@zrep_000000
Sending initial replication stream to localhost:tank/test/d1
Initialization copy of zbank/test/d1 to localhost:tank/test/d1 complete
Filesystem will not be mounted
+ zrep failover zbank/test/d1
Setting readonly on local zbank/test/d1, then syncing
sending zbank/test/d1@zrep_000001 to localhost:tank/test/d1
Reversing master properties for io:zbank/test/d1
Setting master on localhost:tank/test/d1
Setting master properties for io:tank/test/d1
Mounting io:tank/test/d1
+ touch /tank/test/d1/t1
+ zrep refresh zbank/test/d1
Expiring zrep snaps on tank/test/d1
Error: zrep_expire Internal Err caller did not hold fs lock on tank/test/d1
+ exit

What is wrong with fs lock?

ppbrown commented 4 years ago

first let me ask you, which specific version of zrep are you using? release, or latest git?

On Thu, Jul 23, 2020 at 11:26 PM kingfisher77 notifications@github.com wrote:

I am trying to implement the backup mode from here http://www.bolthole.com/solaris/zrep/zrep.documentation.html This is my script for testing:

zpool list -o name -H

tank

zbank

!/bin/bash

set -x zfs destroy -r tank/test zfs destroy -r zbank/test zfs create tank/test zfs create tank/test/d1 zfs create zbank/test zfs create zbank/test/d1 zrep init zbank/test/d1 localhost tank/test/d1 zrep failover zbank/test/d1 touch /tank/test/d1/t1 zrep refresh zbank/test/d1 exit

This is what happens when the script runs:

  • zfs destroy -r tank/test
  • zfs destroy -r zbank/test
  • zfs create tank/test
  • zfs create tank/test/d1
  • zfs create zbank/test
  • zfs create zbank/test/d1
  • zrep init zbank/test/d1 localhost tank/test/d1 Setting zrep properties on zbank/test/d1 Creating snapshot zbank/test/d1@zrep_000000 Sending initial replication stream to localhost:tank/test/d1 Initialization copy of zbank/test/d1 to localhost:tank/test/d1 complete Filesystem will not be mounted
  • zrep failover zbank/test/d1 Setting readonly on local zbank/test/d1, then syncing sending zbank/test/d1@zrep_000001 to localhost:tank/test/d1 Reversing master properties for io:zbank/test/d1 Setting master on localhost:tank/test/d1 Setting master properties for io:tank/test/d1 Mounting io:tank/test/d1
  • touch /tank/test/d1/t1
  • zrep refresh zbank/test/d1 Expiring zrep snaps on tank/test/d1 Error: zrep_expire Internal Err caller did not hold fs lock on tank/test/d1
  • exit

What is wrong with fs lock?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bolthole/zrep/issues/160, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANEV6ONAT3NELAHR67YMPLR5ESS7ANCNFSM4PGNLYDQ .

kingfisher77 commented 4 years ago

Its zrep 1.8

With the setup above zrep sync tank/test/d1 works without error message.

Is the error message related to the fact that Source and Backup is on the same machine?

ppbrown commented 4 years ago

just replying to this quickly from memory, without looking in detail. I vaguely recall fixing something around this area relatively recently. So I suggest pulling the latest git version and seeing if it works for you.

On Fri, Jul 24, 2020 at 8:30 AM kingfisher77 notifications@github.com wrote:

Its zrep 1.8

With the setup above zrep sync tank/test/d1 works without error message.

Is the error message related to the fact that Source and Backup is on the same machine?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

kingfisher77 commented 4 years ago

Latest git version is version 1.8 right? We have version 1.8

ppbrown commented 4 years ago

thats not what I mean. I mean, please try the most recent source. ie: download and try

https://github.com/bolthole/zrep/raw/master/zrep

On Fri, Jul 24, 2020 at 1:02 PM kingfisher77 notifications@github.com wrote:

Latest git version is version 1.8 right? We have version 1.8

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bolthole/zrep/issues/160#issuecomment-663709728, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANEV6JKI2Z5OK6FKTQAOHTR5HSDTANCNFSM4PGNLYDQ .

kingfisher77 commented 4 years ago

Now we get this:

root@io:~/work # /usr/local/src/zrep refresh zbank/test/d1                                                                                                                                  │·······················································································
ERROR: Dont know what to do with: _refreshcomplete tank/test/d1@zrep_000009 zrep:sent=1595623934                                                                                            │·······················································································
                                                                                                                                                                                            │·······················································································
zrep v1.8: a program to replicate a zfs filesystem to another                                                                                                                               │·······················································································
in an ongoing basis.                                                                                                                                                                        │·······················································································
                                                                                                                                                                                            │·······················································································
  Philip Brown, 2012-2019                                                                                                                                                                   │·······················································································
                                                                                                                                                                                            │·······················································································
Simple usage summary:                                                                                                                                                                       │·······················································································
zrep (init|-i) [-v] ZFS/fs remotehost remoteZFSpool/fs                                                                                                                                      │·······················································································
zrep (sync|-S) [-f] [-v] [-q seconds] ZFS/fs                                                                                                                                                │·······················································································
zrep (sync|-S) [-f] [-v] [-q seconds] all                                                                                                                                                   │·······················································································
zrep (sync|-S) [-f] [-v] ZFS/fs snapshot  -- temporary retroactive sync                                                                                                                     │·······················································································
zrep refresh [-f] [-v] ZFS/fs             -- pull partner of sync                                                                                                                           │·······················································································
zrep failover [-L] ZFS/fs                                                                                                                                                                   │·······················································································
zrep takeover [-L] ZFS/fs                                                                                                                                                                   │·······················································································
                                                                                                                                                                                            │·······················································································
Status and side operations                                                                                                                                                                  │·······················································································
zrep (status|-s) [-v] [(-a|ZFS/fs)]                                                                                                                                                         │·······················································································
zrep (list|-l) [-Lv]  [fs/names]                                                                                                                                                            │·······················································································
zrep (list|-l) -s     [fs/names]  -- list all relevant zrep snapshots                                                                                                                       │·······················································································
zrep lastsent ZFS/fs    -- prints out snapshot of last successful sync sent                                                                                                                 │·······················································································
zrep (expire|-e) [-L] (ZFS/fs ...)|(all)|()                                                                                                                                                 │·······················································································
zrep uptodate ZFS/fs    -- returns 0 if no writes since last sync.                                                                                                                          │·······················································································
zrep version                                                                                                                                                                                │·······················································································
zrep clear [-f] ZFS/fs  -- REMOVE ZREP CONFIG AND SNAPS FROM FILESYSTEM                                                                                                                     │·······················································································
zrep (changeconfig|-C) [-f] ZFS/fs remotehost remoteZFSpool/fs                                                                                                                              │·······················································································
zrep (changeconfig|-C) [-f] [-d] ZFS/fs srchost srcZFSpool/fs                                                                                                                               │·······················································································
                                                                                                                                                                                            │·······················································································
   -q option to sync says to Quietly ignore locked filesystems                                                                                                                              │·······················································································
    that have synced more recently than the given amount of seconds                                                                                                                         │·······················································································
                                                                                                                                                                                            │·······················································································
Paired commands for high-transaction systems:                                                                                                                                               │·······················································································
   zrep snaponly (ZFS/fs ... |all)                                                                                                                                                          │·······················································································
   zrep synconly (ZFS/fs ...|all)                                                                                                                                                           │·······················································································
The above two commands split the simple sync subcommand, into two                                                                                                                           │·······················································································
separate steps, so that a database, etc. may resume while the sync                                                                                                                          │·······················································································
completes in the background                                                                                                                                                                 │·······················································································
                                                                                                                                                                                            │·······················································································
zrep defaults to using ssh. However, if remotehost is set to localhost,                                                                                                                     │·······················································································
zrep will use a simple pipe instead.                                                                                                                                                        │·······················································································
                                                                                                                                                                                            │·······················································································
 More detailed examples can be found at:                                                                                                                                                    │·······················································································
http://www.bolthole.com/solaris/zrep/zrep.documentation.html                                                                                                                                │·······················································································
                                                                                                                                                                                            │·······················································································
See the above documentation for details on using the -t flag to use zrep tags                                                                                                               │·······················································································
Expiring zrep snaps on tank/test/d1                                                                                                                                                         │·······················································································
Error: zrep_expire Internal Err caller did not hold fs lock on tank/test/d1
kingfisher77 commented 4 years ago

Version 1.7.7 had it already

root@io:~/work # ./zrep-test.sh                                                                                                                                                             │·······················································································
+ ZREP=/usr/local/src/zrep                                                                                                                                                                  │·······················································································
+ ZREP=/root/work/zrep-1.7.7/zrep                                                                                                                                                           │·······················································································
+ zfs destroy -r tank/test                                                                                                                                                                  │·······················································································
+ zfs destroy -r zbank/test                                                                                                                                                                 │·······················································································
+ zfs create tank/test                                                                                                                                                                      │·······················································································
+ zfs create tank/test/d1                                                                                                                                                                   │·······················································································
+ zfs create zbank/test                                                                                                                                                                     │·······················································································
+ zfs create zbank/test/d1                                                                                                                                                                  │·······················································································
+ /root/work/zrep-1.7.7/zrep init zbank/test/d1 localhost tank/test/d1                                                                                                                      │·······················································································
Setting properties on zbank/test/d1                                                                                                                                                         │·······················································································
Creating snapshot zbank/test/d1@zrep_000000                                                                                                                                                 │·······················································································
Sending initial replication stream to localhost:tank/test/d1                                                                                                                                │·······················································································
Initialization copy of zbank/test/d1 to localhost:tank/test/d1 complete                                                                                                                     │·······················································································
Filesystem will not be mounted                                                                                                                                                              │·······················································································
+ /root/work/zrep-1.7.7/zrep failover zbank/test/d1                                                                                                                                         │·······················································································
Setting readonly on local zbank/test/d1, then syncing                                                                                                                                       │·······················································································
sending zbank/test/d1@zrep_000001 to localhost:tank/test/d1                                                                                                                                 │·······················································································
Reversing master properties for io:zbank/test/d1                                                                                                                                            │·······················································································
Setting master on localhost:tank/test/d1                                                                                                                                                    │·······················································································
Setting master properties for io:tank/test/d1                                                                                                                                               │·······················································································
Mounting io:tank/test/d1                                                                                                                                                                    │·······················································································
+ touch /tank/test/d1/t1                                                                                                                                                                    │·······················································································
+ /root/work/zrep-1.7.7/zrep refresh zbank/test/d1                                                                                                                                          │·······················································································
Expiring zrep snaps on tank/test/d1                                                                                                                                                         │·······················································································
Error: zrep_expire Internal Err caller did not hold fs lock on tank/test/d1                                                                                                                 │·······················································································
+ exit

zrep does its work, the data is in the backup, snapshots are there.

ppbrown commented 4 years ago

seems like you added some odd stuff as debug. that could be interfering.

btw: yes i know it did the sync. it's erroring only when trying to do the "expire" :) There's a race condition there that I vaguely recall

On Fri, Jul 24, 2020 at 2:01 PM kingfisher77 notifications@github.com wrote:

Version 1.7.7 had it already

root@io:~/work # ./zrep-test.sh │·······················································································

  • ZREP=/usr/local/src/zrep │·······················································································
  • ZREP=/root/work/zrep-1.7.7/zrep │·······················································································
  • zfs destroy -r tank/test │·······················································································
  • zfs destroy -r zbank/test │·······················································································
  • zfs create tank/test │·······················································································
  • zfs create tank/test/d1 │·······················································································
  • zfs create zbank/test │·······················································································
  • zfs create zbank/test/d1 │·······················································································
  • /root/work/zrep-1.7.7/zrep init zbank/test/d1 localhost tank/test/d1 │······················································································· Setting properties on zbank/test/d1 │······················································································· Creating snapshot zbank/test/d1@zrep_000000 │······················································································· Sending initial replication stream to localhost:tank/test/d1 │······················································································· Initialization copy of zbank/test/d1 to localhost:tank/test/d1 complete │······················································································· Filesystem will not be mounted │·······················································································
  • /root/work/zrep-1.7.7/zrep failover zbank/test/d1 │······················································································· Setting readonly on local zbank/test/d1, then syncing │······················································································· sending zbank/test/d1@zrep_000001 to localhost:tank/test/d1 │······················································································· Reversing master properties for io:zbank/test/d1 │······················································································· Setting master on localhost:tank/test/d1 │······················································································· Setting master properties for io:tank/test/d1 │······················································································· Mounting io:tank/test/d1 │·······················································································
  • touch /tank/test/d1/t1 │·······················································································
  • /root/work/zrep-1.7.7/zrep refresh zbank/test/d1 │······················································································· Expiring zrep snaps on tank/test/d1 │······················································································· Error: zrep_expire Internal Err caller did not hold fs lock on tank/test/d1 │·······················································································
  • exit

zrep does its work, the data is in the backup, snapshots are there.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bolthole/zrep/issues/160#issuecomment-663732442, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANEV6JE4564IRWC7UWPYVDR5HZDJANCNFSM4PGNLYDQ .

kingfisher77 commented 4 years ago

Its just this script

#!/bin/bash
set -x

ZREP=/usr/local/src/zrep
#ZREP=/root/work/zrep-1.7.7/zrep
zfs destroy -r tank/test
zfs destroy -r zbank/test
zfs create tank/test
zfs create tank/test/d1
zfs create zbank/test
zfs create zbank/test/d1
$ZREP init zbank/test/d1 localhost tank/test/d1
$ZREP failover zbank/test/d1
touch /tank/test/d1/t1
$ZREP refresh zbank/test/d1
exit
ppbrown commented 4 years ago

where did all the

--------

lines come from?

On Fri, Jul 24, 2020 at 2:14 PM kingfisher77 notifications@github.com wrote:

Its just this script

!/bin/bash

set -x

ZREP=/usr/local/src/zrep

ZREP=/root/work/zrep-1.7.7/zrep

zfs destroy -r tank/test zfs destroy -r zbank/test zfs create tank/test zfs create tank/test/d1 zfs create zbank/test zfs create zbank/test/d1 $ZREP init zbank/test/d1 localhost tank/test/d1 $ZREP failover zbank/test/d1 touch /tank/test/d1/t1 $ZREP refresh zbank/test/d1 exit

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bolthole/zrep/issues/160#issuecomment-663736789, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANEV6LKZTAB37IICDFMCJ3R5H2SBANCNFSM4PGNLYDQ .

kingfisher77 commented 4 years ago

Sorry, it was a copy and past error from a tmux session. Here it is clean:

root@io:~/work # ./zrep-test.sh
+ ZREP=/usr/local/src/zrep
+ zfs destroy -r tank/test
+ zfs destroy -r zbank/test
+ zfs create tank/test
+ zfs create tank/test/d1
+ zfs create zbank/test
+ zfs create zbank/test/d1
+ /usr/local/src/zrep init zbank/test/d1 localhost tank/test/d1
Setting zrep properties on zbank/test/d1
Creating snapshot zbank/test/d1@zrep_000000
Sending initial replication stream to localhost:tank/test/d1
Initialization copy of zbank/test/d1 to localhost:tank/test/d1 complete
Filesystem will not be mounted
+ /usr/local/src/zrep failover zbank/test/d1
Setting readonly on local zbank/test/d1, then syncing
sending zbank/test/d1@zrep_000001 to localhost:tank/test/d1
Reversing master properties for io:zbank/test/d1
Setting master on localhost:tank/test/d1
Setting master properties for io:tank/test/d1
Mounting io:tank/test/d1
+ touch /tank/test/d1/t1
+ /usr/local/src/zrep refresh zbank/test/d1
ERROR: Dont know what to do with: _refreshcomplete tank/test/d1@zrep_000002 zrep:sent=1595657439

zrep v1.8: a program to replicate a zfs filesystem to another
in an ongoing basis.

  Philip Brown, 2012-2019

Simple usage summary:
zrep (init|-i) [-v] ZFS/fs remotehost remoteZFSpool/fs
zrep (sync|-S) [-f] [-v] [-q seconds] ZFS/fs
zrep (sync|-S) [-f] [-v] [-q seconds] all
zrep (sync|-S) [-f] [-v] ZFS/fs snapshot  -- temporary retroactive sync
zrep refresh [-f] [-v] ZFS/fs             -- pull partner of sync
zrep failover [-L] ZFS/fs
zrep takeover [-L] ZFS/fs

Status and side operations
zrep (status|-s) [-v] [(-a|ZFS/fs)]
zrep (list|-l) [-Lv]  [fs/names]
zrep (list|-l) -s     [fs/names]  -- list all relevant zrep snapshots
zrep lastsent ZFS/fs    -- prints out snapshot of last successful sync sent
zrep (expire|-e) [-L] (ZFS/fs ...)|(all)|()
zrep uptodate ZFS/fs    -- returns 0 if no writes since last sync.
zrep version
zrep clear [-f] ZFS/fs  -- REMOVE ZREP CONFIG AND SNAPS FROM FILESYSTEM
zrep (changeconfig|-C) [-f] ZFS/fs remotehost remoteZFSpool/fs
zrep (changeconfig|-C) [-f] [-d] ZFS/fs srchost srcZFSpool/fs

   -q option to sync says to Quietly ignore locked filesystems
    that have synced more recently than the given amount of seconds

Paired commands for high-transaction systems:
   zrep snaponly (ZFS/fs ... |all)
   zrep synconly (ZFS/fs ...|all)
The above two commands split the simple sync subcommand, into two
separate steps, so that a database, etc. may resume while the sync
completes in the background

zrep defaults to using ssh. However, if remotehost is set to localhost,
zrep will use a simple pipe instead.

 More detailed examples can be found at:
http://www.bolthole.com/solaris/zrep/zrep.documentation.html

See the above documentation for details on using the -t flag to use zrep tags
Expiring zrep snaps on tank/test/d1
Error: zrep_expire Internal Err caller did not hold fs lock on tank/test/d1
+ exit
kingfisher77 commented 4 years ago

We have found another solution for backing up zfs data sets. Thanks a lot anyway!

ppbrown commented 4 years ago

Thank you for the update.

As a note to anyone who hits this with search:

(yea I need to update the error message, but)

ERROR: Dont know what to do with: _refreshcomplete tank/test/d1@zrep_000002 zrep:sent=1595657439

indicates mismatched version of zrep. the master has a later version than the destination.

Yes this is even possible with replication to "localhost", because for the remote invocations, zrep calls

$ZREP_PATH which uses the default $PATH to find where zrep is.

So theposter was initiating by calling /usr/local/src/zrep, but it is probably calling /usr/local/bin/zrep or similar on the back end.

On Mon, Jul 27, 2020 at 10:07 AM kingfisher77 notifications@github.com wrote:

We have found another solution for backing up zfs data sets. Thanks a lot anyway!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bolthole/zrep/issues/160#issuecomment-664521698, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANEV6JDOIALVDVJHY7ZVKDR5WX6TANCNFSM4PGNLYDQ .