Upgrade Fedora to current release

From @pbinkley on March 29, 2016 17:53

Spike to try upgrading Fedora to the current version (4.5) and determine how much work will be needed on the rest of the stack to make it work. Our aim is to catch up to the current Fedora if possible, but if the work involved is too big we may decide to make the Fedora upgrade part of the PCDM migration process (so as to avoid redundant work making our current codebase work with Fedora 4.5 and then throw it all out again when we move on to the PCDM work).

Replaces #293, #465, #466

Copied from original issue: ualbertalib/HydraNorth#1034

Questions:

[x] How do we deploy Fedora 4.7 onto the HydraNorth Fedora datastore? What changes are needed? For all of this, we will produce a document explaining how
[ ] Upgrade ActiveFedora to improve performance
[ ] Upgrade OAI server as needed (not tested yet)
[x] How do we change datastore to relational database? What are the performance gains? What is the migration process? We want to capture all relevant information about our process for the community.

From @piyapongch on June 6, 2016 20:25

Dev Environment Fedora 4.5 Upgrade

ansible-dev

install jdk8 (ansible-dev/roles/hydranorth/task/main.yml)
- name: Install java
- yum: name=java-1.8.0-openjdk state=latest
set default to jdk8 (ansible-dev/roles/hydranorth/task/main.yml)
- name: Set java default to openjdk-1.8.0
- alternatives: name=java path=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

hydranorth

upgrade hydra-jetty to v8.7.0 version on (HydraNorth/lib/tasks/jetty.rake)
- Jettywrapper.hydra_jetty_version = "v8.7.0"
- Jettywrapper.url = "https://github.com/projecthydra/hydra-jetty/archive/v8.7.0.zip"

Now looking at Fedora 4.6

From @weiweishi on October 24, 2016 18:50

And now looking at Fedora 4.7 RC2, and will involve data migration as Fedora 4.7 removes the infinispan layer.

From @pgwillia on October 25, 2016 21:50

First I set up hydranorth master configured to use fcrepo_wrapper version: 4.7.0-RC-2.

Then backup and restore from one Fedora instance to the other:

# Backup from existing repository
curl -X POST localhost:8080/rest/fcr:backup

# Restore to 4.7.0
curl -X POST -d "/path/to/backup/directory" localhost:8080/rest/fcr:restore

[from https://wiki.duraspace.org/display/FF/Release+Testing+-+4.7.0#ReleaseTesting-4.7.0-MigrationofExistingRepositories]

Run rake hydranorth:solr_reindex_all

Works for ~14 items and collection with binaries and derivaties from hydranorthdev instance

where "works" is confirmation that the items appear in the 4.7.0-RC-2 Fedora instance and searched in the Hydranorth interface.

From @pgwillia on October 25, 2016 22:8

Sort of works for ~3349 items and collections (most without binaries and derivatives) from plano instance.

dittest@ualberta.ca [admin] can see these objects but permissions and collection relationships don't seem to be maintained by the solr_reindex task. Can mitigate this by using existing Solr index. [Took a snapshot from plano (solr/replication?command=backup) and used the output (jetty/solr/development-core/data/snapshot.datestamp) as the index (jetty/solr/development-core/data/index) on my dev vm.]

From @pgwillia on October 27, 2016 22:34

Tried running backup on test Fedora but ran out of space after ~30 minutes. @henryzhang87 also reported that he hasn't successfully been able to get backup to run on large Fedora repos.

...
Problems writing document "/usr/share/tomcat7/temp/1477584277882-0/binaries/aa/05/05/aa05057c915437b7f396bdc19ce9f9a094b0205c.bin" to backup: /usr/share/tomcat7/temp/1477584277882-0/binaries/aa/05/05/aa05057c915437b7f396bdc19ce9f9a094b0205c.bin (No such file or directory)
Problems writing document "/usr/share/tomcat7/temp/1477584277882-0/changes/summary_of_changes.json" to backup: No space left on device
Problems closing backup files in "/usr/share/tomcat7/temp/1477584277882-0": No space left on device

real    26m11.925s
user    0m0.103s
sys 0m0.660s

From @pgwillia on November 7, 2016 22:57

[x] review Fedora 4 performance testing
[x] ansible playbook for Fedora 4.7 deploy with postgress db store
- [x] JAVA_OPTS
- [x] logging (logback.xml)
- [x] modeshape config (repository.json)
[x] Fedora update plan with estimates for outages and how to roll back at any step
- [x] update to 4.3 (replace war file requires tomcat stop)
- [x] run fcrepo4-upgrade-utils-4.3.0 tool (requires app started)
- [x] update to 4.6.1 (replace war file requires tomcat stop)
- [x] create fcr:backup (requires app started)
- [x] restore fcr:restore (requires fedora 4.7 started)
- [x] when to freeze Solr index? (needs to be synched with fedora)
- [ ] how to prevent user deposits in application?

From @pgwillia on November 7, 2016 23:0

gillingham (era-test fedora)

fcrepo_data is 278G
fcr:backup is 274G (in 3:15:28 with 10G for tomcat)

From @pgwillia on November 10, 2016 17:39

Notes on configuring postgres.

create the db and user
edit pg_hba.conf to use password auth (rather than Ident)
Version 8.6 which is available from the base repo does not work!

Caused by: org.fcrepo.kernel.api.exception.RepositoryRuntimeException: org.modeshape.persistence.relational.RelationalProviderException: org.postgresql.util.PSQLException: ERROR: syntax error at or near "NOT"
  Position: 17

documentation states it's tested with v 9.4-9.5. Seems to work with 9.6 as well.
on fcr:restore encountered error which required backup-fixer.jar to fix backup

java.sql.BatchUpdateException: Batch entry 585 INSERT INTO MODESHAPE_REPOSITORY (ID, CONTENT) VALUES ('87a0a8c317f1e7jcr:system/mode:metadata', ?) was aborted.  Call getNextException to see the cause.

the above exception required a tomcat restart -- note that the filesystem storage did not require extra steps

From @pgwillia on November 10, 2016 17:45

Used awood's util to walk the tree before and after backup/restore demonstrates all resources are accounted for and get is successful.

The advice that there needs to be an intermediate step at Fedora 4.3 and Fedora 4.6.1 seems to be overly cautious. fcr:backup/fcr:restore seems to work but likely needs more scrutiny.

From @pgwillia on November 15, 2016 19:8

Rats! The era-test scale restore failed after 2:41 hr

java.sql.BatchUpdateException: Batch entry 705 INSERT INTO MODESHAPE_REPOSITORY (ID, CONTENT) VALUES ('87a0a8c7505d6419c9342d-0bb9-4dc8-a4ec-9b3284677a77', ?) was aborted.  Call getNextException to see the cause.
real    161m6.726s
user    0m0.000s
sys 0m1.346s

My VM with 10GB ran out of memory, kind of not a surprise

Nov 16 17:02:06 fedora server: # There is insufficient memory for the Java Runtime Environment to continue.
Nov 16 17:02:06 fedora server: # Native memory allocation (mmap) failed to map 10737418240 bytes for committing reserved memory.
Nov 16 17:02:06 fedora server: # An error report file with more information is saved as:
Nov 16 17:02:06 fedora server: # /usr/share/tomcat/hs_err_pid5157.log
Nov 16 17:02:06 fedora systemd: tomcat.service: control process exited, code=exited status=1
...
Nov 16 19:26:55 fedora server: 2016-11-16 19:26:55,673 [Transaction Reaper] WARN  com.arjuna.ats.arjuna - ARJUNA012117: TransactionReaper::check time
out for TX 0:ffff7f000001:a8d0:582cf38b:84c in state  RUN
Nov 16 19:26:55 fedora server: 2016-11-16 19:26:55,675 [Transaction Reaper Worker 0] WARN  com.arjuna.ats.arjuna - ARJUNA012095: Abort of action id 0
:ffff7f000001:a8d0:582cf38b:84c invoked while multiple threads active within it.
Nov 16 19:26:55 fedora server: 2016-11-16 19:26:55,676 [Transaction Reaper Worker 0] WARN  com.arjuna.ats.arjuna - ARJUNA012108: CheckedAction::check
 - atomic action 0:ffff7f000001:a8d0:582cf38b:84c aborting with 1 threads active!
Nov 16 19:26:55 fedora server: 2016-11-16 19:26:55,677 [Transaction Reaper Worker 0] WARN  com.arjuna.ats.arjuna - ARJUNA012121: TransactionReaper::d
oCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000001:a8d0:582cf38b:84c
Nov 16 19:27:27 fedora server: 2016-11-16 19:27:27,913 [http-bio-8080-exec-1] WARN  com.arjuna.ats.arjuna - ARJUNA012077: Abort called on already abo
rted atomic action 0:ffff7f000001:a8d0:582cf38b:84c
Nov 16 19:27:27 fedora server: ERROR 19:27:27.923 (FedoraRepositoryRestore) Problems restoring up the repository:
Nov 16 19:27:27 fedora server: ERROR 19:27:27.929 (FedoraRepositoryRestore) java.sql.BatchUpdateException: Batch entry 69 INSERT INTO MODESHAPE_REPOS
ITORY (ID, CONTENT) VALUES ('87a0a8c317f1e7jcr:system/mode:metadata', ?) was aborted.  Call getNextException to see the cause.
Nov 16 19:27:27 fedora server: ERROR 19:27:27.929 (FedoraRepositoryRestore) java.sql.BatchUpdateException: Batch entry 0 INSERT INTO MODESHAPE_REPOSI
TORY (ID, CONTENT) VALUES ('87a0a8c317f1e7mode:locks', ?) was aborted.  Call getNextException to see the cause.
Nov 16 19:27:27 fedora server: ERROR 19:27:27.929 (FedoraRepositoryRestore) javax.transaction.RollbackException: ARJUNA016063: The transaction is not
 active!

Took 4:55 hr for the simple-file config to run out of memory.

From @pgwillia on November 18, 2016 19:59

For @piyapongch. This isn't the best example because Plano objects are metadata only. There are a few pdfs and images though that have been added.

Plano's data dir:

[root@plano pjenkins]# tree /var/www/sites/hydranorth/jetty/fcrepo4-data
/var/www/sites/hydranorth/jetty/fcrepo4-data
|-- com.arjuna.ats.arjuna.common.ObjectStoreEnvironmentBean.default.objectStoreDir
|   `-- ShadowNoFileLockStore
|       `-- defaultStore
|-- com.arjuna.ats.arjuna.objectstore.objectStoreDir
|   `-- ShadowNoFileLockStore
|       `-- defaultStore
|           `-- Recovery
|               `-- TransactionStatusManager
|                   |-- 0_ffff8180de1e_8678_581b727f_0
|                   |-- 0_ffff8180de1e_99e0_581a5197_0
|                   |-- 0_ffff8180de1e_adfc_577c3b2e_0
|                   |-- 0_ffff8180de1e_af76_580fc9de_0
|                   |-- 0_ffff8180de1e_b1ff_57e439b1_0
|                   |-- 0_ffff8180de1e_c743_5774c7cb_0
|                   `-- 0_ffff8180de1e_df7e_57adffec_0
|-- fcrepo.activemq.directory
|   |-- db-10.log
|   |-- db.data
|   |-- db.redo
|   `-- lock
|-- fcrepo.binary.directory
|   |-- 02
|   |   `-- 49
|   |       `-- 7a
|   |           `-- 02497af76bb8db956f148f00ec4c8f04f42e5deb
|   |-- 04
|   |   `-- 1e
|   |       `-- 51
|   |           `-- 041e51e6cad2a431d4391348c41f26737b1616d4
|   |-- 09
|   |   |-- 5a
|   |   |   `-- c5
|   |   |       `-- 095ac56968c9753ba7df4a722b87545af9c38afa
|   |   `-- 92
|   |       `-- eb
|   |           `-- 0992eb9df81190e23e81087eec48fe3899a62225
|   |-- 0d
|   |   `-- ff
|   |       `-- 8f
|   |           `-- 0dff8f1894088e2e4fff7874cdef226a8dfb1440
|   |-- 0f
|   |   `-- d8
|   |       `-- 07
|   |           `-- 0fd807ad3b07ae90f9e05e5145c35be74f171f39
|   |-- 12
|   |   `-- e5
|   |       `-- f2
|   |           `-- 12e5f2da18960dc085ca27bec1ae9e3245389cb1
|   |-- 18
|   |   `-- bb
|   |       `-- fa
|   |           `-- 18bbfa150d06ba09fb6fd7729ade051d73ee8a6f
|   |-- 1a
|   |   `-- 1a
|   |       `-- 60
|   |           `-- 1a1a6067b6ffe6c854913ed2e1cb24e4239e85a8
|   |-- 1b
|   |   `-- 72
|   |       `-- 9a
|   |-- 1f
|   |   `-- 33
|   |       `-- 9f
|   |           `-- 1f339f13048f66e9226ad4c7d1b0cf2ba1afc55f
|   |-- 25
|   |   |-- 55
|   |   |   `-- 16
|   |   |       `-- 25551653a52f2199b66a5f61c6782b2079f2e7b7
|   |   `-- 8c
|   |       `-- 57
|   |           `-- 258c57cc2383b90a8a968e7577136b1cccc17503
|   |-- 26
|   |   `-- 1a
|   |       `-- 3a
|   |           `-- 261a3aa09ad0df7a31c2e4aacc289d08e435cb67
|   |-- 28
|   |   `-- 88
|   |       `-- 9a
|   |           `-- 28889a91353697b80d5ae15d5fc29d545972e4fd
|   |-- 2b
|   |   |-- 16
|   |   |   `-- a0
|   |   |       `-- 2b16a027d1791253f9337bf0dbdfd1fb8b930968
|   |   `-- d0
|   |       `-- 60
|   |           `-- 2bd06000649cc593ea8dab7b6e1d71047419135c
|   |-- 2d
|   |   `-- 0c
|   |       `-- 14
|   |           `-- 2d0c1420ec2f2d0cc69ae796487aaba22cfe37dd
|   |-- 2f
|   |   `-- 6f
|   |       `-- 57
|   |           `-- 2f6f578b240affbdf15acbf2661b2cd7d8ce98bb
|   |-- 30
|   |   |-- 88
|   |   |   `-- 88
|   |   |       `-- 3088887ed1b3a411ef88f53368c1cee91e6b78fa
|   |   `-- ee
|   |       `-- 29
|   |           `-- 30ee2987141e4b5c508ce8215d5f4420484fc80c
|   |-- 32
|   |   `-- 1e
|   |       `-- 4a
|   |           `-- 321e4ab8e29c973f82b6ea13a89acf3f21cc697f
|   |-- 39
|   |   `-- b9
|   |       `-- af
|   |           `-- 39b9aff56c8e8d4f209d633d0f64d4b3b93727e9
|   |-- 3e
|   |   `-- a5
|   |       `-- 60
|   |           `-- 3ea5606dc42653483442865329eb7c37bee045a9
|   |-- 43
|   |   `-- c0
|   |       `-- 5e
|   |           `-- 43c05eb7c75b209ecefb13eb8138c0afda6e773f
|   |-- 46
|   |   `-- 84
|   |       `-- 5b
|   |           `-- 46845b4aaf1d76ead465ae9a4d799d5ecb9cb607
|   |-- 48
|   |   |-- 3f
|   |   |   `-- 00
|   |   |       `-- 483f00854addea76bdc6f8667e19301dae4925cb
|   |   |-- 9b
|   |   |   `-- 21
|   |   |       `-- 489b21923ab89daf8630f1f277b0e871e8b1acf5
|   |   `-- e7
|   |       `-- b8
|   |           `-- 48e7b865ab5818035b487f6d3bea73a60c1c3bd0
|   |-- 4c
|   |   `-- cd
|   |       `-- 6d
|   |           `-- 4ccd6d2b117fe84617e02bfa45d893861ea6fbff
|   |-- 51
|   |   `-- d5
|   |       `-- 59
|   |           `-- 51d5592149ede90c30c066f4a163ce022e3a942f
|   |-- 52
|   |   `-- 28
|   |       `-- c0
|   |           `-- 5228c09b5cf50892191e8daf0dfff98365fd1b2e
|   |-- 55
|   |   `-- 99
|   |       `-- b5
|   |           `-- 5599b5f8810a79b3ff6f628c3e70f16912f1262e
|   |-- 56
|   |   `-- 5f
|   |       `-- 0d
|   |           `-- 565f0dc708226eee77169c3bc63694c57945b583
|   |-- 59
|   |   `-- 49
|   |       `-- 1a
|   |           `-- 59491ac99e6e7b192de7de1831b88071e6113fe4
|   |-- 5e
|   |   `-- a1
|   |       `-- 81
|   |           `-- 5ea181478c0441bbf2d714561a84cc27aa97b53d
|   |-- 61
|   |   `-- c2
|   |       `-- a7
|   |           `-- 61c2a79c352f492595b8f8a8be5637d07fb512f5
|   |-- 62
|   |   `-- e2
|   |       `-- 2c
|   |           `-- 62e22c0786796f7d3431972b58291989d94af823
|   |-- 68
|   |   `-- fe
|   |       `-- 4a
|   |           `-- 68fe4a0d62f8d5dd98847542681ddb176bcedcf7
|   |-- 6a
|   |   `-- b1
|   |       `-- 0d
|   |           `-- 6ab10d474abf2007b4ce1831ef85b238d3616967
|   |-- 6b
|   |   `-- 84
|   |       `-- dd
|   |           `-- 6b84dd6f8ebd947f855a80ba625fa9700513cc7e
|   |-- 6d
|   |   |-- 61
|   |   |   `-- f8
|   |   |       `-- 6d61f8f67da3b33bea6b2b76c28c323c9a028740
|   |   `-- a3
|   |       `-- de
|   |           `-- 6da3debe7c7474572ee9d7945a4bc8be472ecb44
|   |-- 74
|   |   `-- 90
|   |       `-- df
|   |           `-- 7490dfb835888ae4c28afe9b118a96d18f7908b6
|   |-- 76
|   |   `-- 33
|   |       `-- fb
|   |           `-- 7633fbc349fb885b0594cd2a8ec9f26b4d581147
|   |-- 78
|   |   `-- c3
|   |       `-- 4d
|   |           `-- 78c34d25d618d5abc74e8a8964b4a11a2d5da288
|   |-- 80
|   |   `-- 63
|   |       `-- 33
|   |           `-- 806333c7b607c89b6ebde0e7b0bd0e940e309b41
|   |-- 82
|   |   |-- 2d
|   |   |   `-- 65
|   |   |       `-- 822d65c8b1250519b30c3a12c06040fb1684092f
|   |   `-- 54
|   |       `-- f6
|   |           `-- 8254f6c73108d3b0a5583b953006d84667704ef3
|   |-- 87
|   |   `-- 19
|   |       `-- 64
|   |           `-- 871964969d8bd9b763b78d7c9da2516cf55c84e1
|   |-- 88
|   |   `-- 65
|   |       `-- 5d
|   |           `-- 88655def21d975ca82106546b91199903e57f45a
|   |-- 90
|   |   `-- db
|   |       `-- 32
|   |           `-- 90db3283250f1c19343209c385b779fcc783896e
|   |-- 92
|   |   |-- 06
|   |   |   `-- a3
|   |   |       `-- 9206a3652eb5574088d74e18baa47bb20f0d3344
|   |   `-- 9f
|   |       `-- c5
|   |           `-- 929fc5afbaa5dd9ad6af4a6a372f61ba7c004367
|   |-- 93
|   |   `-- 57
|   |       `-- bd
|   |           `-- 9357bd6b6028118a97726de13a0ba6c860067d79
|   |-- 96
|   |   |-- 3f
|   |   |   `-- 9c
|   |   |       `-- 963f9c65d1b1434ae2f021b3d9613653662e4ba5
|   |   `-- d3
|   |       `-- 1a
|   |           `-- 96d31ab4bbe8e92e7513bfcfe4d52980cbc82393
|   |-- 9c
|   |   `-- 6e
|   |       `-- 22
|   |-- 9d
|   |   |-- 95
|   |   |   `-- 0d
|   |   |       `-- 9d950d8bb5e359149c84ef52f093dfd872e23915
|   |   `-- 98
|   |       `-- 05
|   |           `-- 9d980592ed7b4fdf4a759357dd54d378eb505ef6
|   |-- aa
|   |   `-- 6a
|   |       `-- 34
|   |           `-- aa6a347faa61ea5cc34622da62a59c38bb2888a4
|   |-- b0
|   |   `-- 1c
|   |       `-- 42
|   |           `-- b01c42236988835677b7bc2e033e2154a6d22bce
|   |-- b3
|   |   `-- 6f
|   |       `-- 3d
|   |-- b6
|   |   `-- af
|   |       `-- 54
|   |           `-- b6af549e27ef082a3d430c034da548a03e194d0d
|   |-- b8
|   |   `-- 60
|   |       `-- c4
|   |-- b9
|   |   `-- 79
|   |       `-- fa
|   |           `-- b979faea6a73c863480a2f10a59835a7fba7dc98
|   |-- bd
|   |   `-- 0c
|   |       `-- ca
|   |           `-- bd0ccaf7fc57b7c58f9903eb4d99cef567e9ba5b
|   |-- bf
|   |   `-- 9b
|   |       `-- 57
|   |           `-- bf9b57bf40786010ea2ae182c494b6507bcf6851
|   |-- c8
|   |   `-- 82
|   |       `-- 5e
|   |           `-- c8825efe5bd179fb82bbd9ff15c44cb8516c5d77
|   |-- ca
|   |   `-- ef
|   |       `-- b0
|   |           `-- caefb078b07b6c11701278bf7bd61909a10ecc14
|   |-- cb
|   |   `-- a6
|   |       `-- 4c
|   |           `-- cba64cbe48002d6ee9e1185aa1896b3ba487e606
|   |-- cc
|   |   `-- 37
|   |       `-- 34
|   |           `-- cc3734011f36514d19fe902fd72dd4baff0f5e31
|   |-- ce
|   |   `-- 65
|   |       `-- bc
|   |           `-- ce65bc9a5aaaf6454ea6c2e49cbd25b2fc75045d
|   |-- d0
|   |   `-- c7
|   |       `-- 92
|   |           `-- d0c792fb39b23e28743e385772f2183f99412d14
|   |-- dc
|   |   `-- a3
|   |       `-- 5f
|   |           `-- dca35fb80f65fe47fedac793af50924d3ef5144c
|   |-- e0
|   |   `-- e5
|   |       `-- 86
|   |           `-- e0e5868aefb93e6cae8d7928e79ab872adc5646f
|   |-- e4
|   |   `-- b5
|   |       `-- 35
|   |           `-- e4b535af1c5e72d237f9d95c179578211cd01ef2
|   |-- e8
|   |   `-- 3c
|   |       `-- 08
|   |           `-- e83c083e3be192e3f3eb40336dafb7e66ea5571f
|   |-- eb
|   |   `-- 96
|   |       `-- 28
|   |           `-- eb96283497433e9b4a2ccd2d1690e43e22a8debc
|   |-- ec
|   |   `-- 86
|   |       `-- 88
|   |           `-- ec86884f0ce7ba21bc7264684b9d331258e6aac8
|   |-- f1
|   |   |-- 0d
|   |   |   `-- 6a
|   |   |       `-- f10d6ae069b585d229dba1a4c6823af4127c4743
|   |   `-- 43
|   |       `-- a8
|   |           `-- f143a805ffae9589f5a3b21d1ebe25d150a1f6f0
|   |-- f5
|   |   |-- 06
|   |   |   `-- eb
|   |   |       `-- f506ebc94fc227f235e1d3ddf4486bac8e942b17
|   |   `-- 2a
|   |       `-- e5
|   |           `-- f52ae576ffac28ad67fa415298b5feb1d4511e93
|   |-- f6
|   |   `-- fc
|   |       `-- 5d
|   |           `-- f6fc5d9f35c26c31b677b2f6be6f1dc4073d83ad
|   |-- f7
|   |   |-- 94
|   |   |   `-- b2
|   |   |       `-- f794b23c0c6fe1083d0ca8b58261a078cd968967
|   |   `-- e3
|   |       `-- 03
|   |           `-- f7e30383d48e6b970972540d58a7750dc8444858
|   |-- fc
|   |   `-- 13
|   |       `-- c3
|   |           `-- fc13c3933dd4cd239252845d3e5fc4a200224abd
|   `-- trash
|       |-- 02
|       |   `-- 49
|       |       `-- 7a
|       |-- 04
|       |   `-- 1e
|       |       `-- 51
|       |-- 09
|       |   |-- 5a
|       |   |   `-- c5
|       |   `-- 92
|       |       `-- eb
|       |-- 0d
|       |   `-- ff
|       |       `-- 8f
|       |-- 0f
|       |   `-- d8
|       |       `-- 07
|       |-- 12
|       |   `-- e5
|       |       `-- f2
|       |-- 18
|       |   `-- bb
|       |       `-- fa
|       |-- 1a
|       |   `-- 1a
|       |       `-- 60
|       |-- 1f
|       |   `-- 33
|       |       `-- 9f
|       |-- 25
|       |   `-- 8c
|       |       `-- 57
|       |-- 26
|       |   `-- 1a
|       |       `-- 3a
|       |-- 28
|       |   `-- 88
|       |       `-- 9a
|       |-- 2b
|       |   |-- 16
|       |   |   `-- a0
|       |   `-- d0
|       |       `-- 60
|       |-- 2d
|       |   `-- 0c
|       |       `-- 14
|       |-- 30
|       |   |-- 88
|       |   |   `-- 88
|       |   `-- ee
|       |       `-- 29
|       |-- 32
|       |   `-- 1e
|       |       `-- 4a
|       |-- 39
|       |   `-- b9
|       |       `-- af
|       |-- 3e
|       |   `-- a5
|       |       `-- 60
|       |-- 43
|       |   `-- c0
|       |       `-- 5e
|       |-- 46
|       |   `-- 84
|       |       `-- 5b
|       |-- 48
|       |   |-- 3f
|       |   |   `-- 00
|       |   |-- 9b
|       |   |   `-- 21
|       |   `-- e7
|       |       `-- b8
|       |-- 4c
|       |   `-- cd
|       |       `-- 6d
|       |-- 51
|       |   `-- d5
|       |       `-- 59
|       |-- 52
|       |   `-- 28
|       |       `-- c0
|       |-- 55
|       |   `-- 99
|       |       `-- b5
|       |-- 56
|       |   `-- 5f
|       |       `-- 0d
|       |-- 59
|       |   `-- 49
|       |       `-- 1a
|       |-- 5e
|       |   `-- a1
|       |       `-- 81
|       |-- 61
|       |   `-- c2
|       |       `-- a7
|       |-- 62
|       |   `-- e2
|       |       `-- 2c
|       |-- 68
|       |   `-- fe
|       |       `-- 4a
|       |-- 6a
|       |   `-- b1
|       |       `-- 0d
|       |-- 6b
|       |   `-- 84
|       |       `-- dd
|       |-- 6d
|       |   |-- 61
|       |   |   `-- f8
|       |   `-- a3
|       |       `-- de
|       |-- 74
|       |   `-- 90
|       |       `-- df
|       |-- 76
|       |   `-- 33
|       |       `-- fb
|       |-- 78
|       |   `-- c3
|       |       `-- 4d
|       |-- 80
|       |   `-- 63
|       |       `-- 33
|       |-- 82
|       |   |-- 2d
|       |   |   `-- 65
|       |   `-- 54
|       |       `-- f6
|       |-- 87
|       |   `-- 19
|       |       `-- 64
|       |-- 88
|       |   `-- 65
|       |       `-- 5d
|       |-- 92
|       |   |-- 06
|       |   |   `-- a3
|       |   `-- 9f
|       |       `-- c5
|       |-- 93
|       |   `-- 57
|       |       `-- bd
|       |-- 96
|       |   |-- 3f
|       |   |   `-- 9c
|       |   `-- d3
|       |       `-- 1a
|       |-- 9d
|       |   |-- 95
|       |   |   `-- 0d
|       |   `-- 98
|       |       `-- 05
|       |-- aa
|       |   `-- 6a
|       |       `-- 34
|       |-- b0
|       |   `-- 1c
|       |       `-- 42
|       |-- b6
|       |   `-- af
|       |       `-- 54
|       |-- bd
|       |   `-- 0c
|       |       `-- ca
|       |-- bf
|       |   `-- 9b
|       |       `-- 57
|       |-- c8
|       |   `-- 82
|       |       `-- 5e
|       |-- ca
|       |   `-- ef
|       |       `-- b0
|       |-- cb
|       |   `-- a6
|       |       `-- 4c
|       |-- cc
|       |   `-- 37
|       |       `-- 34
|       |-- ce
|       |   `-- 65
|       |       `-- bc
|       |-- d0
|       |   `-- c7
|       |       `-- 92
|       |-- dc
|       |   `-- a3
|       |       `-- 5f
|       |-- e0
|       |   `-- e5
|       |       `-- 86
|       |-- e4
|       |   `-- b5
|       |       `-- 35
|       |-- e8
|       |   `-- 3c
|       |       `-- 08
|       |-- eb
|       |   `-- 96
|       |       `-- 28
|       |-- ec
|       |   `-- 86
|       |       `-- 88
|       |-- f1
|       |   `-- 0d
|       |       `-- 6a
|       |-- f5
|       |   |-- 06
|       |   |   `-- eb
|       |   `-- 2a
|       |       `-- e5
|       |-- f6
|       |   `-- fc
|       |       `-- 5d
|       `-- f7
|           |-- 94
|           |   `-- b2
|           `-- e3
|               `-- 03
`-- fcrepo.ispn.repo.cache
    |-- dataFedoraRepository
    |   |-- 001638.sst
    |   |-- 001664.sst
    |   |-- 001695.sst
    |   |-- 001696.sst
    |   |-- 001714.sst
    |   |-- 001719.sst
    |   |-- 001720.sst
    |   |-- 001721.sst
    |   |-- 001722.sst
    |   |-- 001723.sst
    |   |-- 001746.sst
    |   |-- 001747.sst
    |   |-- 001748.sst
    |   |-- 001749.sst
    |   |-- 001750.sst
    |   |-- 001751.sst
    |   |-- 001752.sst
    |   |-- 001753.sst
    |   |-- 001754.sst
    |   |-- 001755.sst
    |   |-- 001756.sst
    |   |-- 001757.sst
    |   |-- 001758.sst
    |   |-- 001759.sst
    |   |-- 001760.sst
    |   |-- 001761.sst
    |   |-- 001762.sst
    |   |-- 001763.sst
    |   |-- 001764.sst
    |   |-- 001765.sst
    |   |-- 001766.sst
    |   |-- 001767.sst
    |   |-- 001768.sst
    |   |-- 001769.sst
    |   |-- 001770.sst
    |   |-- 001771.sst
    |   |-- 001772.sst
    |   |-- 001773.sst
    |   |-- 001774.sst
    |   |-- 001775.sst
    |   |-- 001776.sst
    |   |-- 001777.sst
    |   |-- 001778.sst
    |   |-- 001779.sst
    |   |-- 001780.sst
    |   |-- 001781.log
    |   |-- 001783.sst
    |   |-- 001784.sst
    |   |-- CURRENT
    |   |-- LOCK
    |   |-- LOG
    |   |-- LOG.old
    |   `-- MANIFEST-001739
    `-- expiredFedoraRepository
        |-- 000019.log
        |-- CURRENT
        |-- LOCK
        |-- LOG
        |-- LOG.old
        `-- MANIFEST-000018

486 directories, 154 files

[root@plano hydranorth]# du -sh jetty/fcrepo4-data
114M    jetty/fcrepo4-data

The backup:

[pjenkins@dig002tj3 ansible-dev]$ tree fcrepo4-backups/1479331637485-0/
fcrepo4-backups/1479331637485-0/
|-- binaries
|   |-- 02
|   |   `-- 49
|   |       `-- 7a
|   |           `-- 02497af76bb8db956f148f00ec4c8f04f42e5deb.bin
|   |-- 04
|   |   `-- 1e
|   |       `-- 51
|   |           `-- 041e51e6cad2a431d4391348c41f26737b1616d4.bin
|   |-- 09
|   |   |-- 5a
|   |   |   `-- c5
|   |   |       `-- 095ac56968c9753ba7df4a722b87545af9c38afa.bin
|   |   `-- 92
|   |       `-- eb
|   |           `-- 0992eb9df81190e23e81087eec48fe3899a62225.bin
|   |-- 0d
|   |   `-- ff
|   |       `-- 8f
|   |           `-- 0dff8f1894088e2e4fff7874cdef226a8dfb1440.bin
|   |-- 0f
|   |   `-- d8
|   |       `-- 07
|   |           `-- 0fd807ad3b07ae90f9e05e5145c35be74f171f39.bin
|   |-- 12
|   |   `-- e5
|   |       `-- f2
|   |           `-- 12e5f2da18960dc085ca27bec1ae9e3245389cb1.bin
|   |-- 18
|   |   `-- bb
|   |       `-- fa
|   |           `-- 18bbfa150d06ba09fb6fd7729ade051d73ee8a6f.bin
|   |-- 1a
|   |   `-- 1a
|   |       `-- 60
|   |           `-- 1a1a6067b6ffe6c854913ed2e1cb24e4239e85a8.bin
|   |-- 1f
|   |   `-- 33
|   |       `-- 9f
|   |           `-- 1f339f13048f66e9226ad4c7d1b0cf2ba1afc55f.bin
|   |-- 25
|   |   `-- 8c
|   |       `-- 57
|   |           `-- 258c57cc2383b90a8a968e7577136b1cccc17503.bin
|   |-- 26
|   |   `-- 1a
|   |       `-- 3a
|   |           `-- 261a3aa09ad0df7a31c2e4aacc289d08e435cb67.bin
|   |-- 28
|   |   `-- 88
|   |       `-- 9a
|   |           `-- 28889a91353697b80d5ae15d5fc29d545972e4fd.bin
|   |-- 2b
|   |   |-- 16
|   |   |   `-- a0
|   |   |       `-- 2b16a027d1791253f9337bf0dbdfd1fb8b930968.bin
|   |   `-- d0
|   |       `-- 60
|   |           `-- 2bd06000649cc593ea8dab7b6e1d71047419135c.bin
|   |-- 2d
|   |   `-- 0c
|   |       `-- 14
|   |           `-- 2d0c1420ec2f2d0cc69ae796487aaba22cfe37dd.bin
|   |-- 30
|   |   `-- ee
|   |       `-- 29
|   |           `-- 30ee2987141e4b5c508ce8215d5f4420484fc80c.bin
|   |-- 32
|   |   `-- 1e
|   |       `-- 4a
|   |           `-- 321e4ab8e29c973f82b6ea13a89acf3f21cc697f.bin
|   |-- 39
|   |   `-- b9
|   |       `-- af
|   |           `-- 39b9aff56c8e8d4f209d633d0f64d4b3b93727e9.bin
|   |-- 3e
|   |   `-- a5
|   |       `-- 60
|   |           `-- 3ea5606dc42653483442865329eb7c37bee045a9.bin
|   |-- 43
|   |   `-- c0
|   |       `-- 5e
|   |           `-- 43c05eb7c75b209ecefb13eb8138c0afda6e773f.bin
|   |-- 46
|   |   `-- 84
|   |       `-- 5b
|   |           `-- 46845b4aaf1d76ead465ae9a4d799d5ecb9cb607.bin
|   |-- 48
|   |   |-- 3f
|   |   |   `-- 00
|   |   |       `-- 483f00854addea76bdc6f8667e19301dae4925cb.bin
|   |   |-- 9b
|   |   |   `-- 21
|   |   |       `-- 489b21923ab89daf8630f1f277b0e871e8b1acf5.bin
|   |   `-- e7
|   |       `-- b8
|   |           `-- 48e7b865ab5818035b487f6d3bea73a60c1c3bd0.bin
|   |-- 4c
|   |   `-- cd
|   |       `-- 6d
|   |           `-- 4ccd6d2b117fe84617e02bfa45d893861ea6fbff.bin
|   |-- 51
|   |   `-- d5
|   |       `-- 59
|   |           `-- 51d5592149ede90c30c066f4a163ce022e3a942f.bin
|   |-- 52
|   |   `-- 28
|   |       `-- c0
|   |           `-- 5228c09b5cf50892191e8daf0dfff98365fd1b2e.bin
|   |-- 55
|   |   `-- 99
|   |       `-- b5
|   |           `-- 5599b5f8810a79b3ff6f628c3e70f16912f1262e.bin
|   |-- 56
|   |   `-- 5f
|   |       `-- 0d
|   |           `-- 565f0dc708226eee77169c3bc63694c57945b583.bin
|   |-- 5e
|   |   `-- a1
|   |       `-- 81
|   |           `-- 5ea181478c0441bbf2d714561a84cc27aa97b53d.bin
|   |-- 61
|   |   `-- c2
|   |       `-- a7
|   |           `-- 61c2a79c352f492595b8f8a8be5637d07fb512f5.bin
|   |-- 62
|   |   `-- e2
|   |       `-- 2c
|   |           `-- 62e22c0786796f7d3431972b58291989d94af823.bin
|   |-- 68
|   |   `-- fe
|   |       `-- 4a
|   |           `-- 68fe4a0d62f8d5dd98847542681ddb176bcedcf7.bin
|   |-- 6a
|   |   `-- b1
|   |       `-- 0d
|   |           `-- 6ab10d474abf2007b4ce1831ef85b238d3616967.bin
|   |-- 6b
|   |   `-- 84
|   |       `-- dd
|   |           `-- 6b84dd6f8ebd947f855a80ba625fa9700513cc7e.bin
|   |-- 6d
|   |   |-- 61
|   |   |   `-- f8
|   |   |       `-- 6d61f8f67da3b33bea6b2b76c28c323c9a028740.bin
|   |   `-- a3
|   |       `-- de
|   |           `-- 6da3debe7c7474572ee9d7945a4bc8be472ecb44.bin
|   |-- 76
|   |   `-- 33
|   |       `-- fb
|   |           `-- 7633fbc349fb885b0594cd2a8ec9f26b4d581147.bin
|   |-- 78
|   |   `-- c3
|   |       `-- 4d
|   |           `-- 78c34d25d618d5abc74e8a8964b4a11a2d5da288.bin
|   |-- 80
|   |   `-- 63
|   |       `-- 33
|   |           `-- 806333c7b607c89b6ebde0e7b0bd0e940e309b41.bin
|   |-- 82
|   |   |-- 2d
|   |   |   `-- 65
|   |   |       `-- 822d65c8b1250519b30c3a12c06040fb1684092f.bin
|   |   `-- 54
|   |       `-- f6
|   |           `-- 8254f6c73108d3b0a5583b953006d84667704ef3.bin
|   |-- 87
|   |   `-- 19
|   |       `-- 64
|   |           `-- 871964969d8bd9b763b78d7c9da2516cf55c84e1.bin
|   |-- 88
|   |   `-- 65
|   |       `-- 5d
|   |           `-- 88655def21d975ca82106546b91199903e57f45a.bin
|   |-- 92
|   |   |-- 06
|   |   |   `-- a3
|   |   |       `-- 9206a3652eb5574088d74e18baa47bb20f0d3344.bin
|   |   `-- 9f
|   |       `-- c5
|   |           `-- 929fc5afbaa5dd9ad6af4a6a372f61ba7c004367.bin
|   |-- 93
|   |   `-- 57
|   |       `-- bd
|   |           `-- 9357bd6b6028118a97726de13a0ba6c860067d79.bin
|   |-- 96
|   |   `-- 3f
|   |       `-- 9c
|   |           `-- 963f9c65d1b1434ae2f021b3d9613653662e4ba5.bin
|   |-- 9d
|   |   |-- 95
|   |   |   `-- 0d
|   |   |       `-- 9d950d8bb5e359149c84ef52f093dfd872e23915.bin
|   |   `-- 98
|   |       `-- 05
|   |           `-- 9d980592ed7b4fdf4a759357dd54d378eb505ef6.bin
|   |-- aa
|   |   `-- 6a
|   |       `-- 34
|   |           `-- aa6a347faa61ea5cc34622da62a59c38bb2888a4.bin
|   |-- b0
|   |   `-- 1c
|   |       `-- 42
|   |           `-- b01c42236988835677b7bc2e033e2154a6d22bce.bin
|   |-- b6
|   |   `-- af
|   |       `-- 54
|   |           `-- b6af549e27ef082a3d430c034da548a03e194d0d.bin
|   |-- bd
|   |   `-- 0c
|   |       `-- ca
|   |           `-- bd0ccaf7fc57b7c58f9903eb4d99cef567e9ba5b.bin
|   |-- bf
|   |   `-- 9b
|   |       `-- 57
|   |           `-- bf9b57bf40786010ea2ae182c494b6507bcf6851.bin
|   |-- c8
|   |   `-- 82
|   |       `-- 5e
|   |           `-- c8825efe5bd179fb82bbd9ff15c44cb8516c5d77.bin
|   |-- ca
|   |   `-- ef
|   |       `-- b0
|   |           `-- caefb078b07b6c11701278bf7bd61909a10ecc14.bin
|   |-- cc
|   |   `-- 37
|   |       `-- 34
|   |           `-- cc3734011f36514d19fe902fd72dd4baff0f5e31.bin
|   |-- ce
|   |   `-- 65
|   |       `-- bc
|   |           `-- ce65bc9a5aaaf6454ea6c2e49cbd25b2fc75045d.bin
|   |-- d0
|   |   `-- c7
|   |       `-- 92
|   |           `-- d0c792fb39b23e28743e385772f2183f99412d14.bin
|   |-- e0
|   |   `-- e5
|   |       `-- 86
|   |           `-- e0e5868aefb93e6cae8d7928e79ab872adc5646f.bin
|   |-- e4
|   |   `-- b5
|   |       `-- 35
|   |           `-- e4b535af1c5e72d237f9d95c179578211cd01ef2.bin
|   |-- e8
|   |   `-- 3c
|   |       `-- 08
|   |           `-- e83c083e3be192e3f3eb40336dafb7e66ea5571f.bin
|   |-- eb
|   |   `-- 96
|   |       `-- 28
|   |           `-- eb96283497433e9b4a2ccd2d1690e43e22a8debc.bin
|   |-- ec
|   |   `-- 86
|   |       `-- 88
|   |           `-- ec86884f0ce7ba21bc7264684b9d331258e6aac8.bin
|   |-- f1
|   |   `-- 0d
|   |       `-- 6a
|   |           `-- f10d6ae069b585d229dba1a4c6823af4127c4743.bin
|   |-- f5
|   |   |-- 06
|   |   |   `-- eb
|   |   |       `-- f506ebc94fc227f235e1d3ddf4486bac8e942b17.bin
|   |   `-- 2a
|   |       `-- e5
|   |           `-- f52ae576ffac28ad67fa415298b5feb1d4511e93.bin
|   |-- f6
|   |   `-- fc
|   |       `-- 5d
|   |           `-- f6fc5d9f35c26c31b677b2f6be6f1dc4073d83ad.bin
|   `-- f7
|       |-- 94
|       |   `-- b2
|       |       `-- f794b23c0c6fe1083d0ca8b58261a078cd968967.bin
|       `-- e3
|           `-- 03
|               `-- f7e30383d48e6b970972540d58a7750dc8444858.bin
|-- changes
|   `-- summary_of_changes.json
|-- documents_000001.bin.gz
`-- documents_000001.bin.gz.orig

[pjenkins@dig002tj3 ansible-dev]$ du -sh fcrepo4-backups/1479331637485-0/
19M fcrepo4-backups/1479331637485-0/

Fedora 4.7 config:

[root@fedora vagrant]# ps aux | grep java
tomcat    9217  8.8  2.5 5537456 356224 ?      Ssl  12:46   0:25 /usr/bin/java -Dfcrepo.modeshape.configuration=classpath:/config/jdbc-postgresql/repository.json -Dfcrepo.postgresql.username=bypassAdmin -Dfcrepo.postgresql.password=bypassAdmin -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Nov 18 12:46:55 localhost server: INFO: Deploying web application archive /var/lib/tomcat/webapps/fedora.war
Nov 18 12:47:04 localhost server: INFO 12:47:04.430 (ModeShapeRepositoryFactoryBean) Using repo config (classpath): file:/var/lib/tomcat/webapps/fedora/WEB-INF/classes/config/jdbc-postgresql/repository.json
Nov 18 12:47:04 localhost server: INFO 12:47:04.432 (DefaultPropertiesLoader) Loading properties
Nov 18 12:47:04 localhost server: INFO 12:47:04.432 (DefaultPropertiesLoader) com.arjuna.ats.arjuna.common.ObjectStoreEnvironmentBean.default.objectStoreDir = /usr/share/tomcat/fcrepo4-data/com.arjuna.ats.arjuna.common.ObjectStoreEnvironmentBean.default.objectStoreDir
Nov 18 12:47:04 localhost server: INFO 12:47:04.432 (DefaultPropertiesLoader) com.arjuna.ats.arjuna.objectstore.objectStoreDir = /usr/share/tomcat/fcrepo4-data/com.arjuna.ats.arjuna.objectstore.objectStoreDir
Nov 18 12:47:04 localhost server: INFO 12:47:04.432 (DefaultPropertiesLoader) fcrepo.object.directory = /usr/share/tomcat/fcrepo4-data/fcrepo.object.directory
Nov 18 12:47:04 localhost server: INFO 12:47:04.432 (DefaultPropertiesLoader) fcrepo.binary.directory = /usr/share/tomcat/fcrepo4-data/fcrepo.binary.directory
Nov 18 12:47:04 localhost server: INFO 12:47:04.432 (DefaultPropertiesLoader) fcrepo.modeshape.index.directory = /usr/share/tomcat/fcrepo4-data/fcrepo.modeshape.index.directory
Nov 18 12:47:04 localhost server: INFO 12:47:04.432 (DefaultPropertiesLoader) fcrepo.activemq.directory = /usr/share/tomcat/fcrepo4-data/fcrepo.activemq.directory
Nov 18 12:47:05 localhost server: log4j:WARN No appenders could be found for logger (org.jboss.logging).
Nov 18 12:47:05 localhost server: log4j:WARN Please initialize the log4j system properly.
Nov 18 12:47:05 localhost server: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Nov 18 12:47:19 localhost server: Nov 18, 2016 12:47:19 PM org.apache.catalina.util.SessionIdGenerator createSecureRandom
Nov 18 12:47:19 localhost server: INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [10,713] milliseconds.
Nov 18 12:47:20 localhost server: Nov 18, 2016 12:47:20 PM org.apache.catalina.startup.HostConfig deployWAR
Nov 18 12:47:20 localhost server: INFO: Deployment of web application archive /var/lib/tomcat/webapps/fedora.war has finished in 24,702 ms
Nov 18 12:47:20 localhost server: Nov 18, 2016 12:47:20 PM org.apache.coyote.AbstractProtocol start
Nov 18 12:47:20 localhost server: INFO: Starting ProtocolHandler ["http-bio-8080"]
Nov 18 12:47:20 localhost server: Nov 18, 2016 12:47:20 PM org.apache.coyote.AbstractProtocol start
Nov 18 12:47:20 localhost server: INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Nov 18 12:47:20 localhost server: Nov 18, 2016 12:47:20 PM org.apache.catalina.startup.Catalina start
Nov 18 12:47:20 localhost server: INFO: Server startup in 24801 ms

Postgres fcrepo db

fcrepo=# select * from modeshape_repository limit 5;
                         id                         |        last_changed        |                                                                                                                                 

                                                                                    content                                                                                                                        

----------------------------------------------------+----------------------------+---------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------
 87a0a8c7505d64d049283c-f023-4e61-9503-cca2b15134a1 | 2016-11-18 12:51:18.544858 | \x64020000036d6574616461746100400000000269640033000000383761306138633735303564363464303439323833632d663032332d346536312d39353033
2d636361326231353133346131000003636f6e74656e74000c020000026b65790033000000383761306138633735303564363464303439323833632d663032332d346536312d393530332d6363613262313531333461310002706172656e74003300000038376130613
8633735303564363463303464333134302d393633332d343932392d396632612d303839303862613431393564000370726f7065727469657300fb00000003687474703a2f2f7777772e6a63702e6f72672f6a63722f312e3000da00000002637265617465644279000c
00000062797061737341646d696e00037072696d61727954797065001a00000002246e616d65000a0000006e743a666f6c64657200000363726561746564002e000000022464617465001e000000323031362d30362d33305431323a32303a35312e3936342d30363a3
0300000046d6978696e547970657300500000000330004800000002246e616d6500380000007b687474703a2f2f6665646f72612e696e666f2f646566696e6974696f6e732f76342f7265706f7369746f7279237d50616972747265650000000000046368696c647265
6e005d00000003300055000000026b65790033000000383761306138633735303564363432626139383764322d313161662d343039362d396263652d32633762333237656635323100026e616d65000a000000777338353967393366000000036368696c6472656e496
e666f001000000010636f756e740001000000000000
 87a0a8c7505d64c85c6afd-a374-488d-8c90-ddfcbade3451 | 2016-11-18 12:51:20.482559 | \xb2010000036d6574616461746100400000000269640033000000383761306138633735303564363463383563366166642d613337342d343838642d38633930
2d646466636261646533343531000003636f6e74656e74005a010000026b65790033000000383761306138633735303564363463383563366166642d613337342d343838642d386339302d6464666362616465333435310002706172656e74003300000038376130613
8633735303564363463613536636465342d336137642d346463392d613339382d393536323538366132653735000370726f7065727469657300ce00000003687474703a2f2f7777772e6a63702e6f72672f6a63722f312e3000ad00000002637265617465644279000c
00000062797061737341646d696e00037072696d61727954797065004900000002246e616d6500390000007b687474703a2f2f6665646f72612e696e666f2f646566696e6974696f6e732f76342f7265706f7369746f7279237d546f6d6273746f6e650000036372656
1746564002e000000022464617465001e000000323031362d30372d30365431323a30303a32302e3235322d30363a3030000000000000
 87a0a8c7505d648b9a739d-86bd-4bf2-a50c-771351505cc4 | 2016-11-18 12:51:18.544858 | \x5d020000036d6574616461746100400000000269640033000000383761306138633735303564363438623961373339642d383662642d346266322d61353063
2d373731333531353035636334000003636f6e74656e740005020000026b65790033000000383761306138633735303564363438623961373339642d383662642d346266322d613530632d3737313335313530356363340002706172656e74003300000038376130613
8633735303564363462613461383739662d653062612d346361302d616531392d323032616136303235313030000370726f7065727469657300fb00000003687474703a2f2f7777772e6a63702e6f72672f6a63722f312e3000da00000002637265617465644279000c
00000062797061737341646d696e00037072696d61727954797065001a00000002246e616d65000a0000006e743a666f6c64657200000363726561746564002e000000022464617465001e000000323031362d30362d33305431303a32393a34342e3438372d30363a3
0300000046d6978696e547970657300500000000330004800000002246e616d6500380000007b687474703a2f2f6665646f72612e696e666f2f646566696e6974696f6e732f76342f7265706f7369746f7279237d50616972747265650000000000046368696c647265
6e00560000000330004e000000026b65790033000000383761306138633735303564363437383233376535622d626335612d346233372d386464312d32326538343566313833326100026e616d6500030000006566000000036368696c6472656e496e666f001000000
010636f756e740001000000000000
 87a0a8c7505d64e27a64e0-1ac6-40c1-8b89-348d0ce17210 | 2016-11-18 12:51:18.544858 | \x50030000036d6574616461746100400000000269640033000000383761306138633735303564363465323761363465302d316163362d343063312d38623839
2d333438643063653137323130000003636f6e74656e7400f8020000026b65790033000000383761306138633735303564363465323761363465302d316163362d343063312d386238392d3334386430636531373231300002706172656e74003300000038376130613
8633735303564363464373535366539342d343731372d343066392d396439332d336634393030656666343637000370726f70657274696573001902000003687474703a2f2f7777772e6a63702e6f72672f6a63722f312e3000b1010000026c6173744d6f6469666965
644279000c00000062797061737341646d696e000275756964002500000065323761363465302d316163362d343063312d386238392d3334386430636531373231300002637265617465644279000c00000062797061737341646d696e00037072696d6172795479706
5001a00000002246e616d65000a0000006e743a666f6c64657200000363726561746564002e000000022464617465001e000000323031362d30362d33305431303a32323a34352e3537382d30363a30300000046d6978696e5479706573009c00000003300049000000
02246e616d6500390000007b687474703a2f2f6665646f72612e696e666f2f646566696e6974696f6e732f76342f7265706f7369746f7279237d436f6e7461696e657200000331004800000002246e616d6500380000007b687474703a2f2f6665646f72612e696e666
f2f646566696e6974696f6e732f76342f7265706f7369746f7279237d5265736f75726365000000036c6173744d6f646966696564002e000000022464617465001e000000323031362d30362d33305431303a32323a34362e3932362d30363a303000000003696e666f
3a6665646f72612f6665646f72612d73797374656d3a6465662f6d6f64656c230021000000046861734d6f64656c001200000002300006000000426174636800000000037265666572726572730048000000037765616b003d000000103837613061386337353035643
63465386538623530372d346263312d343761662d393333642d353131623235333339646363000100000000000000
 87a0a8c7505d646f2c981f-fb2f-4fb6-9a96-8f43c5e1db85 | 2016-11-18 12:51:18.544858 | \x7f020000036d6574616461746100400000000269640033000000383761306138633735303564363436663263393831662d666232662d346662362d39613936
2d386634336335653164623835000003636f6e74656e740027020000026b65790033000000383761306138633735303564363436663263393831662d666232662d346662362d396139362d3866343363356531646238350002706172656e74003300000038376130613
8633735303564363432356433333865312d646638392d346263612d383339382d303566366437626130376363000370726f7065727469657300fb00000003687474703a2f2f7777772e6a63702e6f72672f6a63722f312e3000da00000002637265617465644279000c
00000062797061737341646d696e00037072696d61727954797065001a00000002246e616d65000a0000006e743a666f6c64657200000363726561746564002e000000022464617465001e000000323031362d30362d33305430313a33393a33332e3434352d30363a3
0300000046d6978696e547970657300500000000330004800000002246e616d6500380000007b687474703a2f2f6665646f72612e696e666f2f646566696e6974696f6e732f76342f7265706f7369746f7279237d50616972747265650000000000046368696c647265
6e007800000003300070000000026b65790033000000383761306138633735303564363465373730393936652d376334662d343465362d613763612d36393833643731633461383400026e616d65002500000039323331306664662d663437302d343064312d6234326
32d613730316665363034663430000000036368696c6472656e496e666f001000000010636f756e740001000000000000
(5 rows)

fcrepo=# select count(*) from modeshape_repository;
 count 
-------
 55884
(1 row)

fcrepo=# SELECT pg_size_pretty( pg_database_size( current_database() ) ) As human_size
fcrepo-#     , pg_database_size( current_database() ) As raw_size;
 human_size | raw_size 
------------+----------
 64 MB      | 66738712
(1 row)

and the data directory

[root@fedora vagrant]# tree /usr/share/tomcat/fcrepo4-data/
/usr/share/tomcat/fcrepo4-data/
|-- com.arjuna.ats.arjuna.common.ObjectStoreEnvironmentBean.default.objectStoreDir
|   `-- ShadowNoFileLockStore
|       `-- defaultStore
|-- com.arjuna.ats.arjuna.objectstore.objectStoreDir
|   `-- ShadowNoFileLockStore
|       `-- defaultStore
|           `-- Recovery
|               `-- TransactionStatusManager
|                   `-- 0_ffff7f000001_9625_582f5ab9_0
|-- fcrepo.activemq.directory
|   |-- db-1.log
|   |-- db.data
|   |-- db.redo
|   `-- lock
`-- fcrepo.binary.directory
    |-- 02
    |   `-- 49
    |       `-- 7a
    |           `-- 02497af76bb8db956f148f00ec4c8f04f42e5deb
    |-- 04
    |   `-- 1e
    |       `-- 51
    |           `-- 041e51e6cad2a431d4391348c41f26737b1616d4
    |-- 09
    |   |-- 5a
    |   |   `-- c5
    |   |       `-- 095ac56968c9753ba7df4a722b87545af9c38afa
    |   `-- 92
    |       `-- eb
    |           `-- 0992eb9df81190e23e81087eec48fe3899a62225
    |-- 0d
    |   `-- ff
    |       `-- 8f
    |           `-- 0dff8f1894088e2e4fff7874cdef226a8dfb1440
    |-- 0f
    |   `-- d8
    |       `-- 07
    |           `-- 0fd807ad3b07ae90f9e05e5145c35be74f171f39
    |-- 12
    |   `-- e5
    |       `-- f2
    |           `-- 12e5f2da18960dc085ca27bec1ae9e3245389cb1
    |-- 18
    |   `-- bb
    |       `-- fa
    |           `-- 18bbfa150d06ba09fb6fd7729ade051d73ee8a6f
    |-- 1a
    |   `-- 1a
    |       `-- 60
    |           `-- 1a1a6067b6ffe6c854913ed2e1cb24e4239e85a8
    |-- 1f
    |   `-- 33
    |       `-- 9f
    |           `-- 1f339f13048f66e9226ad4c7d1b0cf2ba1afc55f
    |-- 25
    |   `-- 8c
    |       `-- 57
    |           `-- 258c57cc2383b90a8a968e7577136b1cccc17503
    |-- 26
    |   `-- 1a
    |       `-- 3a
    |           `-- 261a3aa09ad0df7a31c2e4aacc289d08e435cb67
    |-- 28
    |   `-- 88
    |       `-- 9a
    |           `-- 28889a91353697b80d5ae15d5fc29d545972e4fd
    |-- 2b
    |   |-- 16
    |   |   `-- a0
    |   |       `-- 2b16a027d1791253f9337bf0dbdfd1fb8b930968
    |   `-- d0
    |       `-- 60
    |           `-- 2bd06000649cc593ea8dab7b6e1d71047419135c
    |-- 2d
    |   `-- 0c
    |       `-- 14
    |           `-- 2d0c1420ec2f2d0cc69ae796487aaba22cfe37dd
    |-- 30
    |   `-- ee
    |       `-- 29
    |           `-- 30ee2987141e4b5c508ce8215d5f4420484fc80c
    |-- 32
    |   `-- 1e
    |       `-- 4a
    |           `-- 321e4ab8e29c973f82b6ea13a89acf3f21cc697f
    |-- 39
    |   `-- b9
    |       `-- af
    |           `-- 39b9aff56c8e8d4f209d633d0f64d4b3b93727e9
    |-- 3e
    |   `-- a5
    |       `-- 60
    |           `-- 3ea5606dc42653483442865329eb7c37bee045a9
    |-- 43
    |   `-- c0
    |       `-- 5e
    |           `-- 43c05eb7c75b209ecefb13eb8138c0afda6e773f
    |-- 46
    |   `-- 84
    |       `-- 5b
    |           `-- 46845b4aaf1d76ead465ae9a4d799d5ecb9cb607
    |-- 48
    |   |-- 3f
    |   |   `-- 00
    |   |       `-- 483f00854addea76bdc6f8667e19301dae4925cb
    |   |-- 9b
    |   |   `-- 21
    |   |       `-- 489b21923ab89daf8630f1f277b0e871e8b1acf5
    |   `-- e7
    |       `-- b8
    |           `-- 48e7b865ab5818035b487f6d3bea73a60c1c3bd0
    |-- 4c
    |   `-- cd
    |       `-- 6d
    |           `-- 4ccd6d2b117fe84617e02bfa45d893861ea6fbff
    |-- 51
    |   `-- d5
    |       `-- 59
    |           `-- 51d5592149ede90c30c066f4a163ce022e3a942f
    |-- 52
    |   `-- 28
    |       `-- c0
    |           `-- 5228c09b5cf50892191e8daf0dfff98365fd1b2e
    |-- 55
    |   `-- 99
    |       `-- b5
    |           `-- 5599b5f8810a79b3ff6f628c3e70f16912f1262e
    |-- 56
    |   `-- 5f
    |       `-- 0d
    |           `-- 565f0dc708226eee77169c3bc63694c57945b583
    |-- 5e
    |   `-- a1
    |       `-- 81
    |           `-- 5ea181478c0441bbf2d714561a84cc27aa97b53d
    |-- 61
    |   `-- c2
    |       `-- a7
    |           `-- 61c2a79c352f492595b8f8a8be5637d07fb512f5
    |-- 62
    |   `-- e2
    |       `-- 2c
    |           `-- 62e22c0786796f7d3431972b58291989d94af823
    |-- 68
    |   `-- fe
    |       `-- 4a
    |           `-- 68fe4a0d62f8d5dd98847542681ddb176bcedcf7
    |-- 6a
    |   `-- b1
    |       `-- 0d
    |           `-- 6ab10d474abf2007b4ce1831ef85b238d3616967
    |-- 6b
    |   `-- 84
    |       `-- dd
    |           `-- 6b84dd6f8ebd947f855a80ba625fa9700513cc7e
    |-- 6d
    |   |-- 61
    |   |   `-- f8
    |   |       `-- 6d61f8f67da3b33bea6b2b76c28c323c9a028740
    |   `-- a3
    |       `-- de
    |           `-- 6da3debe7c7474572ee9d7945a4bc8be472ecb44
    |-- 76
    |   `-- 33
    |       `-- fb
    |           `-- 7633fbc349fb885b0594cd2a8ec9f26b4d581147
    |-- 78
    |   `-- c3
    |       `-- 4d
    |           `-- 78c34d25d618d5abc74e8a8964b4a11a2d5da288
    |-- 80
    |   `-- 63
    |       `-- 33
    |           `-- 806333c7b607c89b6ebde0e7b0bd0e940e309b41
    |-- 82
    |   |-- 2d
    |   |   `-- 65
    |   |       `-- 822d65c8b1250519b30c3a12c06040fb1684092f
    |   `-- 54
    |       `-- f6
    |           `-- 8254f6c73108d3b0a5583b953006d84667704ef3
    |-- 87
    |   `-- 19
    |       `-- 64
    |           `-- 871964969d8bd9b763b78d7c9da2516cf55c84e1
    |-- 88
    |   `-- 65
    |       `-- 5d
    |           `-- 88655def21d975ca82106546b91199903e57f45a
    |-- 92
    |   |-- 06
    |   |   `-- a3
    |   |       `-- 9206a3652eb5574088d74e18baa47bb20f0d3344
    |   `-- 9f
    |       `-- c5
    |           `-- 929fc5afbaa5dd9ad6af4a6a372f61ba7c004367
    |-- 93
    |   `-- 57
    |       `-- bd
    |           `-- 9357bd6b6028118a97726de13a0ba6c860067d79
    |-- 96
    |   `-- 3f
    |       `-- 9c
    |           `-- 963f9c65d1b1434ae2f021b3d9613653662e4ba5
    |-- 9d
    |   |-- 95
    |   |   `-- 0d
    |   |       `-- 9d950d8bb5e359149c84ef52f093dfd872e23915
    |   `-- 98
    |       `-- 05
    |           `-- 9d980592ed7b4fdf4a759357dd54d378eb505ef6
    |-- aa
    |   `-- 6a
    |       `-- 34
    |           `-- aa6a347faa61ea5cc34622da62a59c38bb2888a4
    |-- b0
    |   `-- 1c
    |       `-- 42
    |           `-- b01c42236988835677b7bc2e033e2154a6d22bce
    |-- b6
    |   `-- af
    |       `-- 54
    |           `-- b6af549e27ef082a3d430c034da548a03e194d0d
    |-- bd
    |   `-- 0c
    |       `-- ca
    |           `-- bd0ccaf7fc57b7c58f9903eb4d99cef567e9ba5b
    |-- bf
    |   `-- 9b
    |       `-- 57
    |           `-- bf9b57bf40786010ea2ae182c494b6507bcf6851
    |-- c8
    |   `-- 82
    |       `-- 5e
    |           `-- c8825efe5bd179fb82bbd9ff15c44cb8516c5d77
    |-- ca
    |   `-- ef
    |       `-- b0
    |           `-- caefb078b07b6c11701278bf7bd61909a10ecc14
    |-- cc
    |   `-- 37
    |       `-- 34
    |           `-- cc3734011f36514d19fe902fd72dd4baff0f5e31
    |-- ce
    |   `-- 65
    |       `-- bc
    |           `-- ce65bc9a5aaaf6454ea6c2e49cbd25b2fc75045d
    |-- d0
    |   `-- c7
    |       `-- 92
    |           `-- d0c792fb39b23e28743e385772f2183f99412d14
    |-- e0
    |   `-- e5
    |       `-- 86
    |           `-- e0e5868aefb93e6cae8d7928e79ab872adc5646f
    |-- e4
    |   `-- b5
    |       `-- 35
    |           `-- e4b535af1c5e72d237f9d95c179578211cd01ef2
    |-- e8
    |   `-- 3c
    |       `-- 08
    |           `-- e83c083e3be192e3f3eb40336dafb7e66ea5571f
    |-- eb
    |   `-- 96
    |       `-- 28
    |           `-- eb96283497433e9b4a2ccd2d1690e43e22a8debc
    |-- ec
    |   `-- 86
    |       `-- 88
    |           `-- ec86884f0ce7ba21bc7264684b9d331258e6aac8
    |-- f1
    |   `-- 0d
    |       `-- 6a
    |           `-- f10d6ae069b585d229dba1a4c6823af4127c4743
    |-- f5
    |   |-- 06
    |   |   `-- eb
    |   |       `-- f506ebc94fc227f235e1d3ddf4486bac8e942b17
    |   `-- 2a
    |       `-- e5
    |           `-- f52ae576ffac28ad67fa415298b5feb1d4511e93
    |-- f6
    |   `-- fc
    |       `-- 5d
    |           `-- f6fc5d9f35c26c31b677b2f6be6f1dc4073d83ad
    `-- f7
        |-- 94
        |   `-- b2
        |       `-- f794b23c0c6fe1083d0ca8b58261a078cd968967
        `-- e3
            `-- 03
                `-- f7e30383d48e6b970972540d58a7750dc8444858

216 directories, 77 files
[root@fedora vagrant]# du -sh /usr/share/tomcat/fcrepo4-data/
2.1M    /usr/share/tomcat/fcrepo4-data/

From @pgwillia on November 24, 2016 23:38

Summarized in

--------- Forwarded message ----------
From: Tricia Jenkins
Date: Thu, Nov 24, 2016 at 2:29 PM
Subject: Fedora upgrade
To: Guanwen Zhang
Cc: LIB Office of Chief Librarian - DIT-DevTeam , Kenton Good

From @pgwillia on November 30, 2016 17:7

Test restore on gillingham2 failing after ~2hrs.

java.sql.BatchUpdateException: Batch entry 171 INSERT INTO MODESHAPE_REPOSITORY (ID, CONTENT) VALUES ('87a0a8c317f1e7jcr:system/mode:metadata', ?) was aborted.  Call getNextException to see the cause.
java.sql.BatchUpdateException: Batch entry 0 INSERT INTO MODESHAPE_REPOSITORY (ID, CONTENT) VALUES ('87a0a8c317f1e7mode:locks', ?) was aborted.  Call getNextException to see the cause.

[http response]

SEVERE: Exception sending context initialized event to listener instance of class org.fcrepo.http.commons.FedoraContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'modeshapeRepofactory': Invocation of init method failed; nested exception is org.fcrepo.kernel.api.exception.RepositoryRunt
imeException: org.modeshape.jcr.value.ValueFormatException: Error converting "fedora:Resource" from String to a Name
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1560)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:540)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:706)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
        at org.fcrepo.http.commons.FedoraContextLoaderListener.contextInitialized(FedoraContextLoaderListener.java:40)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1083)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1880)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.fcrepo.kernel.api.exception.RepositoryRuntimeException: org.modeshape.jcr.value.ValueFormatException: Error converting "fedora:Resource" from String to a Name
        at org.fcrepo.kernel.modeshape.spring.ModeShapeRepositoryFactoryBean.buildRepository(ModeShapeRepositoryFactoryBean.java:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:349)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:300)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:133)
        ... 28 more
Caused by: org.modeshape.jcr.value.ValueFormatException: Error converting "fedora:Resource" from String to a Name
        at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:157)
        at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:46)
        at org.modeshape.jcr.cache.document.DocumentTranslator.childReferenceFrom(DocumentTranslator.java:966)
        at org.modeshape.jcr.cache.document.ImmutableChildReferences$Medium.<init>(ImmutableChildReferences.java:220)
        at org.modeshape.jcr.cache.document.ImmutableChildReferences.create(ImmutableChildReferences.java:53)
        at org.modeshape.jcr.cache.document.DocumentTranslator.getChildReferences(DocumentTranslator.java:887)
        at org.modeshape.jcr.cache.document.LazyCachedNode.getChildReferences(LazyCachedNode.java:449)
        at org.modeshape.jcr.cache.document.SessionNode.getChildReferences(SessionNode.java:1132)
        at org.modeshape.jcr.SystemContent.store(SystemContent.java:309)
        at org.modeshape.jcr.SystemContent.store(SystemContent.java:286)
at org.modeshape.jcr.RepositoryNodeTypeManager.registerNodeTypes(RepositoryNodeTypeManager.java:525)
        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:1135)
        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:936)
        at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:367)
        at org.modeshape.jcr.JcrRepository.getStartupProblems(JcrRepository.java:285)
        at org.fcrepo.kernel.modeshape.spring.ModeShapeRepositoryFactoryBean.buildRepository(ModeShapeRepositoryFactoryBean.java:76)
        ... 35 more
Caused by: org.modeshape.jcr.value.NamespaceException: There is no namespace registered for the prefix "fedora"
        at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:148)
        ... 50 more

[/usr/share/tomcat/log/localhost.log]

From @pgwillia on November 30, 2016 18:9

--- fedora-4.7.0/WEB-INF/lib/fedora-node-types.cnd  2016-10-31 12:22:44.000000000 -0600
+++ fedora-node-types.cnd   2015-02-05 16:56:30.000000000 -0700
@@ -26,7 +26,6 @@
 <rdfs = 'http://www.w3.org/2000/01/rdf-schema#'>
 <premis = 'http://www.loc.gov/premis/rdf/v1#'>
 <ldp = 'http://www.w3.org/ns/ldp#'>
-<ebucore = 'http://www.ebu.ch/metadata/ontologies/ebucore/ebucore#'>

 /*
  * A fedora namespace for properties a user may set on a node that may
@@ -40,7 +39,6 @@
  */
 [fedora:Resource] > mix:created, mix:lastModified, mix:referenceable mixin
   - rdf:type (URI) multiple
-  - fedora:lastModified (DATE)
   - * (undefined) multiple
   - * (undefined)

@@ -68,13 +66,11 @@
  * Some content that can have a checksum
  */
 [fedora:Binary] > fedora:Resource mixin
-  - ebucore:filename (STRING)
-  - ebucore:hasMimeType (STRING)
+  - premis:hasOriginalName (STRING)
   - premis:hasSize (LONG) COPY
-  - premis:hasMessageDigest (URI) COPY multiple
+  - fedora:digest (URI) COPY

-[fedora:Skolem] > mix:referenceable mixin
-  - fedora:lastModified (DATE)
+[fedora:Blanknode] > mix:referenceable mixin

 [fedora:Pairtree] mixin

From @pgwillia on November 30, 2016 21:32

Same error message as Case 4: Repository Restoration Failed - LevelDB API Verification Succeeded

From @leahvanderjagt on January 23, 2017 21:33

@pbinkley can the dev team please groom this issue and add to IR if high priority, keep in hydranorth2 if in scope of current work, or future if is not immediately pressing

Digest of previous comments and links to external references to outline a way forward.

Course upgrade path:

[ ] stop Tomcat/Jetty [1]
[ ] run the fcrepo4-upgrade-utils-4.3.0 tool to migrate technical metadata properties based on changes made in May 2015 [1] [2]
[ ] upgrade to the 4.6.1+ patch release (drop-in) [3]
[ ] fcr:backup [4]
[ ] upgrade to the 4.7.2 release (empty data dir needs fcr:backup/fcr:restore to accommodate the object persistence layer changes) [4]
[ ] fcr:restore [4]
[ ] Verify
- walking a repository and verifying are resources response successfully - both before and after fcr:backup/fcr:restore
- https://github.com/awoods/fcrepo-java-client-etc
- e.g., http://localhost:{$PORT}/fcrepo/rest/prod (fcrepo4-docker)

References: [1] https://groups.google.com/d/msg/fedora-tech/9eik_p0zPWY/BcglXSSEAAAJ [2] https://github.com/fcrepo4-exts/fcrepo4-upgrade-utils [3] https://wiki.duraspace.org/display/FF/Fedora+4.6.1+Release+Notes [4] https://wiki.duraspace.org/display/FF/Fedora+4.7.0+Release+Notes#Fedora4.7.0ReleaseNotes-Summary [5] https://wiki.duraspace.org/display/FF/Fedora+Upgrade+Notes

Todo and open questions:

[ ] test with large dataset to verify Nov. comments
[ ] how to verify objects?
[ ] test OAIProvider
[ ] what should the Fedora setup use? PostgresDB? Triplestore (Fuseki or Blazegraph)? Or as is for now?
[ ] Hydra (Ruby/Rails) config changes
[ ] Hydra (Ruby/Rails) gem updates

Notes:

Regarding testing the course update path with era scale dataset:

~3 hours to run fcrepo4-upgrade-utils-4.3.0 (Java process used 1.6GB, on 16GB RAM, 1TD disk desktop)
~16 hours fcr:backup on Fcrepo_4.6.1 (https://github.com/fcrepo4-labs/fcrepo4-docker) using era_2017-03-20 era filesystem fedora_data directory from previous step (returned response body with the directory containing backup files). No error messages in the Tomcat log.
fcr:restore started ~1:00pm Friday 2017-03-24 into Fedora 4.7.1 (https://github.com/fcrepo4-labs/fcrepo4-docker) starting with empty fedora_data directory. No change to object store from Docker image (i.e., not using PostgreSQL nor MySql).
2017-04-11: walking a repository and verifying are resources response successfully - both before and after fcr:backup/fcr:restore
- https://github.com/awoods/fcrepo-java-client-etc
- output before and after returned the same result with the same number of results and same ids.

HydraNorth flagged bugs with Fedora 4.7

unknown response format. got 'text/turtle;charset=utf-8', but was expecting 'text/turtle'
- http://localhost:3000/files/0p0966881/edit
- might be related to the following change in Fedora: https://groups.google.com/forum/#!searchin/fedora-tech/charset$20content-type%7Csort:relevance/fedora-tech/zpFrW_xFdLI/6DDGPGja9G0J

RuntimeError in GenericFilesController#edit

unknown response format. got 'text/turtle;charset=utf-8', but was expecting 'text/turtle'

Extracted source (around line #68):

          raise "unexpected return value #{resp.status} for when getting datastream content at #{uri}\n\t#{resp.body}"
        elsif resp.headers['content-type'] != 'text/turtle'
          raise "unknown response format. got '#{resp.headers['content-type']}', but was expecting 'text/turtle'"
        end
        resp.body
      end

fcr:restore: tomcat log from ~1:00pm Friday 2017-03-24

Not sure if the restore worked or not (checked on date 2017-03-24). "curl" returned without error. There is a bunch of stuff in Fedora? However the following error occurred in catalina.out:

ERROR 03:38:49.761 (RepositoryIndexManager) Error while refreshing index definitions for the "repo" repository org.modeshape.jcr.value.ValueFormatException: Error converting "webacl:agent" from String to a Name at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:157) at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:85) at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:46) at org.modeshape.jcr.value.basic.AbstractValueFactory.create(AbstractValueFactory.java:96) at org.modeshape.jcr.SystemContent.readIndexColumnDefinition(SystemContent.java:550) at org.modeshape.jcr.SystemContent.readIndexDefinition(SystemContent.java:540) at org.modeshape.jcr.SystemContent.readAllIndexDefinitions(SystemContent.java:576) at org.modeshape.jcr.RepositoryIndexManager.readIndexDefinitions(RepositoryIndexManager.java:811) at org.modeshape.jcr.RepositoryIndexManager.initialize(RepositoryIndexManager.java:170) at org.modeshape.jcr.RepositoryQueryManager.initialize(RepositoryQueryManager.java:105) at org.modeshape.jcr.JcrRepository$RunningState.completeInitialization(JcrRepository.java:1274) at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:370) at org.modeshape.jcr.JcrRepository.start(JcrRepository.java:299) at org.modeshape.jcr.JcrRepository.completeRestore(JcrRepository.java:521) at org.modeshape.jcr.BackupService.restoreRepository(BackupService.java:169) at org.modeshape.jcr.JcrRepositoryManager.restoreRepository(JcrRepositoryManager.java:87) at org.modeshape.jcr.JcrRepositoryManager.restoreRepository(JcrRepositoryManager.java:81) at org.fcrepo.kernel.modeshape.services.RepositoryServiceImpl.restoreRepository(RepositoryServiceImpl.java:145) at org.fcrepo.http.api.repository.FedoraRepositoryRestore.runRestore(FedoraRepositoryRestore.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:442) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1083) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:640) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: org.modeshape.jcr.value.NamespaceException: There is no namespace registered for the prefix "webacl" at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:148) ... 64 common frames omitted

Note that as far as I know (I've checked with Henry) we haven't yet successfully fcr:restored a full fcr:backup of ERA. That part may take some experimentation.

Description of the testing setup:

Dataset: filesystem copy of the fedora_data directory on era.library.ualberta.ca; approx. 300GB (with binary files and metadata)

Environment: desktop Ubuntu 16.04 with Fedora running within Docker images - the following Docker images were employed: https://github.com/fcrepo4-labs/fcrepo4-docker.

The general setup, use a Docker image for each version of Fedora, one for 4.2, 4.3 (testing metadata change script)[https://github.com/fcrepo4-exts/fcrepo4-upgrade-utils], 4.6.2 (to export), and 4.7.2 (to import). Each exposes different ports for Fedora so can be run at the same time and -v to mount a local filesystem volume within the Docker container (containing the Fedora data files so there is no need to copy into the Docker container).

The more granular upgrade path steps:

(1) Create Fedora 4.2 Docker instance, attach to the filesystem copy of the ERA fedora_data dir and manually inspect the /rest endpoint for data

docker run -v /dev/data_2017-03-21/:/mnt/ingest -it -p 8080:8080 -p 9080:9080 -d yinlinchen/fcrepo4-docker:4.2.0

(2) run the fcrepo4-upgrade-utils-4.3.0 tool to migrate technical metadata properties based on changes made in May 2015 to Fedora [1] [2]

java -Dfcrepo.home=/fedora_data/ -jar /fcrepo4-upgrade-utils/target/fcrepo-upgrade-utils-4.3.1-SNAPSHOT.jar

(3) Create Fedora 4.6.2 Docker instance, attach to the filesystem copy of the fedora_data dir from (2) above and manually inspect the /rest endpoint for data (also stop the Fedora 4.2 Docker instance).

docker run -v /fedora_data:/mnt/ingest -it -p 8086:8080 -p 9086:9080 -d yinlinchen/fcrepo4-docker:4.6.2

(4) fcr:backup (or try new import/export utility)

curl -X POST http://x:x@localhost:8986/fedora/rest/fcr:backup - response contains directory

(5) Create Fedora 4.7.1 Docker instance and attach to the filesystem copy of the fedora_data dir from (2) above and manually inspect the /rest endpoint for data (also stop the Fedora 4.2 Docker instance).

docker run -v /fedora_data_4.7:/mnt/ingest -it -p 8087:8080 -p 9087:9080 -d yinlinchen/fcrepo4-docker:4.7.2

(6) Import into an empty Fedora 4.7 install [4]

curl -X POST -d '/path/to/dir/created/by/fcr:import/' 'http://xxx:xxx@localhost:8080/fcrepo/rest/fcr:restore'

(7) Test and verify

In the future - try the 4.6.2 to current (4.7.2+) Fedora migration using (import/export tool)[https://github.com/fcrepo4-labs/fcrepo-import-export] and the (associated verify script)[https://github.com/fcrepo4-labs/fcrepo-import-export-verify]

[1] https://groups.google.com/d/msg/fedora-tech/9eik_p0zPWY/BcglXSSEAAAJ [2] https://github.com/fcrepo4-exts/fcrepo4-upgrade-utils [3] https://wiki.duraspace.org/display/FF/Fedora+4.6.1+Release+Notes [4] https://wiki.duraspace.org/display/FF/Fedora+4.7.0+Release+Notes#Fedora4.7.0ReleaseNotes-Summary [5] https://wiki.duraspace.org/display/FF/Fedora+Upgrade+Notes

Digest of process

Digest of test process and links to external references to outline a way forward.

Upgrade path:

[ ] stop Tomcat/Jetty [1]
[ ] run the fcrepo4-upgrade-utils-4.3.0 tool to migrate technical metadata properties based on changes made in May 2015 to Fedora [1] [2]
[ ] upgrade to the 4.6.1+ patch release (drop-in Fedora war no need for backup/restore for 4.6.1) [3]
[ ] restart Tomcat/Jetty with 4.6.1 and test [1]
[ ] fcr:backup [4]
[ ] stop Tomcat/Jetty 4.6
[ ] install the 4.7.2 release (empty data directory needed for fcr:backup/fcr:restore to accommodate the object persistence layer changes (ModeShape major version update can't use existing data directory needs to be rebuilt from empty) ) [4]
[ ] start Tomcat/Jetty 4.7.2
[ ] fcr:restore [4]
[ ] Verify steps as per #148 and #149

In the future - try the 4.6.2 to current (4.7.2+) Fedora migration using the import/export tool and the associated verify script instead of (fcr:backup/fcr:restore)

fcr:backup/fcr:restore are deprecated as per https://jira.duraspace.org/browse/FCREPO-2044?src=confmacro
"[fcr:backup/fcr:restore]... capability will continue to be available until the Import/Export functionality fully supports backup/restore use cases." - https://wiki.duraspace.org/display/FF/Fedora+4.7.1+Release+Notes

In the future try Fedora 4.8+

"...resources with a large number of members. Danny Bernstein... found that a combination of parallel processing streams and increasing the size of the ModeShape cache produced significant performance gains of 5-8x". Ref:
as of 2017-04-21, uncertain if Fedora 4.8 requires a backup/restore of data

Notes: Regarding testing the upgrade path with era scale dataset:

~3 hours to run fcrepo4-upgrade-utils-4.3.0 (Java process used 1.6GB, on 16GB RAM, 1TD disk desktop)
~16 hours fcr:backup on Fcrepo_4.6.1 (https://github.com/fcrepo4-labs/fcrepo4-docker) using era_2017-03-20 era filesystem fedora_data directory from previous step (returned response body with the directory containing backup files). No error messages in the Tomcat log.
fcr:restore started ~1:00pm Friday 2017-03-24 into Fedora 4.7.1 (https://github.com/fcrepo4-labs/fcrepo4-docker) starting with empty fedora_data directory. No change to object store from Docker image (i.e., not using PostgreSQL nor MySql).
2017-04-11: walking a repository and verifying are resources response successfully - both before and after fcr:backup/fcr:restore
- https://github.com/awoods/fcrepo-java-client-etc
- output before and after returned the same result with the same number of results and same ids.

ualbertalib / Hydranorth2