Sufia7 [CCM]: Infrastructure: Create/Migrate test data for collection/solr join behaviour testing

[mbarnett]

We'll need a significant amount of collections & items in collections to test solr join performance for #1126.

estimate: 4

[pbinkley]

Let's generate artificial items and collections rather than using real metadata for this phase of testing.

[pgwillia]

Use solr_wrapper --cloud -d solr/config/ --collection_name hydra-development when creating the collection.

Use http://sufia7.library.ualberta.ca:8983/solr/admin/collections?action=SPLITSHARD&collection=hydra-development&shard=shard1 to ensure the index has multiple shards. This is significant because previously, in the SolrCloud environment Collection counts and searching failed.

[pgwillia]

Hmm. Job doesn't finish

[qtp110718392-16] WARN org.eclipse.jetty.server.HttpChannel - Could not send response error 500: javax.servlet.ServletException: javax.servlet.ServletException: A MultiException has 4 exceptions.  They are:
1. java.lang.OutOfMemoryError: Java heap space
2. java.lang.IllegalArgumentException: While attempting to create a Proxy for javax.servlet.http.HttpServletRequest in proxiable scope org.glassfish.jersey.process.internal.RequestScoped an error occured while creating the proxy
3. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of org.fcrepo.http.commons.session.SessionProvider errors were found
4. java.lang.IllegalStateException: Unable to perform operation: resolve on org.fcrepo.http.commons.session.SessionProvider

[pgwillia]

Upgraded to Fedora 4.6.0 RC. Improved performance for the smaller sample. While seeding the the larger sample got a new error

[root@sufia7 sufia7]# rake db:seed
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fbbf5052097, pid=20062, tid=0x00007fbbf49f2700
#
# JRE version: OpenJDK Runtime Environment (8.0_101-b13) (build 1.8.0_101-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.101-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v  ~StubRoutines::jbyte_disjoint_arraycopy
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /var/www/sites/sufia7/hs_err_pid20062.log
rake aborted!

The file mentioned was created but is empty.

[mbarnett]

That might be transient, so it might disappear when you re-run it. It shouldn't be possible to generate a SIGBUS in pure Java, so that's essentially a bug in the JRE itself (or possibly a hiccup in your VM somehow).

[pgwillia]

@mbarnett it also happened overnight last night when I ran a similar large seeding job. I missed the error message but a hs*.log file was generated and the java process running fedora had been terminated. Investigating what might be a underlying cause. Preliminary searches suggest disk space.

[atz]

java.lang.OutOfMemoryError: Java heap space

Need more RAM available to Java. You can try tuning java parameters around max allocation (or garbage collection), like -Xmx2048m for giving a particular JVM instance 2 GB of RAM to work with. Not sure from the message whether Fedora JVM or Solr JVM is at issue.