At the beginning of Aug. 2018, there was downtime on Islandora, likely due to multiple videos being uploaded simultaneously (FSCJ uploaded 2 at a time).
This is Randy's list of ways to prevent this in the future from an Aug. 9, 2018 email he sent to the FLVC Islandora Developer's listserv:
Here's a summary of possible solutions I've heard for avoiding the issues we've had this week. Please make additions.
These first four are not meant to be mutually exclusive.
1) disallow institution-initiated video uploads - we'll have them upload the videos and staff will run them through on production ourselves to minimize disruptions to our users.
2) modify the video processing command to use one CPU instead of the default four. This slows down video processing to 25% of normal --> implemented Aug. 11, 2018
3) support video processing by offline ingest, requiring packages that include MODS and manifest files --> Code change is complete for this as of Aug. 13, 2018.
4) modify zip uploads to wrap videos into packages, adding MODS and manifest files, and forwarding to the offline ingest service
Other systemic fixes, from easiest to hardest:
5) Offline ingest should be modified to be resistant to cascading failures when there's a temporary connection problem
6) The JVM memory settings should be modified to avoid the hang when RAM suddenly becomes available
7) We could start to split off services to their own processes (solr and blazegraph can run under jetty)
At the beginning of Aug. 2018, there was downtime on Islandora, likely due to multiple videos being uploaded simultaneously (FSCJ uploaded 2 at a time).
This is Randy's list of ways to prevent this in the future from an Aug. 9, 2018 email he sent to the FLVC Islandora Developer's listserv:
Here's a summary of possible solutions I've heard for avoiding the issues we've had this week. Please make additions.
These first four are not meant to be mutually exclusive.
1) disallow institution-initiated video uploads - we'll have them upload the videos and staff will run them through on production ourselves to minimize disruptions to our users. 2)
modify the video processing command to use one CPU instead of the default four. This slows down video processing to 25% of normal--> implemented Aug. 11, 2018 3)support video processing by offline ingest, requiring packages that include MODS and manifest files--> Code change is complete for this as of Aug. 13, 2018. 4) modify zip uploads to wrap videos into packages, adding MODS and manifest files, and forwarding to the offline ingest serviceOther systemic fixes, from easiest to hardest:
5) Offline ingest should be modified to be resistant to cascading failures when there's a temporary connection problem 6) The JVM memory settings should be modified to avoid the hang when RAM suddenly becomes available 7) We could start to split off services to their own processes (solr and blazegraph can run under jetty)