Stand Up Archivematica box

[davidschober]

Description

In order to work toward our Archivematica IMLS goals, we would like to create a simple POC where Avalon master files are ingested and dropped off in a location that Archivematica can suck up. For this reason we need an archivematica installation near the Spruce dropbox (on the same network).

Done looks like

• [x] a base install of Archivematica is created
• [x] The install of Archivematica has access to where Master files are moved to
• [x] Demo that archivematica is stood up with a manual ingest or an auto ingest.

[cjcolvar]

Maybe we could use this docker containerized version? https://github.com/ualibraries/archivematica

[davidschober]

https://github.com/artefactual-labs/am.git

[davidschober]

Archivematica folks can help to set this up.

[davidschober]

@phuongdh @cjcolvar ^^^^

[davidschober]

ping @ross-spencer

and @jhsimpson

[joncameron]

Where will we set this up (which machine)? @mcwhitaker will talk with @phuongdh about preferences on where to house the docker container.

[jhsimpson]

The https://github.com/artefactual-labs/am.git deployment will create 11 separate containers. That repo assumes you will run am.git from the host machine and all the containers will be in just the one machine. I'd recommend at least 2 cpu cores and 3 - 4gb of ram in the host machine, more is always better.

[phuongdh]

"Archivematica needs access to the drop box directory and and archiver spool-style directory where masterfiles get moved to"

[phuongdh]

On Fluorine now http://fluorine.dlib.indiana.edu:8985 http://fluorine.dlib.indiana.edu:8986

[mcwhitaker]

@phuongdh How can we QA this? Is there a combination of username/password that we can use? Also, I get a message that the connection is not secure. Is that how it should be?

[jhsimpson]

@mcwhitaker, @phuongdh, the Makefile in am.git defines a username/password combo for each webapp in the stack (dashboard and storage service). You can edit those when you do the install or use the defaults: https://github.com/artefactual-labs/am/blob/master/compose/Makefile#L68-L71 (tl;dr use test/test)

Regarding https - the am.git repo doesn't attempt to configure ssl at all. That could be done by changing the nginx container and having it provide https public endpoints, but use http 'internally' (i.e. from nginx to the dashboard and to the storage service).

Not sure if this is helpful - here is an example from a different repo that deploys archivematica in docker containers, that generates ssl certs and configures the nginx container in this way: https://github.com/JiscRDSS/rdss-archivematica/blob/master/compose/am-shib/nginx/create-secrets.sh

For non-docker deployments, at Artefactual we have used this ansible role https://github.com/artefactual-labs/ansible-acmetool to deploy acmetool and use Let's Encrypt certs to secure Archivematica installs. We haven't tried using that with am.git yet.

[phuongdh]

On Spruce, ingested master files are being moved to /srv/avalon/archivematica, this is mounted on Fluorine as /srv/avalon/spruce_masterfiles. We need to setup Archivematica to pick the files up from this location.

[mcwhitaker]

@jhsimpson Justin, now that we have the two ends up and running, I wonder if our next step could be for someone from your team to help us setup the archivematica app to pick up files from our spruce location. Does this seem like a good next step?

[sevein]

@mcwhitaker, I'm part of the Archivematica team. The simplest way perhaps to get AM running locally is to use a Vagrant box that we built for Archivematica 1.6. These are the installation instructions. Once it's up you should be able to set up a VirtualBox shared folder to make your contents visible to the virtual machine. In that box, /home is the default transfer source location so anything you mount under that location should be accessible by AM without extra configuration. You could also log into Storage Service and add extra locations, e.g. /vagrant.

If you need to run the latest version of the code you have two solutions:

• Docker + Compose (containers): https://github.com/artefactual-labs/am/tree/master/compose
• Ansible (single VM): https://github.com/artefactual/deploy-pub/tree/master/playbooks

As a developer I much prefer the first option. It also has a default transfer source location made available as a named Docker external volume that could be your /srv/avalon/spruce_masterfiles.

Either way I think I can provide help if it's needed.

[mcwhitaker]

@sevein We already have AM up. But we don't know how to use it. We need help figuring out how to tell it to pick up files from a given location.

[sevein]

@mcwhitaker oh I see! Has the location been made available in Storage Service? We have some docs here: Locations. Once he location is set up you should be able to start a transfer from the Dashboard. If you want automated transfers we have a tool for that in the automation-tools repository: Automated Transfers.

[mcwhitaker]

@sevein To make this faster for us, we would like to setup a time with one of you to get this done. Can this be?

[jhsimpson]

@mcwhitaker yes we can arrange that - best to do that through Sarah Romkey - I'll send an email.

[nickwilkinson]

@mcwhitaker I can help coordinate this with one of our sysadmins. I'll email you for the connection details to where AM is set up, and to schedule a time when a sysadmin can walk through the process with you.

[phuongdh]

Mounted spruce_masterfiles as /home/avalon_source and manually transferred a test file. We'll figure out the auto ingest later. Thanks Hector.

[mcwhitaker]

We were able to ingest the image files that come with Archivematica for demos. But our videos are not succeeding. We first tried an mkv and it did not succeed. We thought the problem would perhaps be related to the file format and tried an mp4. That also failed.

[ross-spencer]

@mcwhitaker can you share the job name under the micro-service, and the output from that here? - If you click on the cog next to the item, it will bring up additional information about both failures.

E.g. for this failed identification result, the cog button will bring up a new window with more information about what happened:

That outputs this (I purposely broke this file, normally it would be okay)

[mcwhitaker]

@ross-spencer Sorry, I said they failed, but they have not failed. They just don't complete:

The images ingest has that check mark but the videos have the processing icon.

[ross-spencer]

@mcwhitaker not at all! This is really helpful. I saw you've followed up by email, so this will get picked up by that work, but this screenshot gives me a lead I can follow in the mean time. Many thanks :+1: